Research incentives are messed up, big time. So much so that Sturgeon's Law of fiction writing applies, doubled: when someone protested to him about the presentations at a science fiction convention, that 90% of it was crud, he replied, "90% of everything is crud!" When people's careers are on the line, when jobs, promotion, tenure and salary all depend on "Publish or Perish", virtue vanishes. Young, idealistic researchers become jaded, cynical cheaters. One medical author has written that as much as 99% of published medical research is valueless or even damaging. Another wrote,
"One must not underestimate the ingenuity of humans to invent new ways to deceive themselves."This quote is found on page 192 of Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions by Richard Harris. Author Harris admits that his title is a bit tongue-in-cheek, because rigor mortis literally means the stiffness of a corpse, while "rigor" also means strictness in carrying out a procedure. While it might be more accurate to title the book Mortis Rigoris (the death of rigor) or Mortuus est Rigor (rigor has died), it wouldn't resonate with doctors and others of us who know Latin.
More accurately, however, while experimental rigor is neglected more than adhered to, and may be on the ropes, it isn't quite dead yet. The ten chapters in Rigor Mortis illustrate and document every major aspect of medical research, from experimental design (The "gold standard" of the double-blind trial is nearly always compromised to save expenses, and frequently foregone entirely) to animal studies (Suppose you were told that a certain medicine was tested exclusively on women pregnant in their first trimester, of ages between 22 and 25, all from a specific ethnic group in Scandinavia? That's the analogy to a typical mouse study) to statistical analysis (The p-test is dramatically misleading, and we'll get into that one anon).
Have you ever heard of the "desk drawer file"? It is a lot like a Roach Motel; experiments with "negative results" check in, and are never checked out. Some of the few honest researchers left in the field are agitating for a requirement that every study funded with tax dollars be published, no matter what the outcome. The good news: transparency. The bad news: a ten- to 100-fold increase in the number of papers published. There is already an overwhelming deluge of publication! Gack!!
We need look no further than this to validate Sturgeon's Law. Consider the much-overused p-test, or p-value. You take a bunch of numbers, grind the formula (found in every statistical software package out there, including Excel), and out pops a number. Is it smaller than 0.05? Publish! That number gets inverted into "95% statistical probability that the result shown is not due to chance." Hmm. But what is it due to? Sunspots? Batch effects (perform run 1, clean equipment, perform run 2; do they differ because of the cleaning?)? Something you would never think of in your wildest dreams (all too frequently, yes)? But just suppose all those "95% chance it's right, 5% it's wrong" papers actually do have the "correct" cause and effect. How many experiments went to the "desk drawer" since the last time you published? 5, 10, 20, 100? The average is (wait for it) about 20! So, ignoring the desk drawer, five out of 100 publications must be reporting a chance association or correlation. Add the desk drawer factor of 20, and now at least half of them are reporting a correlation due to chance. Just by the way, it is amazing that the vast majority of studies that report a p-value have a number just under 0.05: "Dig around until you get a p-value you like, then stop looking."
Add in other factors, all detailed in Rigor Mortis, and there is little chance that more than a tiny fraction of published research results will stand the test of time. And that is a problem. A little time? That is OK. If a lot of further research and even development and marketing are based on a faulty result, and it takes "medical science" 5, 10, 20 years or more to find and correct the mistake, how many people die or suffer needlessly?
Is there a way out of it? Only partially. Transparency is part of the answer. But bureaucrats are lazy, so even with a law on the books that all studies funded by NIH must publish all results, for example, it is poorly enforced. There are a lot of partial answers out there. Here is my answer: We must live with what we have now, while things are possibly getting better, but today is today. When I must choose a new doctor or specialist, I inspect the waiting room, and later the visitation room. How many drug company trinkets can I find (pens, calendars, note pads, posters, and many more)? The fewer the better. My current doctor's rooms don't have anything with a logo on it. That's a great start; it means the doctor has better-than-usual resistance to high-pressure sales. Such a doctor is more likely to make a decision on medical grounds. Secondly, who do I actually see? Curiously, I prefer to be seen by a PA or NP, rather than a DO or MD. They haven't had all their good sense educated out of them yet. In my experience they are also a lot more willing to answer questions and do so more meaningfully. Also, I do ask a lot of questions, because a brusque doctor is likely to be impatient in the operating room also. In medicine, patience isn't just a virtue, it is a necessity! There is more, but if you aren't doing these things, start there.
What else can you or I do? Educate yourself. Not from medical journals, but from summary materials on things that are known to work. WebMD and Healthline are just the beginning. Don't limit your reading to a single source. When offered a "new" drug, always ask, "Is there an older one that works well enough, perhaps with fewer side effects?" There are always side effects. Some you can live with, some you can't. Do avoid, desperately, a drug that needs another drug to deal with side effects. My wife takes a statin drug for high cholesterol. She was originally prescribed the strongest one, and even taking a tiny dose, had troubling side effects. Her "undrugged" total cholesterol is 240, but that drug is best used for folks in the 400+ range. She demanded a weaker one, and even then, splits the pill in thirds. She has no noticeable side effects, and her "drugged" total cholesterol is about 160. Good enough!
I've learned to tell a doctor, "I am not a patient. I am a customer. You and I will collaborate. I will never cede my right to make decisions, except during anesthesia that we have agreed upon together." Call it an intelligence test. For the doctor. Occasionally a doctor fails it, and then I get another doctor. When needed, I make a doctor aware how skeptical I am of the "evidence" presented in modern journals.
Rigor Mortis is scary. Is it right? Sadly, yes, it is more right than the average published medical study. But don't let that drive you to the amorphous world of "alternative medicine", at least not wholesale. Allopathic medicine has produced amazing health in most Americans and others in the First World. For a generation or so medical research has gone astray. Will it return? Maybe. Until it does, we must be our own best doctors.
No comments:
Post a Comment