Medical Backtracking
Gail Collins proclaims the first ten years of the new millennium “the Decade of Medical Backtracking.”
Somewhere between the reports that Pap smears and tests for prostate cancer aren’t all they were cracked up to be and the news that a high fiber diet doesn’t do anything to prevent cancer, the health establishment began looking decidedly nonomniscient. Then this week, a federal task force reported that most women don’t need annual mammograms. Even more fascinating, they suggested that doctors stop telling their female patients to self-examine their breasts for lumps.
[…]
Every rational American wants qualified experts to keep re-examining current medical practices. The only thing that bothers me about the mammogram report is all the emphasis on the “anxiety” that might follow a false-positive. We live in a time when we are constantly being reminded that a fellow plane passenger might be trying to smuggle explosives in his sneakers. We can manage anxiety.
I am going out on a limb to say that the real problem with a test that creates a lot of false-positive results is that it leads to a lot of other medical procedures, some involving hospitals. Unless you are genuinely sick, there is no more dangerous place to be hanging around than a hospital.
And let’s not forget the longer-term changes of mind on things like silicon breast implants, artificial sweeteners, and the danger of eating eggs.
Collins is right that we want medical science to constantly challenge prevailing assumptions and give us their best guess as to the truth. I continue to wonder, however, about the rigors of medical scholarship, which seems to frequently draw wide conclusions based on studies of very small, self-selected samples.
In related news:
Medicine does improve, but that progress sometimes misleads us to the Utopian idea that health is (or soon will be) perfectible.
I think we need to carefully distinguish between the science and the policies. The issue is more with the latter, I think.
Was there any change in the scientific community on breast implants and artificial sweeteners?
Anon makes a good point. Also to be kept in mind is the difference between a scientific study and the way the media reports on that study. Even when a study is filled with caveats on how it is to be interpreted, media tend to find the ‘exciting’ point and run with it.
As always, there’s a thin line between skepticism and cynicism. It helps to know on which side of the line you operate on a daily basis.
Statistics can only take you so far, and most of the medical “news” we get seems to be primarily based on statistical analysis rather than, dare I say it, science. I think it was Chesterton who said of someone that, “He uses statistics like a drunk uses a lamp post, more for support than illumination.”
Still recall that “study” showing that drinking decaf increased your chance of heart disease.
Thought to myself: “Self, who but someone at risk of heart disease would voluntarily subject themselves to decaf?”
That’s when I pretty much figured that these studies are purposefully undertaken in order to be controversial, thus increasing the odds of publication, media simplification and interviews. Nobody gets more grant money for confirming that bran makes you poop or that bacon is the ultimate foodstuff, but if you fly in the face of convention (even knowing that your study has built in flaws that do not pass the most cursory sanity check) you are guaranteed attention.
You understand that statistics is the language of science, right?
I don’t agree with this at all. Most of the studies that lead to public health policy changes are going to be epidemiological analysis of large datasets. Conclusions based on such statistical analysis can be problematic, but that’s why they are always just the first step. You identify the possible risk factors, and then design an intervention to test to see if you can improve outcomes. Now, yes, you’ll get many small scale studies to whittle down the field of possible interventions… but the “jury is still out” until you get the huge multi-center trials of hundreds to thousands of participants. It is perfectly understandable (and expected!) that you will see some of these large scale trials show that the earlier conclusions are spurious… but that’s how it’s supposed to work. We don’t have infinite money, so we can’t fund a $30 million dollar trial for every possible intervention for every risk factor that pops out of the statistics.
If anything is the culprit here, it’s probably medical science journalism that often over hypes (often the PIs are willing participants in this unfortunately) what are essentially preliminary findings.
All of this is, of course, the beginnings of the main argument for “Evidence Based Medicine”… since if you understand that we can’t really be sure how effective something is without thousands of patients studied, you realize we really shouldn’t be prescribing these treatments on such a large scale before we do so… since unnecessary treatments can be genuinely dangerous.
But, unfortunately, that’s what a large part of your side of the aisle likes to call “Death Panels”.
This paper, published at an on-line medical journal, was both amusing and influential (in some circles):
Why Most Published Research Findings Are False
Clovis, first, the coffee study randomly assigned participants decaf or regular. So a person with heart problems couldn’t chose to drink decaf. Second, remember that a scientific publication is supposed to be a informative record, not a definitive statement. Furthermore, they have have to be conducted in the real world, with all that implies. Let’s say that you want to investigate the effects of caffeine on heart disease. So what do you do?
Well, maybe the first thing that you do is write a $1,000,000 proposal to the NIH. Well, it gets shot down. They say it is too expensive, and you don’t have enough evidence to show that this study is worth conducting.
So, you decide to write a small proposal. You figure that you can get $15,000 for a small study. So you only propose a limited study. You get some interesting results. They are not definitive. What do you do? Well, it makes no sense to publish only definitive studies, because the way that definitive studies come about is because there is enough suggestive (but inconclusive) evidence that some funding agency is willing to put in big bucks.
This is not to say that there are no problems in medical research. There are.
It’s true that statistics only get you so far, but statistics are definitely part of science. You use the statistics to figure out what to study. Ultimately, we hope to be able to understand the mechanism.
But science occurs in a community, over a long period of time, and one person or lab can’t do it all. So it would make no sense for a statistical study to be kept from view, unpublished, until a mechanism could be found for it.
Truly understanding mechanisms is hard, and takes a long time. The more statistical evidence that can be found to point to the right direction for trying to find a mechanism, the better.
Statistics is a tool of science but statistical conclusion need to be tested. Too often statistics are misuse, abused, or plainly done wrong. I love statistics and science but hate how so many misuse them.
One thing that really irks me is how the MSM treats studies. They often treat one study that they like the conclusion of, as fact. Even when they are later proven inaccurate or wrong the MSM will still treat the lame study as fact. How many of the MSM and people believe that subliminal messages work or that 1998 was the hottest year on record? They have been properly refuted multiple times yet people persist that they are facts.
To precursor some replies, there are differences between disputable facts and non-disputable ones. Just because you believe strongly in something doesn’t make it indisputable.
By the way, here is a simple explanation for why it might be possible for many published findings to be false. Let’s say that you think people with blue eyes might be smarter. So you create a random sample, do some testing, etc. Everything is proper.
So far so good. But now suppose that 1000 other researchers get the same idea. They all do a similar study.
Well, we know that probabilistically some of those 1000 studies, just by chance, are going to find some significant correlation. Guess which study gets published. The ones that don’t find any correlation, or the ones that find correlations?
Depends on the eye color of the follow-up researchers. (Grey eyes are smarter.)
Anon at November 19, 2009 13:47 Permalink
Think we might be remembering different decaf studies, but it was some time ago so I perhaps recalled it incorrectly. Might be that I’m conflating the study with the statistical analysis that was the precursor to the proposal.
My point (aside from the one on my head) is that no one on the pure science side is going to put big dinero into confirmation, at least after the initial experiment has been confirmed once or twice. To get that big funding you have to show some hint that your experiment will set the world on its ear or lead to some counter-expectational result.
Now in some areas of applied science you can have guaranteed lifetime employment simply by performing experiments to confirm that, say, Cheerios do not cause brain tumors, deforestation, tectonic plate shifts or whatever the cause du jour is.
You will not get rich or invited on Oprah that way, but you will be performing better science than those who (consciously or not) seek sensation.
Clovis, just FYI, here is where I got my information about the coffee study.
You have a point that you won’t get big funding for something that doesn’t seem exciting. But one way to show some “hint that your experiment will set the world on its ear” is with a pilot study. Furthermore, the more counter-expectational, the stronger your evidence will need to be.
Certainly, it will also be hard to get big money for confirming something that everyone already believes anyway.
The trick is to be ahead of the pack, but not too far ahead, since then no one will believe you. Of course, now I’m talking more like someone trying to get funding, rather than a member of the public.
(And for the record, I don’t have any NIH grants, but I do have several NSF and DOE ones, but not in medical research.)
Well, no. Mathematics yes, statistics, on a more limited basis as it is merely a limited subset of mathematics. Unless you think science is limited to a B.F. Skinner-like black box approach. Statistics can give you some insight into what but they reveal next to nothing about the why, and for my money, the why is what science is all about.
Oh, and don’t start talking about quantum mechanics either for reasons far too lengthy to engage here.
The thing is you need the what before you can explore the why. When dealing with epidemiological research often the only ethical way to proceed is with statistical analysis of available data. Certainly there needs to be research into mechanisms as well, but both are real science and offer real insight.
True enough, but it is the most important dialect for the biological sciences.
The main issue is with bad science reporting and policies purportedly based on science that have only a tenuous connection to the available science. The recent story on dietary guidelines is a great example of the latter.
You are hitting on a key point here that way too much certainty is applied to scientific results which are a good deal less certain than they seem to policy makers. Certianly no real scientist is ever going to say, “the debate is over.”
Climate reference Charles? No scientist is going to say “though researchers in the field believe 9:1 one way, I will believe the other, without new evidence.”
It’s a child’s argument to say “since I can believe the other way, I will” (without better evidence).
odograph, when you stop intentionally mischaracterizing what I say with strawmen you can set aflame and patting yourself on the back for the ease with which you can incinerate your strawmen, perhaps we can have a conversation.
But it is stunningly hilarious that you use the word “believe” along with a poll in your argument. Seems to illustrate my point rather nicely. Perhaps if you spent some time outside your echo chamber you wouldn’t make such, ahem, childish mistakes.
Odo
Depends in where you are getting your sample for that 9:1 ratio. If you are getting it from a bias group like sceincetolgy or the sampler is bias, I would be weary. I like to make my own conclusions. Present the facts to me and I will decide for myself. This following the crowd mentality isn’t for me especially when someone main argument is†just follow the crowd that I’m followingâ€. Also the crowd gets it wrong too often for me.