Outside the Beltway

Outside the Beltway

A New Critique of the Lancet Studies

Steve Verdon
12

David Kane of Harvard University has put forward a new critique of the Lancet studies on the excess deaths attributable to the invasion of Iraq and the subsequent conflict. The gist of the critique is pretty simple overall: when one includes all the relevant information the results of the Lancet studies, at least the first one, are statistically insignificant.

The only reason why the authors found a statistically significant result is that they threw out the data on Fallujah. One could argue that Fallujah is such an extreme outlier that throwing the result out makes sense. However, it is also reasonable to at least include results for all the data in the paper, which for the most part was not done. Further, when one has to throw out data to get a statistically significant result it is usually viewed with some skepticism since one is indeed not using all relevant information. That is, it is usually better to use all the information one has than to limit oneself to just a subset of the information.

Now one thing should be made clear, unlike some commenters, David Kane does not appear, at least in the article linked above, to be saying that things have gotten better. The message appears to be that we can’t tell if things have gotten better or worse using all of the date from the Lancet studies. Indeed, as Iraq Body Count has argued,

Do the American people need to believe that 600,000 Iraqis have been killed before they can turn to their leaders and say “enough is enough”? The number of certain civilian deaths that has been documented to a basic standard of corroboration by “passive surveillance methods” surely already provides all the necessary evidence to deem this invasion and occupation an utter failure at all levels.

Why so many defenders of the Lancet studies seem married to the results is beyond me. Pointing to a fraction of the numbers in the Lancet studies like 60,000 excess deaths speaks very, very poorly of the U.S.’s presence in Iraq and of the invasion overall.

Over at Deltoid, there is more discussion as well. The discussions there are very heavy with statistical concepts so consider yourself warned if you plunge in over there. But one thing did catch my eye,

3) Goodness knows that this discussion would be a lot more productive if the authors were to release their code. Elizabeth Johnson, the graduate student who actually did the calculations, does not respond to my e-mails or phone calls.–David Kane

This seems to be a disturbing trend with some of the scientific researchers who support causes espoused by the Left side of the political spectrum. Mann, et. al. have been very reluctant to pass on both data and code for their global warming research, and now we have the authors of the Lancet studies not sharing code? I think any research that is geared towards having some impact on public policy should also be put into the public domain. That means all code, data, and so forth.

, , ,
, ,

12 responses to “A New Critique of the Lancet Studies”

  1. Dave Schuler Avatar
    Dave Schuler

    I think any research that is geared towards having some impact on public policy should also be put into the public domain. That means all code, data, and so forth.

    Does any research have no impact whatever on public policy?

  2. Steve Verdon Avatar
    Steve Verdon

    How about econometric studies of baseball? Or research articles on general equilibrium models with a non-atomic economy (I made up that last one, but most if not all of the articles in mathematical econ journals probabaly qualify)?

  3. Andy Avatar
    Andy

    This seems to be a disturbing trend with some of the scientific researchers who support causes espoused by the Left side of the political spectrum.

    Care to point to any evidence besides a few cases that you happen to be interested in?

    I’m sure you have statistically significant evidence showing that THE SCIENTISTS OF THE LEFT(tm) witholds evidence more often than those scientists with, er, right wing agendas.

  4. charles austin Avatar
    charles austin

    Yeah, we’ve got to stop all the scientists with right wing agenda’s — both of them!

  5. Anderson Avatar
    Anderson

    The Deltoid commenters pretty much ate Kane’s lunch; slightly more accessible critiques are at Crooked Timber’s post & comment thread.

    “Laughably bad” appears to be the general opinion of Kane’s article.

  6. Michael Avatar
    Michael

    Further, when one has to throw out data to get a statistically significant result it is usually viewed with some skepticism since one is indeed not using all relevant information. That is, it is usually better to use all the information one has than to limit oneself to just a subset of the information.

    Now I admit it’s been a long time since I took my statistics classes, but I seem to remember calculations to do exactly that to gain a more accurate statistic. Sometimes a anomaly will skew the data, and throwing it out gets you a more accurate view.

    Here’s an example: Using average number of murders going back 10 years, is New York City more dangerous in September or October of this year? If you don’t throw out 9/11, it will almost certainly show a higher average murder rate in September, even if October would otherwise have more. Anamolies are bad when you are looking for a trend.

  7. Grewgills Avatar
    Grewgills

    Adding to Michael’s analogy.
    If over the past 20 years October has an average of 30 violent crimes/100,000 people with a SD of 3 and September has an average of 43 violent crimes/100,000 people with a SD of 4 (without including 2001). I have not reinstalled my statistical package since I reformatted my HD, but it is likely if I plugged in normally distributed data sets with these values this would prove significant. Include 2001 and now September has an average of 193 murders and a much higher SD is no longer normally distributed and there is likely no longer a statistically significant difference in the murder rates for September and October.
    Does this mean that there is no real difference in rates of violent crime between Sept and Oct?
    Which test more accurately reflects continuing trends?

  8. Hal Avatar
    Hal

    I think any research that is geared towards having some impact on public policy should also be put into the public domain. That means all code, data, and so forth.

    Coming from the mouth of a libertarian, this is almost comical. Who said irony is dead?

    Again, please illuminate what purpose this would serve? You keep on making this point, so it would help us all if you would devote some blog post to providing some level of details as to why this actually matters, rather than just throwing the assertion out without a single scrap of evidence used to support it.

    Do you have the source code to SAS? What about Excel? How about MathCAD? Mathematica? My lord, do you question every spreadsheet result because you can’t see how they coded it?

    I’ll say this again, slowly, so you’ll get it.

    If you have a problem with the results, you do the same calculations with you’re own code. If you come up with different results, then BINGO! You have an issue. You’re now on firm ground to make hay with the research results in question.

    If you’re too frick’n lazy (as Kane apparently is,not to mention all the Global Warming Denialists(tm) ) to do the calculations yourself, using your own code, or don’t have a grad student or intern you can enslave to do it for you, then you simply don’t even deserve to be criticizing the results.

    Again, this is precisely like saying that Fluke has to provide the schematics for their Voltage meters in order for you to accept their measurements.

    Science doesn’t work like this. In the case of Fluke, there are well developed mechanisms called experiments, peer review and standards organizations which deal with certifying such things. You can even do frickin’ simple experiments to verify that Fluke is, indeed, telling the truth regardless of what they’re doing inside with their proprietary circuits.

    Unless you’re a serial quantum misinterpreter, reality is not dependent on the instruments used to measure it, nor are statistical results contingent on the code used to analyze them.

    Anyone who thinks differently fundamentally misunderstands science.

    It’s really that simple, Steve.

    Seriously, dude, if you really believe this and think I’m full of it, do a blog post on it and fill us all in on the logic behind it.

    Do I have to wait for the crickets to stop chirping?

  9. Steve Verdon Avatar
    Steve Verdon

    Care to point to any evidence besides a few cases that you happen to be interested in?

    Actually McKitrick has been having a very hard time getting data and code out of just about all the global warming researchers he’s corresponded with.

    I’m sure you have statistically significant evidence showing that THE SCIENTISTS OF THE LEFT(tm) witholds evidence more often than those scientists with, er, right wing agendas.

    I think you need to re-phrase as that isn’t what I wrote. What I wrote is that scientists who work on issues that are usually espoused by the Left tend to be doing a bad job at handing over data and code. Note that I didn’t say anything about the politics of the actual scientists.

    The Deltoid commenters pretty much ate Kane’s lunch; slightly more accessible critiques are at Crooked Timber’s post & comment thread.

    “Laughably bad” appears to be the general opinion of Kane’s article.

    Actually, I’d say a great many commenters were off-base. Often times they ascribed to Kane positions that Kane clearly implied were of the Lancet researchers. Daniel Davies made some really egregious blunders in his CT post as well–e.g. the comment about diffuse priors was particularly bad.

    Now I admit it’s been a long time since I took my statistics classes, but I seem to remember calculations to do exactly that to gain a more accurate statistic.

    Actually, the problem is that if you throw out data to get a statistically significant result then that is usually considered suspect. You should at the very least present results with and without the suspect data. And please define the term statistical accuracy? Do you mean statistical significance? If so, see my comments above.

    Michael,

    Here’s an example: Using average number of murders going back 10 years, is New York City more dangerous in September or October of this year? If you don’t throw out 9/11, it will almost certainly show a higher average murder rate in September, even if October would otherwise have more. Anamolies are bad when you are looking for a trend.

    Actually, knowing what 9/11 is and factoring that into the analysis would allow one to avoid the problem you are talking about. The problem you are pointing to is one where we don’t use all available information. We can do that and still include 9/11 and not have a problem.

    Bayesians often point to this as a potential short-coming of Frequentist methods.

    Grewgills,

    See my comment above to Michael.

    Hal,

    Coming from the mouth of a libertarian, this is almost comical. Who said irony is dead?

    I’m a libertarian, a minarchist if you will, not an anarcho-capitalist. Hence I think you have your terms seriously confused. I don’t oppose any and all government activity, but I’ll be damned if I am going to support a policy that will cost trillions without making sure the data and code used by the scientists supporting the policy is at least reasonable.

    Again, please illuminate what purpose this would serve? You keep on making this point, so it would help us all if you would devote some blog post to providing some level of details as to why this actually matters, rather than just throwing the assertion out without a single scrap of evidence used to support it.

    I think the point should be obvious, if the data is bad or fudged, and this does happen, then we shouldn’t be basing far reaching policy (for the next 100 years) on it. If the code is badly written and hence the analysis is flawed and producing bad results, then we shouldn’t be basing far reaching policy on it. For example, there seems the be a severe paucity of research on Global Warming and the economics of various scenarios.

    Now this doesn’t mean that global warming is false or that we shouldn’t do anything, but at least do some due diligence before enacting these policies. You know, making decision based on rational thought, evidence, and analysis vs. emotionalism and bullshit.

    If you have a problem with the results, you do the same calculations with you’re own code. If you come up with different results, then BINGO! You have an issue. You’re now on firm ground to make hay with the research results in question.

    Actually, some have already redone some of the calculations for global warming and found them wanting. The problem is that the researchers for the initial research refuse to release the code and additional data. As for the Lancet study the data is kept secret as with the code. So your rather stupid request simply isn’t possible. Idiot.

    Do I have to wait for the crickets to stop chirping?

    You are a woefully ignorant fool.

  10. Grewgills Avatar
    Grewgills

    Actually McKitrick has been having a very hard time getting data and code out of just about all the global warming researchers he’s corresponded with.

    Care to be more specific?

    I think you need to re-phrase as that isn’t what I wrote. What I wrote is that scientists who work on issues that are usually espoused by the Left tend to be doing a bad job at handing over data and code.

    Again, care to be more specific? Do you have any real evidence that scientists who work on these issues (global climate change, Iraq death tolls, what else?) are significantly more reticent to divulge information than scientists working on any other topic?

    Actually, knowing what 9/11 is and factoring that into the analysis would allow one to avoid the problem you are talking about.

    How would you go about doing this, short of recognizing that it is an outlier? It seems to me that they knew what Falujah was and factored that into the analysis by recognizing it was an outlier and treating it subjectively in their analysis.

    I think the point should be obvious, if the data is bad or fudged, and this does happen, then we shouldn’t be basing far reaching policy (for the next 100 years) on it. If the code is badly written and hence the analysis is flawed and producing bad results, then we shouldn’t be basing far reaching policy on it.

    You have thusfar failed to show that any data was fudged, any code badly written, any analysis flawed, or any results bad on either front. Absent this why should we base far reaching policy on your position?

    Now this doesn’t mean that global warming is false or that we shouldn’t do anything, but at least do some due diligence before enacting these policies.

    There has been considerable due diligence, beginning in the late 70s, early 80s and continuing till the present and on. For some there will never be enough evidence and no amount of diligence will be sufficient. It is beginning to look like you are in this camp. No serious scientist doubts that global climate change is happening and very few doubt anthropogenic causation, the major debate left is about consequences and what we should do in response.

    Actually, some have already redone some of the calculations for global warming and found them wanting.

    Care to share your calculations for our review? Please provide the code and all data you used.

    Re: the Lancet article and Elizabeth Johnson
    Elizabeth Johnson has had e-mail correspondence with David Kane and holes remain in his analysis. Perhaps you should update your post.

    You should lay off the ad hominem attacks, they add nothing of value to the discussion.

  11. Andy Avatar
    Andy

    What I wrote is that scientists who work on issues that are usually espoused by the Left tend to be doing a bad job at handing over data and code.

    You still offer exactly zero evidence for this assertion.

    This all strikes me as more of a typical reactionary response to global warming and, as Hal as pointed out, a rather fundamental misunderstanding of the scientific process.

  12. Steve Verdon Avatar
    Steve Verdon

    Guys, go read Climate Audit for a few days then come back. There is plenty of evidence over there, or read some of my older posts on this issue. Or, you know, google it.

Trending