The China Study: Crushed by its Own Data

You may have already seen this outstanding analysis of the data from "The China Study". If you haven't, I highly recommend you give it a read. It's long, but well worth the effort. Readers of this blog know my opinion of T. Colin Campbell and his "scientific" work. Now somebody has taken the time to actually crunch the numbers, using Campbell's own data to demonstrate that his conclusions are baseless (at least when confined to this data), and probably the result of confirmation bias.

I also love the observation that, despite his constant whining about the "dangers of reductionism" in science, Campbell's entire argument against animal protein really hinges on a strongly reductionist experiment, namely the isolated effect of casein fed to rats in large doses. Snap!

Readers know of my criticisms of classical statistics, but it should be noted that I don't really have a problem with the mathematics, but the application. Math is what it is, either right or wrong. My issue is that classical statistics is used incorrectly, to draw inferences about hypotheses, when the underlying mathematical framework has nothing to do with inference. The key problem is that "statistics" are just numbers derived from data, like correlations. They don't say anything about a hypothesis: you will calculate the same correlation between two datasets, regardless of your hypothesis about what causes that correlation. Anyway, I don't want to get off on a rant. My point here is that the author, Denise Minger, does an excellent job of confining her analysis and conclusions within the bounds of what classical statistics can tell you. And along the way, she does a great job of demonstrating how easy it is to fool yourself (as T. Colin Campbell did - repeatedly) by over-interpreting these numbers which, in the end, cannot tell you anything more than what's in the data.

Ms. Minger has also done a great service in providing a concrete example of the issues in observational studies. You've likely read often that epidemiological studies are of little use in distinguishing between competing hypotheses. Now you have an example, replete with numbers. Ms. Minger demonstrates in several cases how a seemingly "obvious" conclusion vanishes once you dig into the large number of uncontrolled variables inherent in all observational studies. It's easy to find correlations in large datasets with many uncontrolled variables. The problem is that people take these correlations to mean more (or less) than they really do in terms of supporting/undermining a particular hypothesis, and the conclusions they draw are essentially ad hoc, not based on any rigorous mathematical analysis, but rather hand-waving about what is "obvious". An oft-quoted example is that men who shave daily have a higher incidence of heart disease. It is "obvious" that heart disease is not caused by shaving, right? Or is it? There's a whole lot of other information that goes into that judgment. We generally take this sort of thing for granted, especially when made in pronouncements from "esteemed" scientists like T. Colin Campbell. But if you dig into the reasoning behind these conclusions, you generally find a tangled web of assumptions, hypotheses assumed to be true, but which have varying (if any) actual evidence to support them. Ms. Minger does a great job of teasing these out of Campbell's reasoning, and demonstrating how the data itself provides little evidence one way or another, precisely because it cannot distinguish between the potential effects of the many intertwined and uncontrolled variables.

Anyway, enough of my babbling. Go read the article, you'll be glad you did (unless you're an uncritical fan of T. Colin Campbell, in which case you've got bigger problems).