Five Out Of Four Studies Are Wrong

Untitled document

Well, that's a bit of an exaggeration, but it is not very far off the mark, either. According to a number of very authoritative studies by Dr. John Ioannidis, many research studies are wrong. And the hotter the research field, the more likely the studies are tainted.

We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong.

Dr. Ioannidis is an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass. In a series of influential analytical reports, he has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.

These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."

The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.

Dr. Ioannidis has been publishing a lot about this for a while now. Here's one of his papers that lays out a lot of very, very interesting conclusions. (The paper may be downloaded from PLoS Medicine. It is the most requested article they have ever published.)

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. This seemingly paradoxical corollary follows because, as stated above, the PPV of isolated findings decreases when many teams of investigators are involved in the same field. This may explain why we occasionally see major excitement followed rapidly by severe disappointments in fields that draw wide attention. With many teams working on the same field and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritize on pursuing and disseminating its most impressive “positive” results. “Negative” results may become attractive for dissemination only if some other team has found a “positive” association on the same question. In that case, it may be attractive to refute a claim made in some prestigious journal. The term Proteus phenomenon has been coined to describe this phenomenon of rapidly alternating extreme research claims and extremely opposite refutations [29]. Empirical evidence suggests that this sequence of extreme opposites is very common in molecular genetics [29].

And what is one of the hottest fields right now? Why, it's global warming. What a surprise. With more and more lurid findings being reported almost daily by the press. Hmmmm. From the WSJ article again:

Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. "People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual," Dr. Ioannidis said.

This should raise some very serious red flags for people.

UPDATE: I just love being linked by someone who teaches philosophy who says things like this:

Exhibit #1: Blue Crab Blvd: "And what is one of the hottest fields right now? Why, it's global warming. What a surprise. "

Does Dr. Ioannides say that global warming research is one of the problem areas? No. This is entirely Mr. Crabs' interpolation.

It might help the professor to understand the terminology:

In the mathematical subfield of numerical analysis, interpolation is a method of constructing new data points from a discrete set of known data points.

In engineering and science one often has a number of data points, as obtained by sampling or experiment, and tries to construct a function which closely fits those data points. This is called curve fitting or regression analysis. Interpolation is a specific case of curve fitting, in which the function must go exactly through the data points.

Whereas extrapolation is:

In mathematics, extrapolation is the process of constructing new data points outside a discrete set of known data points. It is similar to the process of interpolation, which constructs new points between known points, but its results are often less meaningful, and are subject to greater uncertainty.

So, did I make a best fit interpolation or a good (but possibly uncertain) guess based on the report? Well, actually, unlike the good professor, I actually read Dr. Ioannidis' paper. And his findings would appear to be applicable across scientific fields – because he is looking at the entire way research is conducted. So if anything, it is a best fit, not a guess.

Hey, prof, just so you know, my degree is not in philosophy. It is in mathematics. And thanks for acknowledging that I "interpolated" it correctly.

This entry was posted in Environment, Science. Bookmark the permalink.

3 Responses to Five Out Of Four Studies Are Wrong

  1. Mwalimu Daudi says:

    You hit the nail on the head, Gaius!

  2. Off topic:
    More about those two Egyptian students arrested in Florida. arrested with the PVC pipe.

    “Investigators seized the laptop at the traffic stop. They later found that the most recently viewed files included videos showing Qassam rockets being fired along with visits to Web sites relating to Hamas and videos involving martyrs. It was not clear whether those files were being viewed as the two students drove.”

    Building rockets it would seem. Of the terrorist sort.

  3. feeblemind says:

    It is nice to see the WSJ bring this to light. Pseudo research had been a problem for many years. As long as there are stacks of government money handed out for research, this problem will persist.

Comments are closed.