Fighting The Lernaean Hydra Bias

I'll just only mention the heads I do cut off

In one Greek myth, Hercules takes on the task of killing a serpent-like, many-headed beast. This is made more difficult by the fact that its heads regenerate, so even if Hercules chops one off with his sword, another will simply sprout in its place. John Ioannidis uses this frustrating scenario as an analogy for a problem in the world of scientific publishing in his discussion of meta-analyses (doi:10.1002/jrsm.19).

The example Ioannidis employs to explain this problem is his experience doing a meta-analysis on the pharmacogenetics of certain polymorphisms for asthma treatment (doi:10.1097/01.fpc.0000236332.11304.8f). There were many studies that fit the criteria, but they each evaluated their own endpoints and genetic contrasts. That is, in most of the studies, the vast majority of possible correlations that could have tested with the data between phenotype and genotype were either not done or not reported.

So the surface problem, in so far as this case generalizes to others, is that published studies are not as exhaustive as they could be. But the central, troubling implication is that these studies do not fail to be exhaustive because of time or computational constraints, but because the researchers want to emphasize the usefulness and/or interestingness of their results. This is more insidious--this is why the hydra heads regenerate.

Now, one can use meta-analysis to retrospectively "chop off" findings that are truly insignificant by combining the results of many different data sets. But meta-analysis itself can be biased in many ways (e.g., during study selection), and moreover, later researchers can just come back to the issue and cherry pick more novel associations, thus "sprouting" more statistically significant findings.

When faced with the hydra, Hercules knew he couldn't go it alone, so he called on his nephew for help, who suggested that they cauterize the stumps with fire before the heads could regrow. An analogy to this strategy might be to post warnings on the electronic copy of papers that have been called into question by later studies. Such a warning would be much milder and hopefully less political than a retraction, which typically implies some sort of error. Publishing a potentially informative result that is eventually overturned is still laudable.

But instead of this type of patchwork fix, a more fundamental approach seems more fruitful. In the original myth, only one of the hydra's heads was truly immortal, and this was the one that Hercules needed to chop off to finally defeat the beast. The immortal head of the scientific publishing hydra is the incentive structure pushing researchers towards significance hunting in the first place.

Reworking these incentives is what Ioannidis is fundamentally arguing for, as the way to kill the Lernaean hydra bias once and for all: more standardization, more consortia, and more of a push towards openness and replicability. Every study might combine previous data with its own for estimating the posterior probability of the parameters it is examining, and all research might be seen as a continuous and cumulative meta-analysis. Maybe one day.

(photo credit to Frank Rafik)