Thursday, March 17, 2011

The Seven Most Discussed Scientific Biases

From David Chavalariasa and John Ioannidis (pubmed, doi:10.1016/j.jclinepi.2009.12.011):
  • confounding bias = when you think you are measuring the effect of variable X on variable Y, but in reality there is another variable Z that correlates with X and also affects Y, which you haven't considered.
  • selection bias = when you think that all the various sub-groups of the population are proportionally just as likely to be in your sample, but in reality certain groups are more likely to be present than proportional, because of the way you collect your data.
  • publication bias = when you are more likely to publish or tell others about your results if they 1) conform to what you expect, or 2) are what you think others would prefer to hear.
  • response bias = when respondents answer your questions in the way they think you want them to answer, rather than according to their true beliefs; this could also happen in animal research if you reward animals for responding in a certain way outside of the main test.
  • attention bias = when you focus only on data that supports your hypothesis and ignore data that would make your hypothesis less likely.
  • recall bias = when respondents are more likely to remember the content of your question if they hold a certain belief on it.
  • sampling bias = when you think your sample is representative of the population, but really it is not, because it is skewed in ethnicity, attractiveness, age, gender, and/or etc, casting doubt on your generalizations from the sample to the population. (this is actually a sub-category of selection bias, with the distinction of external vs internal validity that sounds cool but also troublesomely postmodern)
The legendary Ioannidis apparently had a dream of writing a "book of bias," but settled for this paper. They also have a cool cluster map of how all the biases relate to one another, but unfortunately this paper is not open access so I can't post it here for your viewing pleasure. The above is based on text-mining from pubmed, which of course has its own biases.

Added 3/21: Eduardo Zugasti has translated this post to Spanish here.