Peter Freed has written a pretty ambitious critique of Jonah Lehrer's summary of this study (pdf) on the wisdom of the crowds. The crux is that:
In prediction markets, the most recent price of a transaction doesn't always best represent the current beliefs of the market. There's more info if you look at the whole distribution of orders. Similarly, it is unfair of Freed to dismiss the whole data set just because one type of estimator is flawed. This is one of the coolest parts of statistics, using potentially counter-intuitive methods to extract useful info out of data, to find the wisdom in the crowds.
But now that I realized he really meant median, and that maybe he didn’t know what median meant. Because median guesses are not guesses by a crowd, as Lehrer states. They are guesses by a single person... [Lehrer] is talking about that 0.7% single-person data point: one person, selected after giving their answer, got close to the correct answer on one of six questions. One person guessed 10,000 when the answer was 10,067. That’s one hit out of 144 x 6 = 864 attempts. That seems about right to me, from a common sense perspective. Which is to say, that is a shitty batting average.Scrolling through the comments, I was pleased to see Ian Sample point out the critique of Freed's critique that I was going to make:
In Wisdom of Crowds studies you can look at the mean and / or the median. The median usually gives the best result if the guesses *do not* follow a normal distribution. The mean, of course, exploits the error-cancelling advantage that WOC is known for, that is, as many people under-estimate as over-estimate the right answer, so averaging cancels all but systematic biases. But to my point. To dismiss the median answer – one guy’s response – misses the fact that without the crowd you have no median answer to dismiss. Without the crowd, you do not know which value to pick. That’s the whole point. The crowd steers you to the median value, which in many cases outperforms the mean.The median is indeed generated by only one person, but it becomes interesting only in the context of all the other estimates. It is useful here because it offers resistance to outliers. For example, some less numerate soul might have guessed 1,000,000, which is way off from the true value of ~ 10,000, thus skewing the arithmetic mean. In that case you'd much prefer a more robust statistic like the trimmed mean or the median.
In prediction markets, the most recent price of a transaction doesn't always best represent the current beliefs of the market. There's more info if you look at the whole distribution of orders. Similarly, it is unfair of Freed to dismiss the whole data set just because one type of estimator is flawed. This is one of the coolest parts of statistics, using potentially counter-intuitive methods to extract useful info out of data, to find the wisdom in the crowds.
