Sunday, November 7, 2010

Trust The Ratings Of Others

An '09 paper (link here, pdf here, HT to TC) claims that people make more accurate emotional predictions about a future event when they are simply told how someone else reacted to that event, as opposed to when they are given info about the event. This is somewhat counter-intuitive, so let's look at their evidence.

One of their tests was speed dating. Each guy submitted a photo and some demographic info about himself. Then, each girl predicted how much she would enjoy the date based on either this photo / info or the enjoyment rating of a girl who earlier had gone on a date with the same guy. Next, the guy and girl had their five minute date, (ignore the heteronormativity, my fellow Vassar alums), and finally the girl rated how much she enjoyed herself on a sliding scale of 1 - 100. 

The authors define prediction error as the difference between the girl's predicted and actual enjoyment ratings. Participants made more accurate predictions when they used the first girl's enjoyment rating to predict their own (the avg error was 11.4 +/- 8.7) than when they predicted their enjoyment on the basis of the info (an avg error of 22.4 +/- 10.8).

In classic psyc study fashion, they also asked their participants to say which condition they thought would lead to more accurate predictions. 75% said the info would be more useful than the rating of a girl who had already been on a date with that guy. Oops. Now, indulge me in a few reactions:

1) Why do people underestimate the value of someone else's rating? Probably because people think of themselves and their opinions as more unique than they actually are. This sets up my public choice theory for why popular critics like Anthony Lane tend to be negative and contrarian. Although on the surface this annoys readers, people on a deeper level prefer to read opinions about art that they disagree with, because it allows them to think of their own opinions as more unique.

2) There is only one specified relationship between the study participants: they are all undergrads at the same school. So although the authors toss the word "social network" in towards the end of the paper, their results do not speak to the predictive power of a friend's opinion as opposed to a stranger's opinion. This remains an open question--in predicting their own enjoyment, will people find the opinion of someone in their network more valuable than the average opinion of strangers? Even if you say yes, you must take into account the trade-off of sample size, which is larger when you listen to the masses. The high valuation of sites like facebook relies in large part on the assumption that we'll prefer recommendations from those in our network, but I'm not so sure.

3) A subsequent study looked at how people combine their own mental simulations and third-person reports of other's experiences in making judgments. Corroborating the results of this study, they found that people assign far too much weight to their own simulation of how an event will play out as opposed to the feelings of other people who have actually experienced the event. I myself find this all very relevant to imdb's movie ratings. Remind me again why you trust yourself to judge a movie instead of deferring to the aggregated ratings of others?