Against The Word "Scientist"

After seeing Dan Ariely's rant about how unwilling companies are to test things empirically, this ngram showing that "natural philosopher" used to be the term describing one who did experiments, Aaron Haspel's slightly too profound tweet that "what scientists say is not science" (that should be obvious), Katja's exposition of how schools fail to teach science in a way that students can actually apply to their lives, and unconsciously synthesizing all of this info at some point during its consolidation from short-term memory in the hippocampus into long-term memory in the cortex, I have come to the opinion that the word "scientist" should not be used to describe any people or profession. Instead it seems better to refer to those who work in science labs as "researchers" or more specifically by their specialty, like immunologist. If the research community, and by that I refer to the group of people who fantasize daily of having a paper published in Nature, were to stop monopolizing the term "scientist," perhaps others would be more apt to employ the scientific method in their own endeavors, to the benefit of all.

Why Not Uniqueness?

First, check out the comments on my recent post about uniqueness, in which I was (rightfully) ripped to shreds. It is wonderful to have such a critical readership! What Brian's critique suggests is that humans, when they want to be unique (which is often), tend to do so in a particular way that most other animals do not. That is, humans are more likely to share major beliefs with others in their immediate group, in order to signal their loyalty, and instead try to differentiate their group from other groups. He's mostly right and I was lazy to write that in the first place. Also in the comments, Justin asks "what's so bad about uniqueness"? Bingo responds basically as I would (must be in the DNA), but in the next three paragraphs I'll expand a bit on that. 

Betting on horse races is systematically biased such that longshots are overbet and the favorites are underbet. There are many explanations for this, but I think the best is that the bettors, who are typically also spectators, prefer to have their "own horse" that they in particular can root for. Similarly, participants in the intellectual sphere like to have their own "horse" in any ongoing subject of conversation, in part because it allows them to signal their uniqueness, and in part because it is simply more fun (these two parts are probably correlated). This implies that the marketplace of ideas, much like the horse betting market, is likewise systematically biased; "favorites" or obvious ideas are not valued as highly as they ought to be. So the key reason intellectual uniqueness (or intellectual hipsterdom) is bad is because it diffuses our focus from the most important questions and problems.

One key question: is informing people about this phenomenon likely to change their behavior? Probably not. Think of hipsters. Every hipster knows that hipsters are generally annoying, and most hipsters are personally annoyed by other hipsters, but this stops almost no one from acting in their self-interest and being hipster-ish themselves. So this is probably a dead end.

What we can do is make rating systems that quantify and rank, as precisely as possible, the various options in each of the idea spheres. This way people can talk all they like about their particular "horses" in the intellectual marketplace, but when it comes time to make important decisions, we can rely upon the ratings rather than these uniqueness-biased discussions. (A prediction market is an example of a rating system, and when possible to set up these tend to perform the best. But they are not always feasible to set up, so we should consider other options too.) Figuring out how to set up rating systems to answer important questions, and how to incentivize idea consumers to actually rate things in such systems, seem like key questions, and they are questions this blog will continue to track closely.

Hanson To Fellow Bloggers: Do As I Say, Not As I Do

Robin Hanson responds to Vladimir M's post about when one should or should not trust an academic consensus. First a bit from Vladimir,      

When looking for information about some area outside of one’s expertise, it is usually a good idea to first ask what academic scholarship has to say... [I]t would be astonishing if there didn’t exist at least some areas where the academic mainstream is detached from reality on important issues... Areas affected by ideological biases... tend to drift much further into outright delusion, possibly lacking a sound core body of scholarship altogether... I’ll have to read much more on the technical background of these subjects before I can form any reliable opinion on these questions. 

Then Robin, 

But if you plan to mostly ignore the experts and base your beliefs on your own analysis, you need to not only assume that ideological bias has so polluted the experts as to make them nearly worthless, but you also need to assume that you are mostly immune from such problems!...  If ideology severely compromises others’ analysis on this subject, then most likely it severely comprises yours as well.  You should mostly just avoid having opinions on the subject.  But if you must have reliable opinions, average expert opinions are probably still your best bet.

The conundrum is that Robin himself does this all the time. See for example this post literally two days earlier, where he is skeptical of an academic consensus because he has reason to believe that the authors are biased, does some digging on his own from the raw data, and draws some interesting conclusions. That's exactly what Vladimir is saying one ought to do.

So Robin's disagreement seems a bit inconsistent, but it's really not. He has admitted that he doesn't specialize as much as he thinks people should. Specialization requires trusting the surface arguments in most fields in order to focus on one's own. In other words, he thinks people should generally not be contrarian, if only because it's a waste of time. Consensus definitions of hypocrisy emphasize that it's OK (i.e., not hypocritical) to preach temperance for others when you yourself have not conquered your passions, as long as you do not claim to have done so. So Robin is not being hypocritical.

Still, it might be nice if he admitted to often not following his own "don't be a contrarian" rule.

Trade Off #18: Speed vs Accuracy

As you decrease the time in which you complete a task, you must give up some of the control which allows you to ensure that each part of the process goes as desired. This trade off between speed and accuracy is very widespread and is one of the most intuitive trade offs in the canon.

You can test yourself on this trade off by going to any typing test website and varying the speed with which you type. At more words per minute you will tend to have more misspelled words, while at fewer wpm you will have fewer misspelled words. Other examples:

• Watson, the AI created by IBM to compete at Jeopardy, faces a speed vs accuracy trade-off in that the longer it has to search its databases and run its algorithms, the more confidently it can assign a high probability to one of its answers. (see here)
• If you take longer to respond to emails, people will expect your responses to be more accurate, whereas if respond right away, their expectations will be much lower. This is because they intuitively recognize the trade off you face between speed and accuracy. (see here)
• In the world memory championships, as contestants take longer to look at a card, their mental representation of it become richer, making subsequent recall more accurate. So they must race to make it interesting. (see here)

This trade-off seems to not apply everywhere, in that people's predictions about soccer games may actually decrease in accuracy with more conscious attention. But this is probably because people tend to over fit their models to the data with too much conscious processing, an example of the precision vs simplicity trade-off.

To say that there is a trade-off doesn't completely preclude free lunches. Ray Allen can shoot 3's more accurately than I can, and he can shoot them faster. The key is that if Allen were to shoot basketballs at 2x his own comfortable speed, he'd no doubt be less accurate. So, with respect to just this speed vs accuracy trade off, the amount he has practiced seems like a free lunch, but we know that once we consider other trade-offs, it is not.

(photo credit to flickr user Odd; the slower the sun sets, the more accurately you can predict the exact moment it will cross the horizon)

Squirrel Uniqueness

Kimberly Pollard and Daniel Blumstein recently published a study (HT Ed Yong) examining whether group size correlates with the individuality of squirrel alarm calls:

Discriminating among individuals is a critical social behavior in humans and many other animals and is often required for offspring and mate recognition, territorial or coalitional behaviors, signaler reliability assessment, and social hierarchies. Being individually discriminated is more difficult in larger groups, and large group size may select for increased individuality–signature information–in social signals, to facilitate discrimination. Small-scale studies suggest that more social species have greater individuality in their social signals, such as contact calls....

[W]e show a strong positive evolutionary link between social group size in sciurid rodents and individuality in their social alarm calls. Social group size explained over 88% of the variation in vocal individuality in phylogenetic independent contrasts. Species living in larger groups, but not in more complex groups, had more signature information in their calls.

Extrapolating not unjustly from this, it seems that humans, the most social of all animals, should have the strongest drive towards uniqueness. So the desire for your own opinion, as opposed to just the best one, is probably an evolved tendency. We should all be similarly troubled by this.

In Praise Of The Obvious

Ryan Holiday recently wrote a post encouraging us, in a roundabout way, to seek the counter-intuitive. On the contrary, I think the counter-intuitive is overrated. Consider:

• It is very difficult to find an RSS feed that only lists the basic, most important findings from any particular field, even though this would be the most useful to an outsider. 
• Inception is unlikely to win the Oscar for best picture (currently 80 to 1), even though it has by far the highest weighted average rating of any movie this year. 
• Academics love to conduct and publish paradigm-shifting research, but are much less interested in replications or confirmations. This leads to lots of problems.

The best way to explain these conundrums is to admit that we love the counter-intuitive, we love to associate ourselves with it, and we love to demonstrate our uniqueness in doing so. Far from encouraging these tendencies, we should instead shun them. If the obvious thing is best, we must endorse it just as strongly, or we risk sending confusing signals to others.

Granted, there are some trade offs here, mainly between finding new ideas and reducing uncertainty about our current ones (an example of the more general breadth vs depth trade off). That is, if we only focused on what was currently obvious, we wouldn't come up with any new angles. But, at the margins of our current intellectual climate, it still seems to me that we should prefer more obviousness.

Frustratingly, all else equal, counter-intuitive thoughts are a good sign that somebody has a deep understanding of a subject and has thought it through beyond the status quo. So if we really want to encourage obviousness, we need to value the traits that lead to it, like formality and diligence, even more so than we value traits that lead to novel ideas, like creativity. As Ice T says, it's not about being mad about everything, it's about being really mad at the right things.

So Bad It's Good, On imdb

My roommate was recently trying to convince me to watch The Room, employing the argument that, although it is poorly rated at a 3.2, it is "so bad it's good." The question is, can we quantify this?

The intuitive way seems to be to look at the distribution of scores, and indeed that is what Tomasz Węgrzanowski has suggested. The idea is that your typical movie tends to have a single peak at its mode, and the percentage of votes will drop off monotonically on both sides of that peak. Movies that are "so bad it's good," on the other hand, will have a bimodal distribution, with lots of high scores (9's and 10's) and lots of low scores (1's and 2's), and few in-between.

For example, The Hunt For Red October is a fairly standard movie, and you can see that it does show a bell-shaped trend, albeit with a ceiling effect:

So what does The Room's rating distribution look like? Frighteningly bimodal:

Eventually we started watching the movie. It is truly disgustingly bad, and in fact it's hard to even call it a movie, as the plot seems to be just a cheap excuse for softcore porn. Infamously, they seem to use the same sex scene twice, although the director vehemently denies this. Whatever. It's hard for me to evaluate whether the movie is "so bad it's good," but the above ratings speak for themselves.

Bottom line: It's a better sign if a movie is rated higher rather than lower. But, holding average rating constant, you should prefer movies with a wider distribution of votes, as they will tend to be more interesting.