Against The Word "Scientist"

After seeing Dan Ariely's rant about how unwilling companies are to test things empirically, this ngram showing that "natural philosopher" used to be the term describing one who did experiments, Aaron Haspel's slightly too profound tweet that "what scientists say is not science" (that should be obvious), Katja's exposition of how schools fail to teach science in a way that students can actually apply to their lives, and unconsciously synthesizing all of this info at some point during its consolidation from short-term memory in the hippocampus into long-term memory in the cortex, I have come to the opinion that the word "scientist" should not be used to describe any people or profession. Instead it seems better to refer to those who work in science labs as "researchers" or more specifically by their specialty, like immunologist. If the research community, and by that I refer to the group of people who fantasize daily of having a paper published in Nature, were to stop monopolizing the term "scientist," perhaps others would be more apt to employ the scientific method in their own endeavors, to the benefit of all.

Why Not Uniqueness?

First, check out the comments on my recent post about uniqueness, in which I was (rightfully) ripped to shreds. It is wonderful to have such a critical readership! What Brian's critique suggests is that humans, when they want to be unique (which is often), tend to do so in a particular way that most other animals do not. That is, humans are more likely to share major beliefs with others in their immediate group, in order to signal their loyalty, and instead try to differentiate their group from other groups. He's mostly right and I was lazy to write that in the first place. Also in the comments, Justin asks "what's so bad about uniqueness"? Bingo responds basically as I would (must be in the DNA), but in the next three paragraphs I'll expand a bit on that. 

Betting on horse races is systematically biased such that longshots are overbet and the favorites are underbet. There are many explanations for this, but I think the best is that the bettors, who are typically also spectators, prefer to have their "own horse" that they in particular can root for. Similarly, participants in the intellectual sphere like to have their own "horse" in any ongoing subject of conversation, in part because it allows them to signal their uniqueness, and in part because it is simply more fun (these two parts are probably correlated). This implies that the marketplace of ideas, much like the horse betting market, is likewise systematically biased; "favorites" or obvious ideas are not valued as highly as they ought to be. So the key reason intellectual uniqueness (or intellectual hipsterdom) is bad is because it diffuses our focus from the most important questions and problems.

One key question: is informing people about this phenomenon likely to change their behavior? Probably not. Think of hipsters. Every hipster knows that hipsters are generally annoying, and most hipsters are personally annoyed by other hipsters, but this stops almost no one from acting in their self-interest and being hipster-ish themselves. So this is probably a dead end.

What we can do is make rating systems that quantify and rank, as precisely as possible, the various options in each of the idea spheres. This way people can talk all they like about their particular "horses" in the intellectual marketplace, but when it comes time to make important decisions, we can rely upon the ratings rather than these uniqueness-biased discussions. (A prediction market is an example of a rating system, and when possible to set up these tend to perform the best. But they are not always feasible to set up, so we should consider other options too.) Figuring out how to set up rating systems to answer important questions, and how to incentivize idea consumers to actually rate things in such systems, seem like key questions, and they are questions this blog will continue to track closely.

Hanson To Fellow Bloggers: Do As I Say, Not As I Do

Robin Hanson responds to Vladimir M's post about when one should or should not trust an academic consensus. First a bit from Vladimir,      

When looking for information about some area outside of one’s expertise, it is usually a good idea to first ask what academic scholarship has to say... [I]t would be astonishing if there didn’t exist at least some areas where the academic mainstream is detached from reality on important issues... Areas affected by ideological biases... tend to drift much further into outright delusion, possibly lacking a sound core body of scholarship altogether... I’ll have to read much more on the technical background of these subjects before I can form any reliable opinion on these questions. 

Then Robin, 

But if you plan to mostly ignore the experts and base your beliefs on your own analysis, you need to not only assume that ideological bias has so polluted the experts as to make them nearly worthless, but you also need to assume that you are mostly immune from such problems!...  If ideology severely compromises others’ analysis on this subject, then most likely it severely comprises yours as well.  You should mostly just avoid having opinions on the subject.  But if you must have reliable opinions, average expert opinions are probably still your best bet.

The conundrum is that Robin himself does this all the time. See for example this post literally two days earlier, where he is skeptical of an academic consensus because he has reason to believe that the authors are biased, does some digging on his own from the raw data, and draws some interesting conclusions. That's exactly what Vladimir is saying one ought to do.

So Robin's disagreement seems a bit inconsistent, but it's really not. He has admitted that he doesn't specialize as much as he thinks people should. Specialization requires trusting the surface arguments in most fields in order to focus on one's own. In other words, he thinks people should generally not be contrarian, if only because it's a waste of time. Consensus definitions of hypocrisy emphasize that it's OK (i.e., not hypocritical) to preach temperance for others when you yourself have not conquered your passions, as long as you do not claim to have done so. So Robin is not being hypocritical.

Still, it might be nice if he admitted to often not following his own "don't be a contrarian" rule.

Trade Off #18: Speed vs Accuracy

As you decrease the time in which you complete a task, you must give up some of the control which allows you to ensure that each part of the process goes as desired. This trade off between speed and accuracy is very widespread and is one of the most intuitive trade offs in the canon.

You can test yourself on this trade off by going to any typing test website and varying the speed with which you type. At more words per minute you will tend to have more misspelled words, while at fewer wpm you will have fewer misspelled words. Other examples:

• Watson, the AI created by IBM to compete at Jeopardy, faces a speed vs accuracy trade-off in that the longer it has to search its databases and run its algorithms, the more confidently it can assign a high probability to one of its answers. (see here)
• If you take longer to respond to emails, people will expect your responses to be more accurate, whereas if respond right away, their expectations will be much lower. This is because they intuitively recognize the trade off you face between speed and accuracy. (see here)
• In the world memory championships, as contestants take longer to look at a card, their mental representation of it become richer, making subsequent recall more accurate. So they must race to make it interesting. (see here)

This trade-off seems to not apply everywhere, in that people's predictions about soccer games may actually decrease in accuracy with more conscious attention. But this is probably because people tend to over fit their models to the data with too much conscious processing, an example of the precision vs simplicity trade-off.

To say that there is a trade-off doesn't completely preclude free lunches. Ray Allen can shoot 3's more accurately than I can, and he can shoot them faster. The key is that if Allen were to shoot basketballs at 2x his own comfortable speed, he'd no doubt be less accurate. So, with respect to just this speed vs accuracy trade off, the amount he has practiced seems like a free lunch, but we know that once we consider other trade-offs, it is not.

(photo credit to flickr user Odd; the slower the sun sets, the more accurately you can predict the exact moment it will cross the horizon)

Squirrel Uniqueness

Kimberly Pollard and Daniel Blumstein recently published a study (HT Ed Yong) examining whether group size correlates with the individuality of squirrel alarm calls:

Discriminating among individuals is a critical social behavior in humans and many other animals and is often required for offspring and mate recognition, territorial or coalitional behaviors, signaler reliability assessment, and social hierarchies. Being individually discriminated is more difficult in larger groups, and large group size may select for increased individuality–signature information–in social signals, to facilitate discrimination. Small-scale studies suggest that more social species have greater individuality in their social signals, such as contact calls....

[W]e show a strong positive evolutionary link between social group size in sciurid rodents and individuality in their social alarm calls. Social group size explained over 88% of the variation in vocal individuality in phylogenetic independent contrasts. Species living in larger groups, but not in more complex groups, had more signature information in their calls.

Extrapolating not unjustly from this, it seems that humans, the most social of all animals, should have the strongest drive towards uniqueness. So the desire for your own opinion, as opposed to just the best one, is probably an evolved tendency. We should all be similarly troubled by this.

In Praise Of The Obvious

Ryan Holiday recently wrote a post encouraging us, in a roundabout way, to seek the counter-intuitive. On the contrary, I think the counter-intuitive is overrated. Consider:

• It is very difficult to find an RSS feed that only lists the basic, most important findings from any particular field, even though this would be the most useful to an outsider. 
• Inception is unlikely to win the Oscar for best picture (currently 80 to 1), even though it has by far the highest weighted average rating of any movie this year. 
• Academics love to conduct and publish paradigm-shifting research, but are much less interested in replications or confirmations. This leads to lots of problems.

The best way to explain these conundrums is to admit that we love the counter-intuitive, we love to associate ourselves with it, and we love to demonstrate our uniqueness in doing so. Far from encouraging these tendencies, we should instead shun them. If the obvious thing is best, we must endorse it just as strongly, or we risk sending confusing signals to others.

Granted, there are some trade offs here, mainly between finding new ideas and reducing uncertainty about our current ones (an example of the more general breadth vs depth trade off). That is, if we only focused on what was currently obvious, we wouldn't come up with any new angles. But, at the margins of our current intellectual climate, it still seems to me that we should prefer more obviousness.

Frustratingly, all else equal, counter-intuitive thoughts are a good sign that somebody has a deep understanding of a subject and has thought it through beyond the status quo. So if we really want to encourage obviousness, we need to value the traits that lead to it, like formality and diligence, even more so than we value traits that lead to novel ideas, like creativity. As Ice T says, it's not about being mad about everything, it's about being really mad at the right things.

So Bad It's Good, On imdb

My roommate was recently trying to convince me to watch The Room, employing the argument that, although it is poorly rated at a 3.2, it is "so bad it's good." The question is, can we quantify this?

The intuitive way seems to be to look at the distribution of scores, and indeed that is what Tomasz Węgrzanowski has suggested. The idea is that your typical movie tends to have a single peak at its mode, and the percentage of votes will drop off monotonically on both sides of that peak. Movies that are "so bad it's good," on the other hand, will have a bimodal distribution, with lots of high scores (9's and 10's) and lots of low scores (1's and 2's), and few in-between.

For example, The Hunt For Red October is a fairly standard movie, and you can see that it does show a bell-shaped trend, albeit with a ceiling effect:

So what does The Room's rating distribution look like? Frighteningly bimodal:

Eventually we started watching the movie. It is truly disgustingly bad, and in fact it's hard to even call it a movie, as the plot seems to be just a cheap excuse for softcore porn. Infamously, they seem to use the same sex scene twice, although the director vehemently denies this. Whatever. It's hard for me to evaluate whether the movie is "so bad it's good," but the above ratings speak for themselves.

Bottom line: It's a better sign if a movie is rated higher rather than lower. But, holding average rating constant, you should prefer movies with a wider distribution of votes, as they will tend to be more interesting.

Kobe In Crunch Time: Efficiency vs Predictability

Henry Abbott breaks it down, showing that Kobe isn't that legit in late game scenarios, despite popular opinion, and his own. But here's what Abbott is missing. Kobe probably would have the highest probability of making the last shot of any player in the NBA, if defenses paid equal attention to everyone. Kobe gets his rep for what he could do in this situation, and that matters for something.

The problem is that defenses adjust for this ability. "Doubling-teaming" isn't always a binary yes or no--it's also about being more apt to help off your man in case of a drive, hedge on off-ball screens, and keep Kobe in your plane of vision. Of course, all of this hinders how well the defense can guard Kobe's teammates. So what Kobe should do is look for them. But he doesn't, because he gets too caught up in thinking about how efficient he can be, and forgets that this makes him predictable.

This is a pretty clear demonstration of the trade off. Crunch time offense is a game, and in any game you want a mixed strategy. That means that with some probability you'll do one thing, and with some probability you'll do something else. If you're the Lakers and you commit to Kobe always shooting, or even assign it a higher probability than you should, the other team will be able to take advantage.

Towards A More Risk Loving imdb

In my view, the "true" rating of a movie is what the average opinion would be if everybody who watches it:

• has some basic knowledge of art and human affairs in general (for example, a working knowledge of the canon of classics), but no knowledge about the movie in particular (no previews, hype, etc);
• is watching alone, so nobody in the theater is laughing at unfunny times;
• experiences neither hunger, thirst, polyuria, tiredness, stress, excessive marijuana-induced paranoia, nor any inclination to rub tongues with the person next to em;
• watches on a ridiculously large screen with speaker cables made of pure silver.

Since the above conditions will never all hold, we will all be biased in one or another way when we watch a movie. So we must be wary of the opinion of any given rater. That is, if the first person to watch a movie gives it a 10/10, our estimate for the true rating of a movie should not be a 10, but instead should be adjusted down towards the average.

The above is all obvious. What's less obvious is that there's no easy way to decide how much one should scale down the rating. That depends on how much of a risk you're willing to take with low sample sizes.

Consider The Passion of Joan of Ark, whose 8.3 rating should be enough to place fairly high on the top 250. For example, Sin City also has an 8.3 and it's currently #104. However, TPoJoA only ranks #210, because its paltry 11k votes push down its score so much.

Here's what I'm proposing. Let us, the users, choose our own scaling parameter. Let us define how much of a risk we want to take in trusting smaller sample sizes. Let us choose our own destinies. Because the current system smacks of hegemony, and I, for one, will only stand for it because I have better things to do.

Hanson On Secular Vs Sacred

He writes:

Ask yourself this simple question: how confident would you need to be on a moral or political conclusion in order to work passionately for it? 99%? 90%? 75%? If you have such an action-threshold, and this threshold is high, well then yes, to let your passion flower, you may need to lie to yourself about your confidence. So that you might actually do something. Would your overconfidence then lead you to do too many things too enthusiastically? If so, perhaps you’d do better to also allow yourself some other more graded psychological reluctance to passion, to counter this bias.

But it would of course be even better if you could see the nobility and glory in doing your best as a limited but well-meaning creature. You shouldn’t need to be absolutely sure of a conclusion to work sincerely and passionately for it.

The emphasis is mine. In the first two paragraphs he articulates the trade off. Making an assumption, which is assigning something a higher prob of being true than you have evidence for, has the upside of allowing you to actually do things. But making an assumption also has the downside of distorting reality (e.g., making you overconfident), so we want ways to protect against too many assumptions.

In the last paragraph, he says that we might circumvent the trade off by admitting that we might be wrong but pushing forward just as hard regardless. I doubt that's possible. Our brains are constantly computing probabilities and using those probabilities to determine policy implications, just as in Bayesian model averaging. So if we downshift the probability that an assumption is true, its relative policy implications also have to fall. There's a reason that overconfident CEOs are more innovative, and people wild enough to be entrepreneurs are typically overconfident. All else equal, if you have less faith in your conclusions, you will work with at least slightly less passion towards them.

The above applies at any given time point. Now, you might be able to trade off some self-awareness for action now, in order to achieve more self-awareness later. This is sort of like what people do when they "build a slave." But that involves a different trade off; no free lunches here.

Trade Off #17: Secular vs Sacred

Ever wanted to rid yourself of all assumptions and reason your way to a full, coherent philosophy? You're not alone.

Rene Descartes made the most famous attempt to do so, but by most contemporary accounts he failed miserably. And in truth, if someone ever did succeed in assuming nothing, we'd probably never hear from em again.

That's because assumptions are necessary for action. They are like the ground, without which you can't propel yourself forward. Assumptions come with a price, though, which is that they can distort reality. (The ground distorts reality, too. Relative to the earth's surface our position appears to be constant, even though the earth is actually hurtling through spacetime at ~ 30 km / s.)

So although you'd prefer to assume as little as possible, you must make some assumptions if you want to get anything done. The trade off is in deciding where precisely to draw this line. Examples:

• Euclid's fifth assumption was the parallel postulate, that two perpendicular lines extended indefinitely will never meet. Making this assumption enabled him to prove many useful geometrical theorems, but it also obscured some other real ways the universe can behave, which later mathematicians would explore. (see here)
• Psychologically, we humans all have a certain amount of anxiety, regarding our place in the universe and our identity. Some believe that everyone has a "dogma quota," which they must assign somewhere, to ethics or empirics or whatever, to overcome this anxiety and function in society. (see here)
• In Kuhn's conception of science, the paradigm is the set of all assumptions that allows scientists to solve puzzles and make advances. But it comes at the cost of institutionalizing the assumptions. So even when the reality-distorting effects of the assumptions become obvious, the institutions supporting these assumptions are difficult to overthrow and require mini-revolutions. (see here)

Mathematicians often make broad assumptions when they first prove a theorem, and then someone else will come along later and "generalize" that theorem by coming to the same conclusions with fewer assumptions. Isn't this a violation of the trade off, because the later proof takes the same "action" with fewer assumptions?

Sort of, and it's true that most people make stronger assumptions than they have to. But at the base level certain assumptions are truly essential. For centuries bright minds tried to generalize Euclid's parallel postulate, to prove the fifth assumption from the first four, with no success. It is this "rate-limiting" case where assumptions are essential that this trade-off describes.

We're a bit worried about this trade off conflicting with precision vs simplicity. From a stats perspective, maybe it's best to think of precision vs simplicity as referring explicitly to the number of parameters in the model, while this trade off refers to the form of the model, including the act of modeling itself. But possibly they can and should be united.

(kudos to Denis Collette for the nice photo)

Ed Yong On Loyalty vs Universality

He explains a study that measured the effects of oxytocin on a number of behavioral correlates in males:

[O]xytocin strengthened the biases of the Dutch volunteers. When they sniffed oxytocin (rather than the placebo), they were quicker to associate positive words with Dutch names than with either German or Arab ones....  [A]fter sniffing oxytocin, they were far less likely to sacrifice the Dutch loners than the German and Arab ones....

This last experiment clearly shows a trend that applied to the whole study: oxytocin boosted favouritism for people who belong to the same group. Only very rarely did it increase negative feelings towards people outside it. For example, in the moral dilemmas, oxytocin made the volunteers less likely to sacrifice members of their own group, but not more likely to sacrifice outsiders.

This sort of favouritism makes a degree of evolutionary sense. It could bolster trust and cooperation within a community, such that groups whose members stuck together more would out-compete those that did not.... But such preferential treatment has an obvious dark side – it leads to all sorts of moral and cultural problems, including inequality, discrimination, prejudice, and conflicts between different groups.

Fascinating, but I'm not sure we need any group selectionism to understand this. It seems from the above that oxytocin nudges humans towards in-group loyalty on the trade-off between loyalty and universality. So in order to increase the fitness of individuals, oxytocin levels should tend to rise in situations when signaling loyalty is adaptive, and fall when signaling universality is adaptive.

Hmmm... do you think it is more adaptive to signal loyalty or universality during and immediately after sex? Call me prude, but I doubt you'd boost pair bonding by telling someone that you love them, but you love everyone else just as much. Then again, I s'pose that's an empirical question.

What Tech Trades Off

Ignoring his sometimes outlandish claims, Kevin Kelly's book What Technology Wants has much wisdom to dispense. At its core, the book describes the trade-off between protection and freedom, with respect to a society's level of tech use.

At one extreme, a society can minimize its adoption of technology, protecting its members from potential dangers, but limiting their choices. The Amish are an example of such prudishness.

At the other extreme, a society can maximize its use of technology, allowing its members the most possible choices, but limiting its ability to protect people from the potential dangers of tech. Is anyone so audacious? Kelly doesn't offer examples, but presumably some transhumanists would fit here.

Some benefits of protection from technology are:

• The opportunity to nudge people towards choices they might consider more desirable upon sober reflection. (e.g., drug regulation, exploitation in compensated medical trials)
• Reduced existential risk, if a particular tech could wipe humans out, and restricting tech development will decrease that probability. (controversial, but nuclear winter, bacteria with reversed chirality, exponentially self-improving unfriendly AI, runaway anthropogenic global warming, etc., could all fit)

Some benefits of freedom in technology are:

• An increased ability for people to express their unique preferences and match their aptitudes to their environment. (e.g., transportation tech allows people to move away from the place they were born, if they find it disagreeable)
• More rapid feedback via proxies for natural phenomena enables more experimentation, which increases knowledge. (e.g., changes in blood pressure allow you to gauge heart attack risk, so you can measure the effect of a given treatment before you actually die)

Most everyone agrees that by increasing the exposure to tech, you broadly accelerate tech development. As a counter example, think of the Amish, who sometimes free-ride on old tech, but provide few novel inventions themselves. This positive loop makes the consequences of the above trade-off more pressing.

In the long-run, it's hard to see how tech adopters wouldn't out-compete non-tech adopters. This is why the expansion of tech is in some senses "inevitable." But evolution requires replication, and nerds marry late and have few if any kids, so who knows.

(Thanks to the insightful Tyson B for a convo about this)

No Shame Gossip

Here's Jonathon Haidt in his nugget-rich Happiness Hypothesis:

Gossip is overwhelmingly critical, and it is primarily about the moral and social violations of others. (For college students, this meant a lot of talk about the sexuality, cleanliness, and drinking habits of their friends and roommates.) People do occasionally tell stories about the good deeds of others, but such stories are only one tenth as common as stories about transgressions. When people pass along high-quality ("juicy") gossip, they feel more powerful, they have a better shared sense of what is right and what's wrong, and they feel more closely connected to their gossip partners....

[M]ost people hold negative views of gossip and gossipers, even though almost everyone gossips.... [We] came to believe that gossip is underappreciated. In a world with no gossip, people would not get away with murder but they would get away with a trail of rude, selfish, and antisocial acts, often oblivious to their own violations. Gossip extends our moral-emotional toolkit.... Without it, there would be chaos and ignorance. 

Based on the above, it seems like not gossiping would be selfish. By abstaining, you save yourself from feelings of guilt, and you earn a reputation as reserved and trustworthy, if boring. Yet if nobody gossiped, we'd all be much worse off. This can still be true even if the marginal piece of gossip is harmful, and is another example of why categorical imperatives ("don't gossip" has 1.25 million google hits) are often flawed.