Hindsight Is 2010

The NYTM's Year in Ideas is consistently good. My favorites for this edition were D.I.Y. Macro, Performance Enhancing Shoes, Relaxation Drinks, and The 2000's Were a Great Decade. Inspired by their ideas, my retrospective for the year will consist of twelve articles / blog posts, one for each month, that seem especially representative of the year in ideas.

January: "Lessons from a pandemic," Nature editorial, 685 words. The H1N1 virus ended up not being that lethal, but it could have been, and this article highlights the lessons. Among them is that six months is too long of a time for vaccine production, given that viruses now easily spread around the world in "a matter of weeks."

February: "The biomechanics of barefoot running," editor's summary, 283 words. Running on one's toes is healthier than running on one's heels, even though most running shoes promote the latter. This finding is largely academic for me personally, as over the years I have come to loathe jogging. Nevertheless, it is emblematic of the larger "back to nature" craze that has taken over in 2010. This includes the paleo diet, probiotics, and restroom posture designed to prevent hemorrhoids.

March: "Snake oil? The scientific evidence for health supplements," by David McCandless and Andy Perkins, infographic. Aside from being fascinating, this is a good example of an effort to harness the academic lit for the benefits of the masses. Also, it is representative of the open data movement, as the authors transparently aggregate their data set in a google doc, which anyone can view.

April: "The data-driven life," by Gary Wolf, 5808 words. Discusses the growing trend of self-experimentation. More generally, he discusses how many more people are using tech and data to inform decisions, trumping their raw intuition.

May: "The moral life of babies," by Paul Bloom, 6026 words. He discusses how our preference towards actors who "do the right thing" emerges very early. That is, it presumably emerges far earlier than the babies would be cogent enough to consciously reason about morality. This is part of a movement in psychology that is emphasizing the arbitrariness of our beliefs and decisions.

June: "Smarter than you think: IBM's supercomputer to challenge 'Jeopardy!' champions," by Clive Thompson, 6609 words. At any given point, the AI iteratively calculates the probability that an answer is correct, and then checks whether that probability passes a certain threshold. This probabilistic thinking seems to be invading fields beyond just machine learning, so it's important to understand.

July: "New developments in AI," by Steve Steinberg, 5496 words. An innocuous and perhaps unfortunate title, but a tour de force of a blog post. He discusses trends in smart cars and massive knowledge-bases, and speculates on how they will affect society. One sentence that's particularly near and dear to my heart is when he writes, "consider that 'what is the best burrito in SF' (an opinion), and 'what do most people consider the best burrito in SF' (a fact) are normally considered equivalent."

August: "A world without mosquitoes" by Janet Fang, 1929 words. She discusses whether we should try to eliminate all of these nasty, virulent insects. The downside is that it would mess with biodiversity in ways difficult to predict, while the upside is that it could save millions of lives. We will face plenty of these type of trade-offs in the coming years, specifically with respect to climate change and geoengineering, and more generally in changing aspects of our natural world that we disapprove of.

September: "Jumping to joy," by Robin Hanson, 212 words. He wonders whether we should experiment more with different lifestyles, and what our failure to do so implies about our precarious sense of self. Questioning which of our selves is the "real" one is trendy these days, boosted in part by things like the implicit association test. Experimentation is also enjoying a resurgence, championed by Dan Ariely.

October: "Lies, damned lies, and medical science", by David Freedman, 6022 words. Explains the problems with current scientific publication and data dissemination systems. Many scientists broadly agree with these critiques of their infrastructure, but lack personal incentives to change them. 

Movember: "Hangover theory and morality plays," by Steve Waldman, 1986 words. He discusses the need to frame causes of the recession in moral terms that anyone can understand, synthesizes relevant economic theories, and holds no punches. It'd be hard to describe the ideas of 2010 without including reactions to the recession.

December: "The hazards of nerd supremacy: The case of Wikileaks," by Jaron Lanier, 4704 words. Wikileaks is one of the defining stories of the year. He explains that we might support the hackers in our hearts, because we perceive them to be the underdogs, but that in our heads we should be much more skeptical.

It's been a fun year of blogging and thanks as always for reading.

Lehrer On Plasicity Vs Specialization

He discusses it here, a month ago:

Expertise might also come with a dark side, as all those learned patterns make it harder for us to integrate wholly new knowledge.  Consider a recent paper that investigated the mnemonic performance of London taxi drivers. In the world of neuroscience, London cabbies are best known for their demonstration of structural plasticity in the hippocampus, a brain area devoted (in part) to spatial memory....

The problem with our cognitive chunks is that they’re fully formed – an inflexible pattern we impose on the world – which means they tend to be resistant to sudden changes, such as a street detour in central London....

The larger lesson is that the brain is a deeply constrained thinking machine, full of cognitive tradeoffs and zero-sum constraints. Those chess professionals and London cabbies can perform seemingly superhuman mental feats, as they chunk their world into memorable patterns. However, those same talents make them bad at seeing beyond their chunks, at making sense of games and places they can’t easily understand.

A beautiful exposition. However, I think the trade-off can be found more generally than in just the human brain. Indeed, most evolving biological systems impose limits on plasticity because of the costs. This suggests new minds or systems we might design will probably deal with this trade-off too. But this is all still hotly debated.

Douglas Adams On Robustness Vs Fragility

From The Hitchhiker's Guide:

"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair."

Aging And Happiness

How does subjective well-being vary with age? To find out, Stone et al (here, HT: The Brits (HT: BC)) conducted a large random-digit dial survey of 300,000+ US citizens. They asked about well-being and a few other variables, like age. We can only hope they had a good wireless plan. Here's the big result:

The covariates are unemployment, marital status, whether one has children living at home, and gender. Younger people increase in well-being ratings once you adjust for these because they're more likely to be unemployed. Of course be careful of the axes, as their real rating scale varies from 0 to 10. But the large sample size and continuous trend across age groups lends credence: I buy it.

What about anxiety and age? Here's the proportion of respondents who reported feeling "a lot" of stress the previous day, in different age groups:

I wonder what explains this trend. Perspective? Fiscal and emotional stability? Norepinephrine levels in the amygdala?

Finally, for those readers who do not trust their eyes, here's their table showing the percent of variances explained:

So anger and stress show a pretty consistent decline across age groups, while the curves are more U-shaped for measures of subjective well-being. Note sadness follows an inverted U-shaped curve.

Many top 250 movies explore this curious relationship between happiness and age, like Up, Ikiru, Cinema Paradiso, and The Wrestler. Much of it seems counter-intuitive. Here's a post of mine from '07 wondering whether we become happier with age, but I apparently didn't see the U-shaped curve coming.

Why Nature Didn't Choose Arsenic

Phosphate is of course a part of the structures of DNA and RNA, but it is also in many metabolic intermediates like ATP and glucose-6-phosphate. The idea that a bacterium could survive without it (see here) would require updating some concepts about the flexibility of reaction rates in physiological systems.

A 1986 paper by Frank Westheimer, cited 400+ times, abstract here and pdf here, explains why phosphates are preferred. In particular, he notes how the negative charge of the phosphate ester makes it relatively more resistant to hydrolysis, while it still can act as a leaving group if enzymatically activated. Then, in an intriguing section, he discusses why various other alternatives would not make sense, including arsenic:

Another compound that must be considered as a basis for a possible genetic material is arsenic acid, which is also tribasic. The poisonous effects, however, of compounds of arsenic probably cannot be avoided, since these effects are centered in the lower valence states of arsenic, and the reduction of pentavalent arsenic is much easier than that of pentavalent phosphorus. In any case, arsenic esters are totally unsuitable; the hydrolysis of esters of arsenic acid is remarkably fast. The triisopropyl ester in neutral water at room temperature is completely hydrolyzed in a couple of minutes. Apparently the hydrolysis of the diesters is even faster than that of the triesters.

The idea is that esters of arsenic are too liable to be cut by water, thus making them poor linkers for bases of DNA and RNA. But this is assuming that the reaction will occur under relatively stable conditions (i.e., pH and temp), and perhaps those conditions are somehow altered in this particular bacterium, sufficiently lowering the hydrolysis rate of arsenic. We will have to wait and see, but in the meantime where are the prediction markets when we need them?

Trade Off #16: Impulse vs Incentives

At one of the stoplights on my drive to the gym, there is often someone walking through the cars asking for change. Let's assume, reasonably, that she needs the money more than I do. Let's also assume, somewhat less reasonably, that she'll spend the money in a productive manner. We can now break the decision down into the benefits she'd gain from leveraging my money versus the perverse incentives I'd reinforce by rewarding people for begging in traffic.

Generally, this trade-off is between the benefits from an impulse meant to rapidly improve and stabilize a condition, versus the costs of long-term instabilities that could result. Some examples:

• Keynesian econ emphasizes the multiplier of a gov intervention, which they consider to be an impulse, whereas Austrian econ emphasizes the moral hazard (i.e., bad future incentives) of such impulses. ("in the long run we're all dead", also see here)
• Radiotherapy has a good chance of killing tumors, and thus can be thought of as an impulse. But it also makes mutations in other genes more likely, which could develop into secondary tumors in the future, and thus can be thought of as bad "incentives." (see here)
• WikiLeaks might incite people to speak out against or question their government, thus acting as an impulse to increase freedom, but it also incentives governments to be even more secretive and centralized, thus decreasing freedom. (see here)

The previous trade off we're most worried about conflicting with here is some now vs more later. And indeed, it is true that there is a time component to both of these trade offs. But the difference between them is that in this trade off the two possible outcomes are not quantitatively but instead qualitatively different.

Now, the previous paragraph is somewhat of a technicality, and is probably boring. But if I didn't mention it now, just imagine the kind of incentives that would introduce for sloppiness in the future.

(Kudos to Alan Grinberg for the photo)

Bill Simmons On Uniqueness

He gripes in his most recent chat:

[I]t's funny to take heat from soccer fans that I'm a bandwagon Tottenham fan. I mean... of course I am. I am something like 17 months into this thing. But what I don't get about sports like the UFC/soccer/NHL (and even baseball with the saber community towards people who just like baseball and don't want to dive into the stats) is why the diehards are so protective/condescending towards casual fans. What's the goal there? To just drive away everyone who might like the sport and want to become more of a fan? I think there's a difference between local bandwagon fans (like the Pink Hat Red Sox fans) and "I am starting to like your sport, I genuinely want to follow it and learn about it" fans and it would just seem like the diehards should embrace the latter group. Or am I crazy?...

I do think that diehard fans tend to exclude newcomers - the same phenomenon works with music, you always want your favorite band to be the little band that not everyone knows about (and never have them get to the U2 level). I think Kings of Leon are a good recent example of this and even the band members hated that they became "mainstream" because it brought in fans that they didn't necessarily want. The best breakdown of this was in Steve Martin's book about his standup career when he talks about becoming hugely famous and how he started dreading doing his shows because he felt like people weren't there for the right reasons. It's an interesting topic I think.

Once more people join a given group, affiliation loses lots of its signaling benefits due to diffusion. So it makes sense that fans who have invested in a team / band would discourage newbies from joining.

But why do the musicians / comedians themselves not want more followers? That's a bit trickier. Simmons mentions Steve Martin being wary of his new fans, but the same is even more true of Dave Chappelle. When he skipped out on the third season of his show, he turned down millions.

Any theory to explain this phenomenon also has to account for the fact that neither athletes nor academics tend to express these sentiments. Jordan, Manning, Hawkings, Volkow--they do not worry about "going mainstream." Indeed they tend to welcome it.

Perhaps the musicians / comedians are signaling loyalty to their core constituents, and the real emotional and financial costs they pay in doing so just makes their signaling more credible. So it seems that the less badly you want to go mainstream, the more your pursuit is about signaling as opposed to results.

Our Regrets Change Over Time

Ben Casnocha recently posted about the regrets of the dying, and Robin Hanson replied that in fact people on their deathbed do not spontaneously offer such regrets. Robin is technically correct, but he is taking the claim far too literally.

Indeed, there is broader data to suggest that the regrets of older people are quite different from the regrets of younger people. In particular, as time since a decision grows, people tend to shift their regrets towards not making the hedonistic decision.

An '06 study (link, pdf) shows how the intensity of regrets towards work or enjoyment changes as time passes. For events last week, people express slightly higher levels of regret towards enjoying instead of working (2.2 vs 2.0 out of 6). But for events five years ago, people feel more regret for working instead of enjoying (3.4 vs 1.4 out of 6). This change is even more pronounced for feelings of guilt vs missing out, and there's lots of replicating data (for example, see Ran Kivetz's other papers here).

This is part of what makes using the regret heuristic so complicated. One must not only project future regrets for a decision, weighted by the probability of each outcome, but also consider how those regrets might change in direction and strength over time, and integrate over all probabilistic future time points. If this computation were easy, there wouldn't be so much demand for strategies to get an approximate answer.

P-Value Polemics

As I am always up for a good scholarly debate, I was quite pleased, after reading this '05 article calling for a replacement to the p-value called p-rep (cited 200+ times), to see a somewhat vitriolic '09 rebuttal (pdf). First, the abstract of the '05 paper by P. Killeen:

"The statistic Prep estimates the probability of replicating an effect. It captures traditional publication criteria for signal-to-noise ratio, while avoiding parametric inference and the resulting Bayesian dilemma. In concert with effect size and replication intervals, Prep provides all of the information now used in evaluating research, while avoiding many of the pitfalls of traditional statistical inference."

A rather bold claim! And, shortly after its publication, the journal Psychological Science (6th highest psyc impact factor) recommended that authors report p-rep instead of the traditional p-value. Which makes the rebuttal article by Iverson et al that much more tantalizing. They write:

"This probability of replication prep seems new, exciting, and extremely useful. Despite appearances however prep is misnamed, commonly miscalculated even by its progenitors, misapplied outside a common but otherwise very narrow scope, and its seductively large values can be seriously misleading. In short, Psychological Science has bet on the wrong horse, and nothing but mischief will follow from its continued promotion of prep as a scientifically informative predictive probability of replicability."

Now that is what I call a take down! These same authors calm down quite a bit in their '10 article and even make the level-headed suggestion that p-rep is a step in the right direction, but that is uncool so I won't quote from it.

####

Reading about p-values makes me want to start a blog about them (how does such a blog not already exist?!). A good subtitle could be "where one in every twenty posts will be significant by chance alone."

Trade Off #15: Acquiring Info vs Altering Subject

Paradoxically, when you ask someone a question, their answer will likely be different because you've asked them. On reflection this phenomenon is remarkably general; in acquiring or assessing info about a property of a system, we must be willing to accept the consequences of altering that system. A smattering of examples follow: 

• Increasing the energy voltage in transmission electron microscopy can lead to higher image resolution (meaning more info), but it also does more damage to the tissue. (see here; there are similar trade offs in lots of med imaging techs, like PET scans, see here)
• When a model of a complex psychological phenomenon becomes widespread, reality often begins conforming to the model. This is often called performativity, and it is perhaps why Keynes called economics a "moral" science. (see here) 
• The Heisenberg uncertainty principle says that the more precisely the position of a particle is measured, the less precisely its momentum can be, and vice versa. The explanation for this is controversial, but it's likely due to observer effects--the measuring apparatus delivers a force to the particle which alters it. (see here)

Most examples that fit into this category are true to at least some degree, but it's often tricky to determine precisely how much variance they explain. For example, even when you run a placebo-controlled drug trial, you must be wary of side effects that can tip the participants off to their treatment status (see here). Regardless, this is a widespread trade off, and it now assumes its rightful place in the canon.

(photo is of a peptide fiber, taken with EM, credit to Christoph Meier)

We're All Individuals Now

Christian Jarrett explains how we tend to value our uniqueness, 

Whether it's a gift for small talk or a knack for arithmetic, many of us have something we feel we're particularly good at... this strength then becomes important for our self-esteem... children tend to choose friends who excel on different dimensions than themselves, presumably to protect their self-esteem from threat... [W]hen making hiring decisions, people tend to favour [sic] potential candidates who don't compete with their own particular strengths... Participants tricked into thinking they'd excelled at the maths [sic] tended to choose the potential team member who was weak at maths but stronger verbally, and vice versa for those participants fed false feedback indicating they'd excelled verbally. 

I think this drive can explain two anomalies:

1) We often agree to disagree. Aumann's classic disagreement theorem (pdf) says that rational truth-seekers cannot and will not agree to disagree; given the same priors and the same data, they must each reach the same conclusions. Cowen and Hanson (pdf) discuss how this disagreement result is quite robust. What's the deal? Disagreeing allows us to show off our independence and individual intelligence, which are among the most credible ways to establish uniqueness.

2) Our opinions oscillate in cycles away from what we perceive as the current consensus. A recent PLoS Comp Bio paper shows that in order to explain this "hype cycle" you need to model the preference for individuals to feel unique:

[W]e identify a missing ingredient that helps to fill this gap: the striving for uniqueness. Besides being influenced by their social environment, individuals also show a desire to hold a unique opinion. Thus, when too many other members of the population hold a similar opinion, individuals tend to adopt an opinion that distinguishes them from others....

[T]here is a third, pluralistic clustering phase, in which individualization prevents overall consensus, but at the same time, social influence can still prevent extreme individualism. The interplay between integrating and disintegrating forces leads to a plurality of opinions, while metastable subgroups occur, within which individuals find a local consensus. Individuals may identify with such subgroups and develop long-lasting social relationships with similar others.

There is a trade-off between uniqueness and accuracy in beliefs, so we typically seek uniqueness on opinions for which the cost of being wrong is low: art, politics, sports, etc. That's why there is less of drive for uniqueness regarding med. Few would claim that covering an open wound is a bad idea.

Before you say, "but I'm not an individual!", you need to check out this classic Monty Python scene.

Trust The Ratings Of Others

An '09 paper (link here, pdf here, HT to TC) claims that people make more accurate emotional predictions about a future event when they are simply told how someone else reacted to that event, as opposed to when they are given info about the event. This is somewhat counter-intuitive, so let's look at their evidence.

One of their tests was speed dating. Each guy submitted a photo and some demographic info about himself. Then, each girl predicted how much she would enjoy the date based on either this photo / info or the enjoyment rating of a girl who earlier had gone on a date with the same guy. Next, the guy and girl had their five minute date, (ignore the heteronormativity, my fellow Vassar alums), and finally the girl rated how much she enjoyed herself on a sliding scale of 1 - 100. 

The authors define prediction error as the difference between the girl's predicted and actual enjoyment ratings. Participants made more accurate predictions when they used the first girl's enjoyment rating to predict their own (the avg error was 11.4 +/- 8.7) than when they predicted their enjoyment on the basis of the info (an avg error of 22.4 +/- 10.8).

In classic psyc study fashion, they also asked their participants to say which condition they thought would lead to more accurate predictions. 75% said the info would be more useful than the rating of a girl who had already been on a date with that guy. Oops. Now, indulge me in a few reactions:

1) Why do people underestimate the value of someone else's rating? Probably because people think of themselves and their opinions as more unique than they actually are. This sets up my public choice theory for why popular critics like Anthony Lane tend to be negative and contrarian. Although on the surface this annoys readers, people on a deeper level prefer to read opinions about art that they disagree with, because it allows them to think of their own opinions as more unique.

2) There is only one specified relationship between the study participants: they are all undergrads at the same school. So although the authors toss the word "social network" in towards the end of the paper, their results do not speak to the predictive power of a friend's opinion as opposed to a stranger's opinion. This remains an open question--in predicting their own enjoyment, will people find the opinion of someone in their network more valuable than the average opinion of strangers? Even if you say yes, you must take into account the trade-off of sample size, which is larger when you listen to the masses. The high valuation of sites like facebook relies in large part on the assumption that we'll prefer recommendations from those in our network, but I'm not so sure.

3) A subsequent study looked at how people combine their own mental simulations and third-person reports of other's experiences in making judgments. Corroborating the results of this study, they found that people assign far too much weight to their own simulation of how an event will play out as opposed to the feelings of other people who have actually experienced the event. I myself find this all very relevant to imdb's movie ratings. Remind me again why you trust yourself to judge a movie instead of deferring to the aggregated ratings of others?

Making Me Guess

It is often tricky when you are asked to estimate some odd quantity, like how many people dressed in panda suits there were at a party. The person asking you to guess is usually hoping to surprise you with the large deviation of your guess ("what you'd expect") and the actual quantity. But you, the guesser, can't help but take into account the fact that they are asking you, which they presumably wouldn't be doing unless the quantity to be guessed were extremely unlikely and otherwise surprising.

So, the supposedly naive guesser is forced to either guess especially high and annoy the asker, or guess low and appease the asker. Guessing high makes you look like a smartass, while guessing low is tedious.

As a young lad, I would never really know how to choose between guessing high and low, and the choice often brought on some anxiety. But now I usually just explain the whole situation, indicating how being asked to guess has instantaneously shifted my probability distribution. Incidentally, I no longer have any friends. 

(Inspired by a representative convo with Brittany)

Trade Off #14: Precision vs Simplicity

When you describe something, the more precisely your model explains the given data, the more complicated it must be. Don't believe me? Lo, behold these examples, then:

• In describing the path up to my apartment, I could say "there are stairs", or I could say "there are fourteen stairs"; vagueness is less precise but it is also simpler. The bottom line is that having to walk up any number of stairs is too many.
• In fitting a model to data, one can explain more variance by including more free variables, at the cost of complication. There are plenty of ways to punish a model for having additional parameters and thus make the model earn each of its parameters through explanatory ability. (see here and here)
• The failure of humans to adequately trade off precision and simplicity in certain contexts, like when we say that the prob of X and Y is greater than the prob of just X, is one of our well-documented cognitive biases. (see here)

There are some well-known incidents in the history of science that on first glance appear to be exceptions to this trade-off. For example, Kepler's idea of elliptical planetary orbits eliminated the need for astronomers to model extra epicycles, both simplifying and adding precision to our understanding of planetary motion.

But in the view of this committee, these precision-enabling paradigm shifts are especially complicated, involve the shifting of assumptions at a fundamental level, and only seem simple in distant hindsight. That's one reason why they are so hard to come upon.

(photo of spiral galaxy, which Johannes Kepler probably would have marveled at, goes to NASA's Marshall Center)