"It Teaches You How To Think"

Many pursuits apparently fall under this purview: programming, economics, history, philosophy, english, physics, math (often called "mental weightlifting"), sociology, classics, engineering, college (esp. the liberal arts), grad school, law school, medical school, business school, multi-level marketing, neuro-linguistic programming, and chess, to name a few.

I'm not at all opposed to this idea and I do favor "raising the sanity waterline." But I'd like to see the claims evaluated more systematically against 1) how beneficial the most common alternative activity that the median person who would otherwise invest in these pursuits would be and 2) how beneficial the most useful alternative activity that the median person who would otherwise invest in these subjects could be.

Surely there is not enough time, given our current healthspans, to invest ourselves in all or even most of these activities. So, for those of us who do value good "thinking" skills, how should we choose?

(The above link anecdotes do not prove much, as it is possible to make nearly any point with a little google-fu, but I have read and heard this sort of statement repeated over and over to the point that it seems to me to be nigh-gospel. If anyone knows of a systematic way to determine whether these are commonly held beliefs, I'm all ears.)

The Best Movies Of 2011

In reverse order: 5) Hugo (not bad, but fragmented and too long, e.g. they could have cut out the station inspector storyline, which relied upon sadly hackneyed deterministic developmental psychology), 4) HP7P2 (technically stunning, but Radcliffe just doesn't do it for me), 3) Drive (I do sympathize with the "too violent" critiques, but I loved the pacing), 2) Warrior (I'm slacking), and 1) A Separation (still not out in my area).

All in all this has been a down year for offerings on the big screen. Currently there are only 5 movies in the top 250, and that number will surely drop as they come out on video and the attrition of time does its damage. I predict only 3 movies from this year will still remain on the list come this time next year. Compare that to 7 for 2009 and 8 for 2010. Hollywood needs to stop investing so heavily in sequels!

Six Thoughts On Rewatchability

1) Let's consider movies that are driven at least in part by plot uncertainty. Watching such a movie a second time is by no means necessarily worse (e.g., many enjoy Inception even more the second time through), but it is different.

For example, watching a movie could really force you to question some of your beliefs and adjust your corresponding models of the world. This could be a very worthwhile experience, yet you could quite reasonably expect that watching the movie again would be unlikely to yield the same reward.

In such a case, it'd be fair to say that the experience you derived from that particular movie (i.e., watching it and thinking about it after) was better than the experience you derived from any other, yet you still might not especially want to watch it a second time.

2) Touching the Void is an example of such a movie, for me. Watching it meant a lot to me and I have thought about it often since doing so, but I wouldn't want to watch it again in the near future.

3) I'd agree that, all things equal, wanting to re-watch a movie is a proxy for how much you got out of it. But I don't think people should have to "prove" that they like a movie by watching it over and over.

4) And further, I don't think the metric is all that good of a proxy. I doubt that there'd be a very high correlation between the movies with the highest average subjective experience rating and the ones that people watch over and over.

This is because I expect people ultimately get the most out of movies that challenge them, and in the moment they are less likely to choose to re-watch such a movie, because of the insidious effects of delay discounting.

5) The counter is that people are too likely to be signaling when they merely say what their favorite movie is, which is why we need some measure of how people actually behave. I do sympathize with this argument.

For example, people tend to watch a lot of comedies, yet there are few comedies in the top 250 and actors in comedies hardly ever win awards. Yes, some of this is because humor does not cut across cultures well. But surely on some level comedies are underrated because liking them does not allow us to as effectively signal our sophistication.

6) The main upside to any behavioral metric is that it slices through the noisy opinion market. The downsides are that we might not be measuring what we think we are, and, to the extent that the metric is widely held, it can lead to costly gaming.

Full Disclosure: When a movie that I have seen more than once is brought up in conversation, I will almost always brag about how many times I have seen it, to collect some street cred.

Disease Olympics

This is my favorite new phrase, which I first heard in this article by Carl Bialik about the ethics of using sometimes tenuous statistics to rouse advocacy for diseases. It is used to refer to situations in which different maladies "compete" against one another for funding, attention, empathy, and the like.

As Otis Brawley points out, in many avenues of biomedical research, disease olympics relies upon a false dichotomy. For example, the antimicrotubule agent estramustine was the first targeted therapy for breast cancer cells, for which it failed. Instead, it became the first (and until recently, only) chemotherapy agent for prostate cancer. Sites like prostatecancerpetition.org, although laudable in some respects, often neglect these crucial second-order effects.

I love this phrase because it articulates what was previously an unconscious anxiety of mine, its meaning is intuitively obvious (i.e., it doesn't propagate "insider baseball"), and it addresses an important issue. One thing I don't like about the phrase is that the word "olympics" is usually capitalized. I worry that this will hurt its memetic staying power, because capitalization often perturbs the flow of a sentence. 

Yawn Contagion Increases With Social Bond

The social bond categories are 0 = strangers, 1 = acquaintances, 2 = friends, and 3 = kin and life partners, more details here. Perhaps this would be a good litmus test for how close you are with someone?

Note that merely reading about yawning is often sufficient to elicit a yawn.

Reference

Norscia I, Palagi E (2011) Yawn Contagion and Empathy in Homo sapiens. PLoS ONE 6(12): e28472. doi:10.1371/journal.pone.0028472

Time Estimation Correlates With Math Smarts

Details here. For the five tone durations, the mean correlation coefficient was 0.59, and was still significant when the authors adjusted for a measure of general intelligence.

Suddenly "what time is it?" isn't such an innocuous question.

Reference

Kramer P, Bressan P, Grassi M (2011) Time Estimation Predicts Mathematical Intelligence. PLoS ONE 6(12): e28621. doi:10.1371/journal.pone.0028621

Reviewing Newt's Reviews

That is a histogram of his Amazon book ratings, and here is the source (HT TC). He is quite the pushover, and the above is a classic example of why a good rating system must rate the raters or suffer from bias. What about the content? Two trends stick out:

1) He is highly positive. Among his most common adjectives are "masterful," "remarkable," and "brilliant." Even when he gives a book four stars, he rarely says anything negative, and in fact it's usually not clear why books didn't get the top score of five.

2) He is highly technical. By this I mean that the majority of his sentences are devoted to strict summary rather than analysis. This makes sense, as it is probably smarter for a politician to say something obviously factual (and thus unassailable) than to take a risk.

(A Big) Part Of Science Is Publicizing Your Results

I’ve actually seen this sort of thing—claiming a result while backing off from claiming it, circulating a writeup to a few people but not really circulating it, burying an observation where no one will find it—happen over and over again in science, and its invariable effect is to leave fields in a state of utter confusion. There’s an excellent reason why, 350 years ago, science moved from the “announce-by-cryptogram” model to the model of rapid, widespread dissemination of research. And I’m not willing to forsake the attendant gains in human progress, just because some commenters here seem to enjoy the romantic image of someone stuffing the proof of a theorem into a bottle, throwing the bottle into the ocean, then going back to collecting seashells (or whatever), secure in the knowledge that the history of mathematics will need to be rewritten once the bottle washes up on some distant beach a thousand years later. Sorry, not how it works in this civilization.

That is yet more interestingness by Scott Aaronson in the comments of his blog. There is a thin line here, no doubt, as too much marketing and not enough meat is anathema to progress. But yes, in order for our results to be useful, we must attempt to ensure that they are heard by people who can use them. One way to do this is to split disparate ideas into separate, shorter papers, even though that practice is sometimes disparaged.

####

Of the well-subscribed to bloggers I read, Aaronson is one of the few who consistently responds to commenters. He also seems to have grown quite tired of his commenters, as he has threatened to shut down his blog more than once. Perhaps long-lasting success in the blogosphere selects for people who do not respond to comments. 

What Does "Statistical" Mean To You?

What exactly do [the authors] mean by a quantum state being “statistically interpretable”?... Basically, [the authors] call something “statistical” if two people, who live in the same universe but have different information, could rationally disagree about it.... As for what “rational” means, all we’ll need to know is that a rational person can never assign a probability of 0 to something that will actually happen. 

To illustrate, suppose a coin is flipped, and you (but not I) get a tip from a reliable source that the coin probably landed heads. Then you and I will describe the coin using different probability distributions, but neither of us will be “wrong” or “irrational”, given the information we have.

That's Scott Aaronson discussing a paper about the nature of quantum states. Googling "define statistical," I see, unsurprisingly, "of or relating to the use of statistics," and then googling "define statistics," I see "the practice or science of collecting and analyzing numerical data in large quantities." 

To me, the large quantities bit emphasizes that the role of statistics is to parse signal from noise, which is only possible with more than two data points (or, to be fair, some assumptions). So, I'd consider the authors' use of the word statistical to be sort of non-standard, because it seems to be able to be used for interpreting just one quantum state. 

Quite possibly this is actually standard use of the word statistical among certain physicists, which would make this yet another example of why you shouldn't assume that terminology is at all consistent across disciplines. 

The Self-Reinforcing Effects Of Ignorance

Outside of baseball there had been, for decades, an intellectual revolt, led by a free thinker named Bill James, devoted to creating new baseball knowledge. The movement generated information of value in the market for baseball players, but the information went ignored by baseball insiders. The market’s willful ignorance had a self-reinforcing quality: the longer the information was ignored, the less credible it became. After all, if this stuff had any value, why didn’t baseball insiders pay it any attention? To see the value in what Bill James and his crowd were up to you had first to believe that a market as open and transparent as the market for baseball players could ignore valuable information—that is, that it could be irrational.

That's Michael Lewis, more here. In particular, to find value in old, unorthodox stances, you have to believe in the possibility of status quo bias. 

A New Solution To A Grid Coloring Challenge

It is here, as explained by Alexandre Thiery. The challenge is to find a four color schema such that a 17 x 17 grid has no rectangle with the same four colors at each corner. The best known solution, shown below, has three rectangles. They are denoted by the black lines.

Who can find a schema with no such rectangles? Does one exist?

####

The fact that I enjoy this so much indicates some sort of bias towards colorful things. Or maybe just pretty things, more generally. 

Casnocha On Felten On Loyalty vs Universality

[L]oyalty is essential in human affairs. Without loyalty, trust disappears and relationships crumble. The problem is, loyalties conflict. For example, when friends commit immoral acts, should you stand by them (loyal to friend) or uphold moral principles (loyal to principle)?...  

What's the difference between a little and a lot of wickedness? That's up to you. Figuring it out is an example of a tough decision Felten says we need to make, case by case.

More here. Loyalty vs universality is great example of a trade-off that comes up often and leads to tough decisions. Every time you spend luxury money on yourself or friends that could have gone to an effective charity or basic research, you're implicitly choosing loyalty. Two thoughts:

1) Contra Felten, I'm reluctant to say that decisions for any trade-off must be made on a case by case basis, because there are huge gains to be reaped in cognitive efficiency by creating rules and following them.

2) The trade-off can interact with some now vs more later. That is, being loyal to your associates and thus building connections in the short run can help you achieve your goals in the long run. These long run goals, in turn, may be designed to make a larger set of people (or even "everyone") better off, thus justifying a universally beneficial motive.

Yet, how many historical atrocities were once rationalized with this logic? If you really favor the welfare of everyone equally, it's probably safer to act that way while on the way to your goals, instead of planning to do so in some future world. 

The Value Of Thinking About Rating Systems

A new project I have just started is going to generate personalized movie ratings for users. The way it works is as follows. You rate the movies you have seen. Then the system finds other users with similar tastes to extrapolate how much the you will like some other movies. It is currently written entirely in Python.

That's from Sergey Brin's 1996 resume. Prior, of course, to co-founding Google.

Correlation, causation, or aberration? You tell me. 

Can You Keep Your Job While Innovating?

Justin Lall says, at least in the context of competitive bridge, basically no. This is similar to NFL coaches not going for it on fourth down as often as they should, but a little bit more nuanced.

The situation in bridge is less dire, as percentage plays are respected, and the math is often easily demonstrable. However, a large part of becoming a great bridge player, good enough to be hired to play on professional teams, is having excellent judgement about when situations to go against conventional wisdom or A priori odds.... 

The biggest problem is that you are not just thinking about a bridge problem anymore, that’s the easy part, you’re deciding if making this play is worth possibly getting fired, get a bad reputation, maybe never get hired again and be forced to get a 9-5 job that pays much less.

You can generalize his argument to the claim that some now vs more later trade-offs are a superset of exploration vs exploitation trade-offs.

I agree. In the short run people who follow the status quo have an edge, especially in the absence of good long run indicators.

One prediction of this is that if 1) the football season were longer, or 2) instantaneous ratings for the probability that a team would win became standard (like "pawn units" in chess), more coaches would go for it on fourth down.

####

If you think this post was just an elaborate excuse to use the word "superset," you know too much.

Is One Willpower Trade-Off Fundamental?

 [T]he authors offer no systematic account of the trade-offs the brain must make among goals that differ in their likelihood of success, their time horizons and their evolutionary impact. The old joke about the man in front of a firing squad who refuses the customary last cigarette because he’s trying to quit reminds us that deferring a reward does not always make sense....

That's from Steven Pinker's otherwise mostly positive review of Baumeister and Tierney's Willpower. The relevant trade-offs in my current schema are some now vs more later (delay discounting) and sure bet vs shot in the dark (probability discounting).

Mentioning both in the same sentence reminds me of an old question of whether these trade-offs are actually equivalent on some fundamental level. Green and Myerson's highly cited '04 review addresses this.

From one perspective, you can think of a lower probability for success on a task as indicating that it will need to be repeated more times, and thus take longer. On the flip side, you can think of a longer delay as indicating a lower chance that the expected reward will actually be received.

One problem with both of these theories is that for humans, the discounting rate differs between delay and probability with respect to amount. That is, smaller rewards lose value more quickly as delay increases, but larger rewards lose value more quickly as the odds against increases. Check out this figure for some data.

This amount finding remains contentious, because it has not replicated in studies of non-human animals. And in those studies, it looks like delay discounting could be the more fundamental force of the two.

Still, some intuition remains on the side of considering them as distinct trade-offs. I don't think you'd want to tell an out-of-luck gambler in Vegas that if they just play longer, they'll eventually win.

####

After reading one of Baumeister's original papers, Tierney's exposition, Pinker's review, and Robert Kurzban's three published critiques, I am now in the curious situation of having devoured a fair number of words about a book without any inclination or intention to read the text itself.

Do I feel bad about this? Not really. I still think that the hegemony of the book is a bit anachronistic, and prefer the back-and-forth commentary that blog posts and edited articles provide.

But perhaps this is just a rationalization of my lack of willpower to actually close my computer screen and pick up a book. 

Power To The Wiki People

While reading Robert Kurzban's (mostly good) book Why Everyone (Else) Is a Hypocrite, I came across a fragment of a sentence which annoyed me.

Kurzban is in the midst of explaining a computer science topic when he writes, "according to Wikipedia, which I am usually hesitant to use, but will for this purpose," and then block quotes, starting with "creating a domain-specific language...".

As it turns out, Wikipedia is written by people. Rp, a Dutch software developer, added this sentence on April 24, 2008. The only difference between Rp's change and the current version is a superfluous "of course" which was removed on May 12, 2008. This is not hard to decipher based on a gander at the page's revision history. It took me about four minutes.

Listen, I recognize that Wikipedia is low status, which is why Kurzban had to express his hesitation to use it. But we all know how Wikipedia works now. People edit it. You aren't constrained to merely cite "Wikipedia", you can look back and see exactly which user wrote that passage first, and cite that user.

Some of my clever readers are probably already mentally defending the status quo by saying that some sentences or portions of articles have been edited so many times that it would be difficult to say who wrote them.

Yes, in very rare cases like The Iraq War this is the case, but it happens much less often than you'd expect. The vast majority of articles are edited in chunks of sentences, paragraphs, or sections, and these chunks are eminently traceable.

Perhaps the best thing about such a shift in citation norms is that it would help incentivize people to edit Wikipedia. If you think this is not an issue, you are sorely mistaken.

Consider the page on epigenetics. Inspired in part by Razib's manifesto about the growing importance of this topic, I have subscribed via RSS to the changes made to the page since January. What I expected were the vitriolic edit wars deserving of such an unfolding, important topic.

But I've seen nothing of the sort. In fact the page hasn't changed significantly since Team Cytokine Storm made some edits last November. In the meantime, how many words have been typed about epigenetics for publication elsewhere?

For example, in the past two months, there have been at least four academic reviews on topics in epigenetics (see here, here, here, and here). See for yourself--here is a pubmed search for "epigenetics review."

How many people will read these reviews? Do you think that more will read those reviews than will read the Wikipedia page? Is this a healthy division of labor?

nobody does homework on saturdays

Bottom Line: When quoting or referencing an article hosted on Wikipedia, cite the major user(s) that contributed, instead of just "Wikipedia."

Punishing Praise

[B]ecause we tend to reward others when they do well and punish them when they do badly, and because there is regression to the mean, it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them.

That's Daniel Kahneman, more here. Regression to the mean will occur in situations that involve at least some luck, which is to say, almost everywhere.

Luck seems especially inevitable once we consider that spontaneous fluctuations in your brain's dynamic states (as seen in fMRI BOLD responses) can help account for trial-to-trial variability in behavior.

One study found that the amount of pain that people feel following laser stimulation (equivalent to a pinprick) can be predicted (beyond 5% chance) based on baseline, spontaneous activity in certain brain regions three seconds before the stimulation. (pubmed, PNAS)

Another study found that 74% of the within-participant variability in a button press force task could be attributed to ongoing fluctuations in neural activity. (pubmed)

In light of the fact that seemingly uncontrollable neural fluctuations play such an important role in behavior on any given attempt, punishing people for poor performance on small sample sizes seems particularly pernicious.

But then again, most punishment is probably not really intended to improve future performance, but rather to improve the mood and status of the punisher. 

Are We More Like Chimps Or Bonobos?

In Sex at Dawn, Ryan and Jethá argue that humans are more like bonobos. This is, in part, because we both 1) employ diverse sex positions, 2) have sex for non-reproductive ends, and 3) gaze into each other's eyes during sex (when this jives with #1).

In his review (pdf, OA), Ryan Ellsworth disputes their thesis and makes the case for the chimp model. He emphasizes that humans and chimps (but not bonobos) share "sex-based hierarchies, sex-biased cooperation and coalitions, and intergroup hostility."

I've only skimmed Sex at Dawn, but I find Ellsworth's review much more persuasive. I'm happy being chimp-like.

(photo credit to patries71)

Seven Thoughts On Rules And Willpower

1) John Tierney discusses ego depletion, the idea that willpower is an (unconsciously) expendable resource. He relates it to trade-offs: 

Once you’re mentally depleted, you become reluctant to make trade-offs, which involve a particularly advanced and taxing form of decision making.... To compromise is a complex human ability and therefore one of the first to decline when willpower is depleted. You become what researchers call a cognitive miser, hoarding your energy. If you’re shopping, you’re liable to look at only one dimension, like price: just give me the cheapest.

This is why I think learning about trade-offs can be so useful. The less novel a decision is, the less resources it should use up. The more general your schemas are, the more easily you'll adapt. 

2) How do people typically deal with ego depletion? Apparently,

[E]ventually [you] look for shortcuts, usually in either of two very different ways. One shortcut is to become reckless: to act impulsively instead of expending the energy to first think through the consequences.... The other shortcut is the ultimate energy saver: do nothing. Instead of agonizing over decisions, avoid any choice. Ducking a decision often creates bigger problems in the long run, but for the moment, it eases the mental strain. 

I think Tierney is a bit off here, as he neglects a crucial strategy: devising rules. "No coffee after two, no liquor before five." "Always take the middle option." Even, much more perniciously, statistical discrimination. For better or worse, we advance cognitively by thinking less. 

3) Marketers know that we use rules, and they (wisely) use our rules against us. This makes rarer rules more valuable (holding effectiveness equal), as they will be less exploitable. 

4) Before you go off devising your own un-gameable rule system, recognize that there are trade-offs to thinking about topics like this. It might be a better use of your time to just go along with most of the status quo rules and accept that you'll sacrifice some small amount of money to savvy marketers. As Ice T says, it's not about being mad at everything, it's about being really mad at the right things.

5) If willpower is a muscle, can you train it? Yes, self-control training can improve one's willpower to complete unrelated tasks. For example, in one study, maintaining better posture for two weeks (and keeping a diary about it) significantly improved hand-grip persistance (link, pdf). More examples here. 

6) As Tierney's article discusses, decision fatigue often helps trap people in poverty. But since willpower is apparently like a muscle, shouldn't exercising decision circuits improve willpower enough over time to escape the trap? 

7) I expect that the confound with the above is an interaction with stress. Making self-control decisions when you feel comfortable and empowered increases your set "willpower" level. But the emotionally stable undergraduates studied are not very representative of the population at large. When people must make decisions under psychological duress, it might instead condition a sort of "decision avoidance." Kind of like how overtraining your muscles can actually decrease strength.

Addendum 8/25: See Robert Kurzban's astute criticisms of this model (HT Brian Potter).  

The Motion Mystery

Even to eminent thinkers, an explanation for motion was seemingly unknowable near the turn of the 20th century.

For example, in the late 19th century Thomas Huxley said that "existence, motion, and law-abiding operation in nature are more stupendous miracles than any recounted by the mythologies...".

Santiago Ramón y Cajal went further in the early 20th century, writing, "there is no doubt that the human mind is fundamentally incapable of solving these formidable problems (the origin of life, nature of matter, origin of movement, and appearance of consciousness)."

It turns out that in the early 21st century we now have a pretty good answer to this mystery. That answer comes in the form of molecular machines.

As Steven Pinker explains, "the stuff of life turned out to be not a quivering, glowing, wondrous gel but a contraption of tiny jigs, springs, hinges, rods, sheets, magnets, zippers, and trapdoors...".

One example of such a machine is ATP synthase, which literally works like a rotor:

Now, molecular machines don't explain why atoms themselves move (you'd need to go deeper for that), but their existence certainly does explain why you can move in the absence of an outside force and a rock cannot.

If we don't kill ourselves (from, e.g., an environmental disaster or nuclear war) first, we will to continue to solve puzzles that some of us currently consider intractable mysteries.

And in the meantime, we should cultivate some doubt in our doubt of the potential of science.

Searching For The Imdb Of Books, Part II

As watching imdb's top 250 most highly rated movies has proven to be such a smashing success, I have long yearned to find (and fleetingly, to develop) a similarly authoritative list for fiction books. The keys for a good list are: 1) a large sample size, 2) shrinkage estimation of ratings to the average, 3) a continuous scale (the more levels, the better, but yes we'll often have to settle for five stars), 4) defenses against gaming, and 5) a wide index of titles. To the best of my knowledge no site fulfills all of these requirements. These are the current contenders:

Amazon Reviews: Upside: They have a huge incentive to index all available books and are proficient at combining ratings across different editions of the same text. They also have a useful "was this review helpful to you?" tool which could eventually be employed to rate the raters and thus weight the overall ratings. Downsides: Their insistence on showing the average rating in half-star increments (typically 5, 4.5, 4, or 3.5) means that it involves manual calculation to distinguish between the two radically different scores of 4.24 and 3.76. I also often don't trust the resistance of their ratings to gaming. But most damningly, there is simply no attempt to create a good list of the most highly rated fiction books. Filtering by "highest average rating" in "literature and fiction", their #7 best fiction book of all time is currently Jim Gorant's The Lost Dogs: Michael Vick's Dogs and Their Tale of Rescue and Redemption, which is probably a fine book, but I think the author would be insulted to hear that it was considered fiction, and I think more than three-quarters of the english profs across the country would be insulted to hear it called literature.  

One-Time Votes: By this I refer to ad-hock competitions of various websites which ask users to vote on their favorite books. There are many of these strewn across the web, for example, check out NPR's top 100 science fiction and fantasy books, or Modern Library's top 100 novels. Upside: These tend to get large sample sizes (NPR had >60,000 votes), which makes them more accurate and harder to game. Downside: The process is not iterative and requires manual input to update, so they won't last or scale. More troublingly, many (such as NPR's) only allow the option to select one's favorite books, without voting others down, which unfairly favors books with high variance as opposed to just high average quality. 

Google Books: The site aggregates ratings from elsewhere on the web, including major vendors and online "bookshelves." Upside: Transparent code, takes ratings from diverse sources, and has a clean layout. Downside: Like Amazon, also displays ratings in half-star increments (et tu, google?). But their biggest problem is that different editions of books are stored in different locations and the ratings are not aggregated across editions. See, for instance, the first four results of a search for "pride and prejudice" (here, here, here, and here). Now, even if they did manage to output one total score per novel, it still doesn't seem very google-like to actually curate such a list themselves. But in that case, it wouldn't be hard for someone else to scrape the ratings and convert them into a ranked list. 

Library Thing: Upside: They have scale, with over 10 million ratings, and they already have some pretty cool statistics (check out the most "connected" people--Napolean is #1). They also do have a top 25 books by rating. Downside: They need to split the rankings for non-fiction and fiction. At this point I've given up on searching for a canonical non-fiction ranked list, as those ratings are so context-dependent and world-view driven. And they need to do a better job of categorizing in general. For example, the movie for LoTR:Two Towers, while an awesome movie and in imdb's top 250, should not be among the highest rated 25 books. More importantly, the editors of the site have not implemented a rating system that punishes books with fewer ratings. Instead, books simply need a minimum total of 20 ratings to make the list. This is bothersome, but easily improved, as the editors could simply study and implement the imdb method. 

Good Reads: Upside: As far as I can tell, this is the largest "bookshelf" site with the most user ratings. Huge potential. Downside: They've made no attempt to publish a list of the highest rated books across the site! All I can ask is, what is holding you back, GoodReads editors? Qualms about alienating authors whose works won't make the list? Fears of being labelled imperialistic? These are both hogwash. Our time is scarce and in order to be informed consumers we need to know what the best books are. If you are worried about the arbitrariness of the minimum votes cut-off, then publish multiple lists with different scaling parameters. You will thank me later when the list gets out-of-control traffic. Indeed, a group of passionate GoodReads users recently called for such a list. To this valiant effort I can only say, Viva la Résistance!

Some Now Or More Later In Moth Reproduction

The abstract from a (gated) paper by Nokelainen et al:

[Wood tiger moth males] have two distinct colour morphs: white and yellow. The efficacy of the warning signal of these morphs was tested by offering them to blue tits in the laboratory. Birds hesitated significantly longer to attack yellow than white males. In a field experiment, the survival of the yellow males was also higher than white males. However, mating experiments in the laboratory revealed that yellow males had lower mating success than white males. Our results offer an explanation for the maintenance of polymorphism via trade-off between survival selection and mating success.

Yellow moths are less likely to be attacked because their conspicuous color acts to fend off predators. On the other hand, the authors cannot suggest a clear reason for why white male moths have greater mating success. Whatever the mechanism is, this is a clear example of some now vs more later in evolution.

Bill Simmons' Rating Nihilism

Earlier he claimed, on scarce evidence, that "Rotten Tomatoes scares me as a metric," because "people are idiots," and that "their 'top critics' rating is much more useful."[1]

Now he explains that:

I believe Michael Jordan is the greatest basketball player ever, and I can prove it. I believe The Breaks of the Game is the greatest sports book ever, but I can't prove it. Books can't be measured that way — they hit everyone differently, so when we're evaluating them, we can only say, "You can't mention the greatest books (or albums, paintings, TV shows, movies or whatever) without mentioning that one." That's as far as you can go.

I don't get it. He thinks you can sort of rate movies (if you trust only the experts), but you can never rate books except for saying which ones are "among the best"? This is inconsistent.

He's right that there is a key difference between sports and film/writing, although it is not, as he claims, that the latter "hit[s] everyone differently." The difference is that in sports there is a known goal--for the team to win. That means that, at least theoretically, it is possible to tease out which player stats tend to correlate with winning, and then use those stats to evaluate players.

But notice the causality here. We can't evaluate individual players well until we know which stats are generally good indicators that a player will help eir team win. Intuition does not necessarily serve well here, an insight upon which books have been written and careers have been made.

In film/writing there is no such clear objective, and thus the ratings by individuals who have seen/read them must be subjective. So instead of evaluating statistics based on how they correlate with the objective of winning, we must instead evaluate rating systems based on inter-rater reliability. The goal is that if you added more independent ratings by unbiased raters, there should be as small of a deviation as possible between the new and old ratings.

The obvious suggestion is that, if we want better opinions, we need more of them to average out more of our random biases, like how hungry we were when we first saw the movie.

Again, the input must be subjective. But once we've decided upon the best rating system, its output is objectively our best estimate of that film/book's quality. Just as in sports, personal intuition is not the best estimate of quality, and to believe otherwise is simply hubris.

Perhaps it should not surprise us that a key opinion maker is arguing that we should only trust key opinion makers, instead of wide-scale opinion aggregators. But the rest of us don't have to buy it.

####

[1]: Rotten Tomatoes ratings have many problems, like the fact that they threshold scores into "good" and "bad" and count the percentage of each instead of employing a continuous scale. But that is a straw man for the claim that open, aggregated movie ratings are bad.

Venkatesh Rao On Protection vs Freedom

Rao explores an application of this trade-off,

[The] industrial age of mass-scale production required paycheck workers. People who had to be trained in industrial-style schooling.... [The] unpublicized purpose was to create a class of people that was far more disciplined and risk-averse than natural for the human species. In other words, a domesticated, comfort-loving species. This was achieved through, quite literally, conditioning. Bells rang for waking up and meal-times. Food appeared magically. Retirement was taken care of. Everything happened like clockwork. 

The first few generations resisted being drafted into the industrial workforce mightily. Not despite their intimate familiarity with risks ranging from bad harvests to disease, hunger and death through poverty, but because of it. Because they understood that with those risks came freedom. After that, the next generations were born and raised in captivity and never had a chance to sample the environments that might have made their wilder risk-taking instincts come out.

I get anxious when I read the word "generation", since it's so hard to define the gradients of separation. But thinking back on the older films I've watched, some of them (e.g. Modern Times) do explore this anxiety of becoming trapped by industry.

If Rao is right, does this indicate that part of the "wiseness" of earlier generations can be encapsulated as a more thorough understanding of trade-offs? 

Why Trade-Offs Matter

A respected friend recently gave me some push-back on this, asking: at the meta-level, what exactly is the value in reading a list of trade-offs? I offer three main reasons, in order of descending plausibility:

1) Interestingness: There are many systems used for classifying stories (e.g., here, here, and here). Now, the classical "decision" is not necessarily more "important" than the story. (And they are often intertwined, as we use our decisions to construct and shift the trajectory of our personal narratives). But still, we might consider a similar sort of interestingness as is found in classifying stories to be a lower bound on the value of a good trade-off classification system.

2) Awareness: Daniel Gilbert writes that "because resources are finite, every sensible thing we do is another sensible thing we don't. Alas, research shows that when human beings make decisions, they tend to focus on what they are getting and forget about what we are forgoing." And apparently teaching people about cost-benefit reasoning really can improve their ability and propensity to use it (see here).

3) Tactics: Reading one of the trade-offs might allow someone to recognize a systematic bias towards one side of that trade-off. In the future, when that person identifies a situation which can be classified into that trade-off, they could try to adjust for their tendency towards bias. This ability to treat individual situations as merely examples of broader trends is crucial for aligning short-term decisions with long-term preferences.

Seeing Status-Seeking Statements Everywhere

I follow humblebrag on twitter, and have often chuckled when the feed re-tweets people boasting about their achievements under the guise of humility. But for some reason (curator) Harris Wittels' article summarizing the most flagrant humblebraggers gives me some anxiety about supporting the feed and evangelizing for it.

Perhaps this is just the paradox of funny things losing their appeal once they are explained, kind of like growing to loathe a song once you come to realize what the lyrics actually mean.

But I think my anxiety runs a bit deeper than that. Consider Wittels' sarcastic take-down of "a very specific type of humblebrag, which is the 'some person did something great and I am very good friends with them' Humblebrag."

There's no doubt that mentioning your association with high-status people and institutions is a great way to seem high-status. But we also have to respect these would-be humblebraggers' plausible deniability, which is that they might really just be happy for their friends.

To me, it is this plausible deniability that distinguishes between having a conversation and bragging. It is typically socially acceptable to mention an accomplishment of yours if it comes up naturally in a thread that you did not initiate.

The logical conclusion of humblebrag and like-minded feeds continuing to rise in popularity is that people will become especially conscious of not bragging. Rather than making people more earnest, I think this will likely make them more mysterious. Since I'd prefer more earnestness, I say to err on the side of bragging, as long as it is real.

Full disclosure: I once sent in a tweet to humblebrag@gmail.com that I thought was a particularly good instance of humblebragging and got no response, so I might just be bitter.

The Cost-Benefit Of Considering The Costs And Benefits

Pierre-Simon Laplace, in my opinion the greatest statistician of all time, wrote eloquently about the cost-benefit principle in his magnus opus:

Consequently we ought always in the conduct of life to make the product of the benefit hoped for, by its probability, at least equal to the similar product relative to the loss. But it is necessary, in order to attain this, to appreciate exactly the advantages, the losses, and their respective probabilities. For this a great accuracy of mind, a delicate judgment, and a great experience in affairs is necessary; it is necessary to know how to guard one's self against prejudices, illusions of fear or hope, and erroneous ideas, ideas of fortune and happiness, with which the majority of people feed their selflove.

Napolean made Laplace the Minister of the Interior in 1799, which he apparently begged for. But he struggled with it, as Napolean recounts,

Laplace was not long in showing himself a worse than average administrator; since his first actions in office we recognized our mistake. Laplace did not consider any question from the right angle: he sought subtleties everywhere, only conceived problems, and finally carried the spirit of "infinitesimals" into the administration.

Let's be careful not to read too much into this one anecdote, as it has an n of one. But still, it's scary that thinking too much is a plausible way for a politician to fail.

In other contexts, this paradox is often called "paralysis by analysis." Insofar as it holds true, what exactly would mediate the trade-off between probabilistic thinking and timely, necessary action?

I think it's secular vs sacred. In the real world you need to make assumptions to actually make things happen, but in the world of intellectuals, as Robin Hanson says, "sharp people... distinguish themselves by not assuming more than needed to keep the conversation going." 

There's no doubt Laplace caught onto this. For example, when asked by Napolean why he didn't mention religion in his tome on probability, he famously replied that he "had no need of that hypothesis."

Again, not sure how far this example generalizes and I'd like to see some systematic data, but this is something to ponder.

Reform "Marco Polo"

Back when I was a lifegaurd, I spectated countless sessions of this game. And what a truly awful game it is. Nobody has fun, and the noise pollution is unabashed.

It's once again summer up here in the northern hemisphere, and as my base tan isn't going to improve itself, I have been putting in work by the pool. So this weekend I had to endure another few games of "Marco Polo."

Here's one easy improvement. Just add a finite limit to the number of times the person who is "it" can call "Marco." Say, 20. This will prevent him or her from spamming that ability and actually introduce some strategy.

More generally, the fact that people play so much Monopoly, Tic-Tac-Toe, and Marco Polo, despite their terribleness, is one of the strongest arguments I can think of against status quo bias.

Trade-Off #20: Quality vs Quantity

If the above photo had two flowers instead of just this one, would each seem slightly less pretty? I don't know for sure (we are talking about aesthetics, after all), but I imagine that most would say yes, because each flower would stand out less from the background and seem less unique. The only exception would be if the flowers somehow contrasted or augmented each other's beauty.

In scarce environments, and aside from such cases where individuals interact to produce effects greater than the sum of their parts, the average quality of an agent's choice will be inversely related to its quantity. Here are some examples:

• People often wonder whether they should spend lots of time and energy pursuing one high-quality mate, or distribute those resources pursuing many lower-quality ones. This is a very general quandary, which many if not all reproductive species face. (see here)
• In searching a text, an increase in the proportion of relevant results to total results typically comes at the cost of missing more of the possible relevant results from the whole search space. (see here)
• For an individual using an online social network, adding more "friends" usually decreases the quality of his relationship with his average connection. (see here)

Trade-offs are found everywhere, even in making a list of the most important and widespread trade-offs. So given the quality vs quantity trade-off that we face in adding more trade-offs to this list, the first draft of the canon will end here.

(photo credit to domesticated diva)

Validating The Next Revolutions

From my hopefully not overly-insular vantage point, the two books which have had the biggest impact in the first half of 2011 have been Richard Arum and Josipa Roksa's Academically Adrift and Tyler Cowen's The Great Stagnation.

There are many similarities between them. They both address broad, ongoing trends in American society, in higher ed and macroeconomics. They have both been read largely in e-book format, AA due to the prohibitive price of the print version, and TGS due to some brilliant/lucky marketing. And, oddly, their approaches both owe at least some homage to the views of entrepreneur and raconteur Peter Thiel, who has widely-discussed qualms with higher ed, and who won the dedication of TGS for his insight into the lack of innovation and growth in our economy.

This last point deserves some drawing out. For all the hype, the overarching theme of neither book is necessarily novel. Arum and Roksa make many points that were assumptions, not conclusions, of conversations at Vassar's cafeteria. No one ever wondered with incredulity, "wait, instructors are gaming their end of semester ratings?" The case is similar for Cowen's thesis. Individuals who decry the sluggish innovation and dormant middle class prospects in America are hardly in short supply. Just notice how many express fears that America doesn't "make anything" anymore.

But what both books do accomplish is to make these everyday arguments more rigorous. And, perhaps because the authors are academics, they also serve to validate what might otherwise be seen as merely mumbles and whimpers amongst the broader populace.

The impact of these books also speaks to the fact that we as a culture are still only barely embracing the brave new world of the internet. Content-wise, these both could easily have been "merely" articles. Publishing in some gated academic journal would obviously have reached few, but even if they had been published in the popular press, I doubt they would have had the same success. Regardless of the e-book format, we still love the idea of the book. For instance, I get expontentially more comments and questions IRL about my shelfari page than my del.icio.us page, although I've surely invested more time and energy into curating the latter.

Cowen himself says, on his blog, that, "we’ve yet to really organize our economy around the internet, as we someday will, and then the gains will be enormous." Perhaps the impact of these e-books could be considered Exhibit A of both our current lack of mobilization around the internet idea economy, as well as its potential once we do get our act together.

Boredom Trade-Offs?

Aaron Haspel says that "an above-average capacity for boredom is optimal; a superior one is disastrous." Somewhat similarly, Mike Tully argues that becoming bored with a pursuit will inhibit artistic and athletic greatness.

Perhaps we can think of the capacity for boredom as a cognitive trait that pushes you towards the plasticity side of the plasticity vs specialization trade-off.

But really this idea seems a bit too vague. It's not clear whether one's capacity for boredom extends uniformly across all domains, and there are many other factors involved.

For example, you could argue that Ted Williams was able to specialize because he never grew bored of baseball, or you could argue that he specialized because he so quickly grew bored of everything else. With the former frame he has a below-average capacity for boredom, while with the latter it's above-average, but the end result is still the same.

Which Parts Of Crowds Are Wise?

Peter Freed has written a pretty ambitious critique of Jonah Lehrer's summary of this study (pdf) on the wisdom of the crowds. The crux is that:

But now that I realized he really meant median, and that maybe he didn’t know what median meant.  Because median guesses are not guesses by a crowd, as Lehrer states.  They are guesses by a single person... [Lehrer] is talking about that 0.7% single-person data point: one person, selected after giving their answer, got close to the correct answer on one of six questions.  One person guessed 10,000 when the answer was 10,067.  That’s one hit out of 144 x 6 = 864 attempts.  That seems about right to me, from a common sense perspective. Which is to say, that is a shitty batting average.

Scrolling through the comments, I was pleased to see Ian Sample point out the critique of Freed's critique that I was going to make:

In Wisdom of Crowds studies you can look at the mean and / or the median. The median usually gives the best result if the guesses *do not* follow a normal distribution. The mean, of course, exploits the error-cancelling advantage that WOC is known for, that is, as many people under-estimate as over-estimate the right answer, so averaging cancels all but systematic biases. But to my point. To dismiss the median answer – one guy’s response – misses the fact that without the crowd you have no median answer to dismiss. Without the crowd, you do not know which value to pick. That’s the whole point. The crowd steers you to the median value, which in many cases outperforms the mean.

The median is indeed generated by only one person, but it becomes interesting only in the context of all the other estimates. It is useful here because it offers resistance to outliers. For example, some less numerate soul might have guessed 1,000,000, which is way off from the true value of ~ 10,000, thus skewing the arithmetic mean. In that case you'd much prefer a more robust statistic like the trimmed mean or the median.

In prediction markets, the most recent price of a transaction doesn't always best represent the current beliefs of the market. There's more info if you look at the whole distribution of orders. Similarly, it is unfair of Freed to dismiss the whole data set just because one type of estimator is flawed. This is one of the coolest parts of statistics, using potentially counter-intuitive methods to extract useful info out of data, to find the wisdom in the crowds.

Fighting The Lernaean Hydra Bias

I'll just only mention the heads I do cut off

In one Greek myth, Hercules takes on the task of killing a serpent-like, many-headed beast. This is made more difficult by the fact that its heads regenerate, so even if Hercules chops one off with his sword, another will simply sprout in its place. John Ioannidis uses this frustrating scenario as an analogy for a problem in the world of scientific publishing in his discussion of meta-analyses (doi:10.1002/jrsm.19).

The example Ioannidis employs to explain this problem is his experience doing a meta-analysis on the pharmacogenetics of certain polymorphisms for asthma treatment (doi:10.1097/01.fpc.0000236332.11304.8f). There were many studies that fit the criteria, but they each evaluated their own endpoints and genetic contrasts. That is, in most of the studies, the vast majority of possible correlations that could have tested with the data between phenotype and genotype were either not done or not reported.

So the surface problem, in so far as this case generalizes to others, is that published studies are not as exhaustive as they could be. But the central, troubling implication is that these studies do not fail to be exhaustive because of time or computational constraints, but because the researchers want to emphasize the usefulness and/or interestingness of their results. This is more insidious--this is why the hydra heads regenerate.

Now, one can use meta-analysis to retrospectively "chop off" findings that are truly insignificant by combining the results of many different data sets. But meta-analysis itself can be biased in many ways (e.g., during study selection), and moreover, later researchers can just come back to the issue and cherry pick more novel associations, thus "sprouting" more statistically significant findings.

When faced with the hydra, Hercules knew he couldn't go it alone, so he called on his nephew for help, who suggested that they cauterize the stumps with fire before the heads could regrow. An analogy to this strategy might be to post warnings on the electronic copy of papers that have been called into question by later studies. Such a warning would be much milder and hopefully less political than a retraction, which typically implies some sort of error. Publishing a potentially informative result that is eventually overturned is still laudable.

But instead of this type of patchwork fix, a more fundamental approach seems more fruitful. In the original myth, only one of the hydra's heads was truly immortal, and this was the one that Hercules needed to chop off to finally defeat the beast. The immortal head of the scientific publishing hydra is the incentive structure pushing researchers towards significance hunting in the first place.

Reworking these incentives is what Ioannidis is fundamentally arguing for, as the way to kill the Lernaean hydra bias once and for all: more standardization, more consortia, and more of a push towards openness and replicability. Every study might combine previous data with its own for estimating the posterior probability of the parameters it is examining, and all research might be seen as a continuous and cumulative meta-analysis. Maybe one day.

(photo credit to Frank Rafik)

Color Me Old-Fashioned

A fantastic idea from Risto Saarelma on how to re-design the comment section of the website Less Wrong:

Provide an ambient visual cue on how old a comment is. First idea is to add a subtle color tint to the background of each comment, that goes by the logarithm of the comment's age from reddish ("hot", written in the last couple of hours) to bluish ("cold", written several months or more ago). Old threads occasionally get new comments and get readers in via them, and the date strings in the comments require some conscious parsing compared to being able to tell between "quite recent" and "very old" comments in the same thread by glance.

Too true. Who takes the time to read the actual date of a comment? This way you wouldn't have to.

This sort of subtle clue is something that people will appreciate and pick up on quickly. For example, on the blog Marginal Revolution you can always tell whether Tyler or Alex is posting because Tyler only capitalizes the first word in the title of his posts whereas Alex capitalizes all of the words in his titles. Knowing this, you won't have to waste time scanning the byline as you plow through your RSS feeds because you'll already know who wrote it from the title.

Three Thoughts For Spring

1) If there are things that you can do to increase your perspective on your current problems, like taking a weekend off or writing a journal, then there also should be things you can do to decrease it. But I can't think of any. So is perspective the sort of thing where your typical state is a steady decrease unless you actively increase it through certain, discrete actions? You either have to agree with this model or describe specific ways you can lose perspective.

2) Who will systematically review the systematic reviews? Cochrane reviews, that's who.

3) One thing I wonder, as I try to get into Anki, is how we could make spaced repetition learning into a game. And I don't mean some boring game, like "how many flashcards can I get right today?", but a sweet game, with long-term goals and leveling up and side-missions and bad guys to defeat. I don't know if it could be done, but couldn't you imagine this as a big part of the future of education?

Sometimes Simple?

"Everything is more complicated than you think." - Synecdoche, New York

Is this true? No way. We can easily come up with counterexamples. Take any superstition, like some time in your youth that you were afraid of monsters in your closet and it turned out to just be a broom propped up at a weird angle. Which is more complicated--the angled broom or the hidden monster? That's a layup.

So the better question is: are things on average more complicated than you think? It sort of seems like it. But part of the problem is that we tend to simplify old beliefs to make our current ones look more intelligent in comparison. Consider the history of Dale's principle. Some authors understand this to mean that neurons can only release one type of neurotransmitter. If stated in this form, it's clearly wrong, and so newer researchers can claim credit for debunking it. But when you look at its inception, it turns out that "one neuron = one neurotransmitter" is probably not what the principle was actually meant to imply. So our intuitions about how our beliefs tend to change probably speak more to what we currently believe about the past than to what we will believe in the future.

If we could show conclusively that things in general are more complicated than we think they are, that'd be good to know, because if reality tends to deviate in some predictable way from your expectations, then you're doing something wrong. But I'm not sure that the answer will turn out to be so simple.

Milestones In...

I've recently discovered Nature's milestones index, which links to timelines of the major advances in the research of many fields: light microscopy, gene expression, development, etc. These were chosen by panels of many experts. For example, these 40 helped decide the milestones in cancer research. The timelines have links that explain why each milestone was important, like this one on the first methods of DNA sequencing. Awesome.

I wonder if there's some way that we could allow people to vote on these milestones in a similar way that others have set up for people to vote on milestones in computer science? If so, we could tap into what seems to me like the most productive form of crowdsourcing, where experts define the field, and then the masses rank the entries in that field.

The Wisdom Of Whuber

That's William Huber, whuber for short, dispensed in his answers at the relatively new stats Q&A site, Cross Validated. His answers are the best on there, reputation normalized to the number of answers (with shrinkage). Here he writes about whether the median is a better summary stat than the mean:

Statistics does not provide a good answer to this question, IMO. A mean is ok to use, too, and is relevant in mortality studies for example. But ages are not as easy to measure as you might think: older people, illiterate people, and people in some third-world countries tend to round their ages to a multiple of 5 or 10, for instance. The median is more resistant to such errors than the mean....  Thus, for demographic, not statistical, reasons, a median appears more worthy of the role of an omnibus value for summarizing the ages of relatively large populations of people.

Here he writes about the biggest questions in statistics, from which I'll reproduce two (emphasis his):

• Coping with scientific publication bias. Negative results are published much less simply because they just don't attain a magic p-value. All branches of science need to find better ways to bring scientifically important, not just statistically significant, results to light. (The multiple comparisons problem and coping with high-dimensional data are subcategories of this problem.)
• Probing the limits of statistical methods and their interfaces with machine learning and machine cognition. Inevitable advances in computing technology will make true AI accessible in our lifetimes. How are we going to program artificial brains? What role might statistical thinking and statistical learning have in creating these advances? How can statisticians help in thinking about artificial cognition, artificial learning, in exploring their limitations, and making advances?

And here he writes about whether you should use a normal distribution to assign student grades:

I think that if any of those 800 students were to read this question, they might be offended. How well did they perform? How much learning was accomplished? That is what a grade should reflect, not some arbitrary statistical summary of their position in a group. IMHO this question should be recast in terms of teaching objectives, not statistical procedure, such as "what is a good way to convert raw scores to grades in a way that respects student accomplishments and advances the learning objectives of this class?" Statistics can help, but blind statistics--like standardization--will not.

Although they are often quite quantitative, his answers show how good stats rely on far more than just math. 

Testing Robustness vs Fragility In Chemotaxis

receptor modulation, from doi:10.1371/journal.pone.0011224

Bacterial chemotaxis depends (like most biological functions) upon an intricate signaling network, in which all of the molecules (mostly enzymes) must work in unison. Oleksiuk et al have just published a paper (doi:10.1016/j.cell.2011.03.013) showing convincingly that ambient temperature would affect many of the molecular components of this pathway in E. coli, but the system is optimized to work despite variations in temp.

For example, they show that the pathway's receptor kinases have modification states (see above) with opposing temperature dependencies. So, when the temp changes, the activities of receptors with different mod states compensate for one another to allow the system to maintain the same function.

Since there is apparently a canonical trade-off between robustness and fragility, E. coli's robustness to variability in temperature should come with some costs. One form this cost could take is that it would make a mutation to a temp sensor gene more deleterious. Another form this cost could take is that it would make the bacteria more susceptible to viruses that mess with parts of the temperature regulation system. Maybe some group will show one such cost to be present, or even to be the dominant force? We'll see.

Ranking Ideas In Science

Last summer I bought and read The 100 Most Important Science Ideas after noticing it in a bookstore (my first mistake--I should have checked the ratings online first). I learned a fair amount from it, but I have to say it fares miserably in its attempt to actually rank ideas in science. First, it only covers three subjects: genetics, physics, and math. Second, even within those subjects, the topics are listed merely by date of discovery, not importance. Finally, there was little to no space devoted to methodology.

Subsequent attempts to find lists of the most important science ideas, via google searches and cold e-mails to potentially knowledgeable people, have also left me empty-handed. Uncool.

A good system to rank science ideas, both historically and as they are published, would be so money. The historical list would be really useful for educating the next generations and as outreach to the public. And dynamic, post-peer review ratings would help researchers use their precious time reading the best papers, instead of relying solely on the impact factor of the journal.

Given the above, you can imagine my immense pleasure to see Scott Aaronson's announcement today of a site that allows anyone to vote on milestones in computer science.

There are at least a couple of ways this voting could be done. The first way, as they currently have the site set up, is that users can pick and choose to vote any individual idea on the list up or down. The advantage of this is that users can choose to vote only on the ideas that they actually know something about.

The second way is that the site could present two options to users, the users would choose which of those two are better, and then an algorithm would use those preferences to rank all the ideas. The advantage of this is that it's more fun. Indeed, you might recall that a similar system was employed by the young Mark Zuckerberg in facemash. Wait, you haven't seen The Social Network? C'mon now, it's #190 on the top 250. Step your game up.

Anyway, bravo to Jason, Ammar, and Scott. Now we just need to create similar lists for all other scientific disciplines, incentivize people to vote on them, and aggregate the results. We'll also def need some kind of normalization to account for the fact that computational pursuits will have at least 10x the votes, because those people are on their computers like all day.

When Can We Measure Grit?

Jonah Lehrer's interesting, 15,000 character article about measuring NFL quarterbacks concludes by saying that we have neglected grit in favor of IQ because "grit can't be evaluated in a single afternoon". But this is clearly not true, as earlier in the same article he notes that Angela Duckworth has developed a survey for grit that predicts (well) both Westpoint cadet graduation rates and spelling bee performance. Here (pdf) is Duckworth's rating system for grit, including "self-report and informant-report versions of the Grit Scale, which measures trait-level perseverance and passion for long-term goals." So what's the deal?

I suspect grit is shunned as an aptitude test not because it is un-measurable, but because it is game-able. That is, if NFL scouts started judging players on how they rated themselves 1-10 on perseverance and passion, the players would all give themselves 10's on everything, except maybe one or two 9's to maintain some semblance of honesty. With millions of dollars on the line, wouldn't you?

Still, it does seem to me that you could measure grit in an afternoon, if you wanted to. You'd just have to test it when the player doesn't suspect she is being tested.

Three Cool Active Ideation Innocentives

1) A Communication Platform to engage the “Hidden Community” of Family Caregivers. Link. Reward: $8,000, Deadline: 5/2/11, Description: There are more than half a billion people taking care of someone elderly at home worldwide and the number is growing. Most of these dedicated "at-home" caregivers are not professionally trained to deal with such things as dementia, personal hygiene, medical conditions and complications. Our investigations lead us to believe that this "Hidden Community" would benefit greatly from educational materials, product information /recommendations and established healthcare techniques. We are looking for a "communication platform" to reach out to these individuals to provide educational information and respond to feedback to meet their needs.

2) Educating About the Importance and Acceptance of Purifying Drinking Water. Link. Reward: $5,000, Deadline: 4/27/11, Description: [This org] strives to bring clean, safe water to people in developing countries. With this Challenge they would like suggestions for addressing one of the biggest problems they encounter in this process – namely, that of educating illiterate populations about the importance of purifying drinking water.

3) Humanitarian Air Drop. Link. Reward: $20,000, Deadline: 5/2/11, Description: Humanitarian food and water drops can only be done over an unpopulated drop zone because there is danger of falling debris to people below. We are looking for an alternative way to drop large amounts of Humanitarian food and water packages from an aircraft into populated areas such that there is no danger of falling objects (i.e. non-food items) causing harm to those on the ground.

If you have any ideas on these, write them up and make money!