Saturday, July 05, 2008

When Theory Fails

Horgan and Johnson have a neat dialogue, as usual, over at Blogging Heads.tv. Horgan notes the Tetlock's book documented that experts did worse than chance at predicting major events, relative to a naive alternative (discussed in this good New Yorker here). The idea is that experts are incented to create innovative theories, and the more they know, they better they become at defending, and cherry picking the evidence to support it.


Another example of this is an experiment discussed in the video clip above, where rats were presented with a maze where they were presented food 60% of the time on the left. Quickly, the rats would always go left, successful 60% of the time (smart rats). In contrast, Yale students were presented with the same puzzle but were only successful 52% of the time because they were trying to guess complex patterns. Theory can be an impediment when you infer theories that aren't there.

3 comments:

gabe said...

or maybe just another useless experiment done by academics to get their quota of published papers? what is the rat's utility in finding (not) the cheese vs. yale students' being right?

i have yet to see a study showing how behavioral studies can be profitable applied in the market. yes, we're dumb and make dumb choices. i get it. so? time to move on and make something useful out of all those research grant money.

you gotta give it to that little bastard though.

Anonymous said...

I have seen something similar in profitable trading ideas. Certainly, if profits rather than accolades from a professional group or use as a marketing tool is the goal, the ideas will look quite different. Looking smart often seems to be more important than making money.

J. said...

Cute experiment. Did it demonstrate that lab rats are more intelligent than Yale students? or that rats are motivated by plain cheese while students prefer cheeseburgers? Maybe Yale students are biased against the left? Maybe the rats participating in the experiment were left-handed? More studies are required.