Sunday, July 17, 2011

Predicting Tsunamis


Prediction expert Phillip Tetlock and his colleague Andy Gardiner have a piece over at Cato Unbound, discussing the failure of experts to foresee this year's events:

Even now, only halfway through the year, The World in 2011 bears little resemblance to the world in 2011. Of the political turmoil in the Middle East—the revolutionary movements in Tunisia, Egypt, Libya, Yemen, Bahrain, and Syria—we find no hint in The Economist‘s forecast. Nor do we find a word about the earthquake/tsunami and consequent disasters in Japan or the spillover effects on the viability of nuclear power around the world. Or the killing of Osama bin Laden and the spillover effects for al Qaeda and Pakistani and Afghan politics. So each of the top three global events of the first half of 2011 were as unforeseen by The Economist as the next great asteroid strike.

Killing Osama, revolution in Tunisia, and the tsunami in Japan are all pretty specific outcomes that were unforeseen. But I would say that rather than this year being really crazy, it is rather similar to last year: US mired in Iraq and Afghanistan, deficits as far as the eye can see, no shovel-ready government spending, inner city America pathetic, Harvard outdoing Mississippi State academically, etc. Predicting broad trends is eminently doable but we often just take those for granted. Indeed, later they point out that experts tend to do worse than simple extrapolation models, which just highlights the futility of predicting outliers. So what should one do: predict specifics, or forecast broad trends that necessarily miss specifics?

When I worked as an economist I remember that the statistically optimal forecast was generally an exponential curve from where we are to the long-run historical average. But that's pretty boring, so we would add some little wiggles at the end based on some theory (A causes B which causes C in 18 months), because that kind of reasoning really resonated with the audience; they wanted to learn some new story to apply to their understanding of the world, something more novel than the future will be a lot like the past. Similarly, Tetlock appears to want tsunami forecasts, surely fun to have.

Later, Tetlock notes that statistical models outperform alternatives, but 'have an unfortunate tendency to work well until they don’t.' This, supposedly, is a deep thought. What Tetlock seems to lament is the absence of perfect foresight, because dominant approaches he finds poor. This isn't a very logical conclusion, because he is making the best the enemy of the good here.

Tetlock is fond of grouping experts into hedgehogs and foxes: 'the fox knows many things but the hedgehog knows one big thing.' The experts with modest but real predictive insight were the foxes, but sometimes not. In an interview Tetlock was asked whether he was a hedgehog or a fox, and replied, 'both!', as if everyone else would not also choose this description, highlighting the futility of his schema. Tetlock greatly admires Nassim Taleb, another who, ironically, criticizes pundits who make lots of nonfalsifiable predictions and selectively report their record. I think this highlights Nietzsche's warning that 'He who fights with monsters should look to it that he himself does not become a monster.' In other words, people who make a life out of criticizing overconfidence, dogma, hate, hypocrisy, are often filled with overconfidence, dogma, hate, and hypocrisy.

It's good to dislike error and vices, but to thirst too much for these points just highlights an intellectual diabetes.

4 comments:

r2d2 said...

"as if everyone else would not also choose this description, highlighting the futility of his schema"

doesn't this rather highliht the futility of asking people what they think of themselves? maybe the "schema" is still of some use when applied to others :)

Mercury said...

If accurately extrapolating broad trends is eminently doable I would say that predicting outcomes based on the creation and rearrangement of incentives in the present runs a close second.

Surely the endgame of the American experiment of extending residential mortgage credit to anyone who wanted it wasn't a surprise to everyone.

Harvard will probably still be trouncing 'Ol Miss in ten years. the question is: what conditions will today's newly minted incentives have created in ten years?

Anonymous said...

In the 1980s economic and financial literature, a big deal was made of the finding that to forecast many economic or financial time series, you cannot beat a random walk: tomorrow's value (or next year's) equals today's value, plus an error term that has mean zero. The best you can do is to study the variance of the error term to build your confidence interval (and use higher order moments if the error term is not Gaussian Normal).

PRCalDude said...

As a diabetic, I'M OFFENDED.

Eric, what should a 401(k) saver read after "A Random Walk?"