Monday, June 07, 2010

Dishonest Forecasters


"Econophysicist Accurately Forecasts Gold Price Collapse" says the MIT Technology review. Didier Sornette seems to generate a lot of buzz, because the whole idea that something called 'complex-systems theory' can predict prices seems like a neat trick. His website notes a lot of press coverage, including Nature and the Wall Street Journal. Wikipedia states that 'complex systems theory' is "used as a broad term encompassing a research approach to problems in many diverse disciplines including anthropology, artificial intelligence, artificial life, chemistry, computer science, economics, evolutionary computation, earthquake prediction, meteorology, molecular biology, neuroscience, physics, psychology and sociology." A theory of everything indeed.

I find these models generally look like the multiplier-accelerator model that Paul A. Samuelson introduced back in 1939 because it seemed possible of mimicking the boom and bust nature of economic time series. That thread died because it was barren, but it was born of the same intuition: it looked like it could explain the future, because you could fit it to the past.

Econophysicists have always been on the cusp of some fantastic breakthrough, but these researchers seem to have a very selective memories. Sornette's December 2002 call for a stock market crash in 2003-4, could not have been more wrong, and it seems to have been put down the memory hole, nowhere on his website. Anyone who's even a little dishonest is not to be taken seriously, because once you start selectively picking from your prior forecasts, you generally don't stop.

A professor forwarded me some econophysics paper he co-authored that he thought was a paradigm shifter. It was basically some model that generated chaos, with some positive feedback loops, and I asked about some technical portion of it, he replied he didn't know about that, but his physics co-author did. This suggested to me the mechanism itself was not important, but rather, it generated the right big-picture pattern. How naive. If I want to generate a model of the stock market, I could rationalize a model of geometric brownian motion, with some Poisson jump process, perhaps some auto-regressive volatility, and say that looks like the S&P500, but that's hardly a model, just a description. It doesn't predict anymore than the rand() function predicts lottery numbers (they look like lottery numbers--very random!).

I remember the Stock-Watson model that was to replace the old Leading Economic Indicators in predicting recessions. Prior to the 1990 recession, it was generating a lot of buzz, because it used a Kalman filter, just like real rocket scientists, and seemed obviously better than the simple sum of ten correlates with business cycles. Yet the model failed to predict the 1990-1991 recession, and an updated version of the model then failed to predict the 2001 recession. Each time, however, Stock and Watson were very honest and clear about the degree it failed and why (see here). It's rather straightforward what they did, but remarkably rare for such prominent forecasters. Further, reading this research, I feel I learned something, which I never have done from the econophysicists (other than they don't realize that perfect competition, or Gaussian distributions are not necessary assumptions to most economic theories, see here).

5 comments:

Charles Krohn said...

Actually, I think Sornette is a pretty honest forecaster. I like his method of using a hash code, so he can keep his predictions secret for a time period, but still prove that he made them well in advance. The Financial Bubble Experiment papers are all pretty well publicized, and he's openly admitted the forecasts he got wrong. Furthermore, he publishes refinements to his methodology pretty regularly on Arxiv.

vJD said...

When I look at the outcome of the forecasts I get the feeling that his forecasts are right - but you can't use them anyway. If you short the asset he forecasts to crash it only crashes to a point which is *above* the original point you bought. So it is more rational to ride the bubble and not to short it (see also here: http://www.cxoadvisory.com/fundamental-valuation/bubbles-ride-watch-or-play-the-pop/ )

But this is only a preliminary assessment - I think you should watch how it all develops, perhaps there is some method to the madness...

DaveinHackensack said...

vJD,

I made a couple of bucks last year buying puts on GLD after reading a commenter I respected write that Sornette's formula predicted a crash. Gold didn't really "crash", but it corrected enough to make for a nice return those GLD puts.

More recently though, I've wondered about the exact same thing you mention above: whether it might make more sense to embrace bubbles than bet against them. My one twist is I think it might make sense to hedge while investing in bubbles.

Anonymous said...

Depending on the timing, one could have easily made -- or lost -- substantially on either the long or short side of gold over the past twelve months.

Michael F. Martin said...

Thanks for the link to the Samuelson work. I figured somebody, at some time must have tried to model demand as a damped, driven oscillation. Too bad there was no reliable way to measure any of those variables.

If you will permit him to be called one, Samuelson is the best of the econophysicists. I admire the way he let analogies to physics inspire his work. His followers and interpreters are more to blame than he for the bad consequences of neoclassical theory.