FRBSF Economic Letter: Probability of a Recession vs. Actual Recession Dates
You can teach statistics, but unfortunately, you can't teach people not to overfit data. The problem is that it is too tempting to look at some data, keep applying different inputs and functional forms, until you fit the data. In some sense, that's what a good model does, so the objective, maximizing the R2, is encouraged.
But there's no point fooling yourself, it just wastes time, and only the researcher really knows if they overfit the problem, because outsiders don't know how the ultimate functional form was chosen (iterating over a large set of inputs?). Macro forecasting is especially difficult, and anyone familiar with its history would do well to be modest (see here). The above graph from some San Francisco Fed researchers is clearly overfit because the base recession rate is about 16% since 1945, so the average forecast should be around 16%, not jumping from 0 to 100%. A forecast should cluster around the unconditional expectation, not the extremes. Only with hindsight do these kind of forecasts make sense.
9 comments:
That's done with a probit regression. It's much better to do it with Markov Regime Switching. The one period probability is likely going to be low, but the further out you go, the unconditional estimate will rise.
I modeled binary events at Moody's, and the base of the mode forecast was not zero, even though that was the mode event by far...
My first instinct from the diagram is that I want to know about the recession before it happens, not as it happens. So a probability value telling me that I'm in trouble, when I'm already in trouble might be interesting but doesn't have a whole lot of useful applications.
The scientific method requires first gathering observations, then making a hypothesis about those observations, then making predictions based on your model and finally checking to see if the predictions are valid. If you can't make predictions then you are basically still poking around at the hypothesis stage.
@John Hall: MRS models have been around for 20 years now. What was there track record at predicting the last recession?
No you're not right here; this is meant to be an forecast at time T, using data available at time T, of whether at some point T+x the NBER Business Cycle Dating Committee will declare T to have been during a recession. Since most of the time we are either obviously in a recession or obviously out of a recession, you would expect a sensible model of this sort to have the shape it does. You certainly wouldn't expect it to hover around 16% because we have roughly one recession every six years, not one-sixth of a recession every year.
My first instinct from the diagram is that I want to know about the recession before it happens, not as it happens. So a probability value telling me that I'm in trouble, when I'm already in trouble might be interesting but doesn't have a whole lot of useful applications.
It certainly does. It's not obvious at the beginning of a recession that you actually are in a recession, and there are always plenty of people denying that we actually are.
Bruschettaboy: Well, it's certainly not interesting or useful to anyone, and therefore disingenuous, to consider a forecast when in a recession to automatically have a high probability of being correct merely due to autocorrelation. That isn't an interesting metric. Again, don't fool yourself into thinking you have a useful model via such reasoning, because in the long run it's not a useful skill.
Agreed that a predictive model is far better than a model that says "are we in a recession now." That seems fairly useless.
Disagree with Eric's critique in general. A really good recession probability model should be much lower than 16% most of the time but very high occasionally. That this model looks like this doesn't prove overfitting. The model may in fact be grossly overfit. But Eric's critique amounts to "you can't predict recessions" which is uninteresting and circular. This is how it would look like if you could predict them. The model might suck, but not because it looks like it's supposed to.
Good intincts (overfitting is rampant) badly applied.
Well, I think that concealed somewhere there was a tacit admission that the graph doesn't demonstrate overfitting, which is progress.
Further progress! In fact, as they don't make clear in the article but do in the actual paper, they're forecasting two years out, and it's crystal clear that the ability of their model to classify recession and non-recession periods doesn't have anything to do with autocorrelation. As they point out on p15, autocorrelation actually makes a classification problem harder if it reduces the difference between recession- non-recession states.
They even say:
"Nevertheless, LEI forecast trends indicate that the macroeconomic outlook is likely to deteriorate progressively starting sometime next summer, even if the data suggest that a renewed recession is unlikely over the next several months"
Which, as a forecast made in print in August 2010, looks very good.
Anon,
James Hamilton keeps track:
http://www.econbrowser.com/archives/rec_ind/description.html
I think they work well, but it depends on how you work them.
Post a Comment