Thursday, October 29, 2009

The Scientific Conceit


This interview with David Berlinski contains this precious observation:
The idea that science is a uniquely self critical institution is of course preposterous..scientists are no more self critical than anyone else, they hate to be criticized and never criticize themselves...There are local mechanisms of criticisms in science, within established theories if somebody publishes data that don't work out in a certain way, if there are mathematical flaws in a certain theory, these tend to get know, but large global criticisms of the scientific enterprise are very difficult to find, and certainly not being promulgated by the scientists with any ebullience or enthusiasm ... these people are only human, they hate criticism--me too! The idea that scientists are absolutely eager to get beaten up that's one of the myths, put out by the scientists, and it works out splendidly so that they can avoid criticism.

You know a naive or tendentious science writer when they start talking about how science is so different than other professions in how they objectively present their work for criticism. The journal publication process does filter out a lot of errors and unsubstantiated assertions that a journalist might get away with, but that's really a very narrow domain, it's like noting a New Yorker piece is meticulously checks for grammar (unlike my blog!). It's ruthless criticism in a very specific domain.

Look at Freakonomic's author Steve Levitt's work on the abortion-crime link. No referee thought, gee, how does this relate to the different black-white abortion rate relate to the difference in Black-White crime rate over the next 20 years? How did that relate the crime rate differential between 17 year olds and 35 year olds, 17 years after Roe? How does the small difference in fertility post Roe relate to an selectivity effect (clearly, as abortions went up, so did conceptions)? What if you adjusted for population growth? All of these are large, glaring points against Levitt, really common sense type rebuttals, yet he never had to address them because he presented a panel regression with interaction terms.

I have refereed papers with interaction terms, and they are almost always garbage, because they generate a lot of collinearity, and the nonlinear correlations manifest themselves in a bunch of significant coefficients in linear regressions, because if the 'true' relation is y=x^2, and x=A+2B+e, then a regression on y with A and B will have a positive coefficient on A, and a negative on B. Levitt's study has state*age, year*age, and state*year as explanatory variables. Why stop there, why not state*year*age? With tens of regressors, many product terms, you get garbage. Yet, I've refereed reports from professors at Harvard with this stuff, so it's not something that's necessarily wrong, just practically stupid. There has been no important result that shows up only via interaction terms in regressions, just as there has never been an important relationship evinced solely via 3-stage least squares, or the Generalize Method of Moments (GMM). Ever. Rather than correcting error terms for heteroskedasticity or something inside-the-box, there should be more skepticism applied to these kitchen-sink approaches.

But, Levitt is still considered a top-level researcher, and he has a very thin skin. For example, reading his coauthor's response to criticism of his Global Warming chapter in their new book, the tone seemed very defensive, like someone unused to criticism. In sum, Levitt, like most scientists, is exposed to a very narrow set of criticisms, ones that most laypeople could not counter to be sure (must get one's standard errors correct), but that's really no different than the fact that most people (me included) could not write prose for the New Yorker with their syntax errors. In most ways he is unexposed to real criticism and acts accordingly.

9 comments:

But What do I Know? said...

I've known a number of scientists, and yes, they really hate to be told they are wrong. Some of this is professional survival necessity--if the lab head is laying a techno-rap on the finance-major manager he doesn't want anything going off-message because he's trying to convince someone with limited understanding of the issues involved to make a decision favorable to him--you know, like a political campaign. Or else in academia, you're not going to make tenure my making "mistakes." Of course, the arrogance and vindictiveness of scientists isn't new--the Mach-Boltzmann tiff is a perfect example of pettiness and mean-spiritedness.

But this is true in any endeavor. Perhaps it would be best to say that good scientists are self-critical, but that is one of the characteristics of good thinkers in general. Also, when one learns the history of science, the first thing one learns is that the great minds of the past were wrong or had an incomplete grasp about many things--that is, scientific training implies that the theories we have are subject to revision, just like those of the past. They don't teach mathematics or politics or finance that way.

michael webster said...

Eric, if you keep educating me for free about statistical fallacies, I may not the compulsion to buy your book!

What do you think of the alternative hypothesis by John Mueller, http://www.claremont.org/publications/crb/id.1008/article_detail.asp, that a concept he calls economic fatherhood explains the relevant data?

Second, I cannot follow your argument from the existence of interaction in a regression model to being skeptical about their uses. Could you flesh it out more, for the regression challenged? I am obviously missing something.

Third, I don't think it is reasonable to expect that all intellectuals ought to respond to criticism in a similar manner. People get to be famous for all sorts of good and bad reasons, and their talents may differ in only minor ways from people farther down the fame pole. It would be unreasonable to expect that the higher up one gets on the fame pole, the more sensitive one becomes to other opinions. Of course, this is just to observe that sometimes fame signals that you have a) jumped the shark, and b) nobody has caught on yet.

Eric Falkenstein said...

Michael: my point, perhaps not clear, is that statistics can all be 'correct' at one level, but irrelevant or misleading at a higher level. The referee process focuses on the lower levels, and relies on a lot of path dependence (emulate famous professor A). Big picture criticisms are for journalists, not journals, so they are generally ignored. This allows a lot of nonsense to get published.

Robert Johnson said...

Steve Levitt is a scientist? I don't know. When I think of scientists I think of physicists, chemists, and the like. Not to say there's anything wrong with economics, but let's just say that it's hard to identify an innovation that has come out of economics that is comparable to light amplification through stimulated emission of radiation.

It's probably really unfair, but I think that it' hard to call it science if it doesn't (at least from time to time) produce knowledge that can be used for practical technology. Some practical technology has definitely come out of economics, so someone in the field must be doing science. But I'm not sure that someone is Steve Levitt.

michael webster said...

@Eric,

Thanks, that makes it more clear for me.

I probably agree with you, and it has to be some sort of irony that path dependent choice functions are generally thought to reflect irrational preferences, or at least non rationalizable preferences!

zen babu said...

For the entire post, you call upon Levitt's work. To show that he is thin skinned, you link to a defensive response by Dubner. Is that fair?

Anonymous said...

It shouldn't really matter how thick-skinned a scientist is as long as his work is reproducible and someone else bothers trying to reproduce it. I don't think that happens anymore, and I suspect the problem is tenure. I don't mind the idea of tenure. I do mind that people who have tenure decide who else gets tenure. Once you get past a certain tipping point you wind up with a club.

Anonymous said...

Eric, You are an Interaction Term bigot; your generalization does not hold up. Interaction terms are very important in all kinds of fields, particularly marketing. For example, consumption of wine is best explained by the interaction of education and income, rather than income and education seperately.

J said...

It is true that too many variables confuse the cause-effect logic and "reveal" all kind of bizarre and meaningless associations. You must have been angry when dismissing them in toto.