Tuesday, June 29, 2010

Is Peer Review Sufficient or Necessary?

George Monbiot, the British climate-change alarmist, once wrote of a skeptic: ‘I accepted and floored him with a simple question: Has he published his analysis in a peer-reviewed journal?’, as if this were a truth-seeking silver bullet.

Is peer review necessary for one to have an opinion on something important like tax cuts, climate change, the war in Iraq, etc? The book The Hockey Stick Illusion highlighted that Einstein's paper on special relativity, and Watson and Crick's paper on DNA, were not 'peer reviewed', but somehow managed to be apex of scientific insight. So, peer review is not a necessary condition to presenting a new, true, and important idea.

The peer-reviewed journal process is a good thing, but it's still merely a means to an end: highlighting the good ideas, argument, and data. There's no royal road to this end. I review a couple papers a year for second-tier journals, and often I get to read the other referees comments in the second round, and usually find the other reviewer to be an distracted person who meticulously checks some things (grammar, math, the data, the big picture), but then ignores other dimensions. This is only natural, because often the papers are complicated, and he (or she) is counting on the other reviewer. But, what if I didn't check his empirical results with data I have unique access to (say, on corporate defaults)? Would anyone know? If you know what buttons to push, especially by anticipating the kinds of people who will referee your paper, you can game the system to a large degree.

Peer reviewed articles have been found with fraud (Hwang Woo-Suk and stem cells), errors, but most often, irrelevance. The latter point is especially important, because like grade inflation, as professors need X peer reviewed papers for tenure, there are ever more journals being created to satisfy this criteria. Alas, Sturgeon's law ('90% of everything is crap') holds, and so it becomes harder and harder to find the truth in a sea of irrelevant papers.

In Karl Popper's the Open Society and Its Enemies he wrote that 'Institutions are like fortresses. They must be well designed and manned.' Unfortunately, there are always too few good people to go around. The peer review process is a good thing, but it is still a flawed process, rewarding ritual over substance by the way certain things are treated non-skeptically if you use the rhetoric popular amongst the journal gatekeepers (eg, applying GMM to an equation, or Sokal's hilarious post-modern bunkum).

As Alexander Cockburn notes, 'many cite peer-reviewed science because they're afraid to have the intellectual argument.' Just about any hypothesis has some esteemed person and publication supporting it. If you then go to the 'majority of experts' route, you need a strange faith in the majority opinion (remember, before 2007, most regulators, bankers, investors, and academics believed that lower underwriting standards such as no income verification for home ownership were innocuous if not morally righteous).

12 comments:

Unknown said...

Looks like (in part) a response to some of the comments from yesterday. Well argued, sir.

Dave said...

Maybe this problem will ameliorate itself. If, as you note, much of this is driven by academics forced to "publish or perish" -- and if the current state of higher education is a debt-fueled bubble -- then the bursting of the bubble should lead to an ebbing of weak or irrelevant academic papers as a sea of Ph.Ds is forced to seek non-academic employment.

Drewfus said...

Who will watch the watchmen?

Who will regulate the regulators?

Who will review the peer-reviewers?

One way society has developed to avoid the infinite regress implied in these cases is to simply regard those at top of the respective hierarchy as inherently well intentioned - they don't overseeing.

Similarly to the case of God - no one is required to keep a good eye on the almighty - he is good in all respects by definition. Problem solved.

Anonymous #5 said...

One way society has developed to avoid the infinite regress implied in these cases is to simply regard those at top of the respective hierarchy as inherently well intentioned - they don't [need] overseeing.

On the contrary, our society tends to look for the right balance between hierarchy and openness. In academia for instance, the presumption is that although gatekeepers are naturally inclined to prefer their own favored ideas, the system is sufficiently decentralized and the rewards fpr changing the dominant paradigm are sufficiently high that good ideas will win out in the end, even if they start out unpopular.

Now this may or may not work. Still, it is simply not true that the system's legitimacy is predicated on complete trust in those on top.

I have to say, the idea that climate-change-skeptics have truth entirely on their side but are nevertheless unable to generate any traction whatsoever within academia... this seems to be a very, how should I say it, Marxist critique of science within academia. "You will learn about all the injustice and inefficiency, and then naively assume that some utopian alternative is eminently feasible if we could only get the cabal of oppressors to stop manipulating the system."

Drewfus said...

One way...

Drewfus said...

...the idea that climate-change-skeptics have truth entirely on their side but are nevertheless unable to generate any traction whatsoever within academia...

That is a perfect example of what i mean. Why is the popularity of a theory amongst academics a good proxy for the truth? Only if you assume that these guys are quietly working away to discover the true nature of the physical world in a completely disinterested way. Very much like the wise men of The Republic, they're not in it for their own glory, and certainly not for taxpayers money.

Of course the climate academics aren't the grubby, competitive, status seeking, politically inspired employees of Socialist Science that the sceptics have made them out to be. They are simply not like the rest of us. Their intentions are all good.

Anonymous #5 said...

Why is the popularity of a theory amongst academics a good proxy for the truth? Only if you assume that these guys are quietly working away to discover the true nature of the physical world in a completely disinterested way. Very much like the wise men of The Republic, they're not in it for their own glory, and certainly not for taxpayers money.

Oh OK. Why should I believe that anyone anywhere is telling the truth? Why should I believe that climate-change skeptics are telling the truth? I guess it is your opinion that climate-change skeptics are completely disinterested, that they're gods among men, that it would be better to murder a village than to cause them one minute of suffering, OMG I've totally worked myself up into a frenzy at the injustice of it all!

If you read my comment with any degree of care, you can see that I am hardly assuming that all scientists are angels. On the other hand, if you believe that climate-change skeptics have truth on their side but are *completely unable* to gain traction in academia, then you are saying that there is a massive conspiracy within academia, that there is such pervasive groupthink that infects virtually everyone who is part of the club. And this is indeed a Marxist-type of explanation ("it's all the evil oppressors' fault") that Eric is generally fond of dismissing as ridiculous.

Anonymous #5 said...

1. Climate scientists aren't heavenly angels.
2. Climate scientists are engaged in a massive conspiracy that prevents the appearance of any truth-based articles in peer-reviewed journals. If they weren't such blind ideologues, the peer-reviewed journals would look completely different.

If you want to argue #2, go for it. Stop pretending that you're just arguing #1 and it's really absurd that anyone disagrees with you.

Drewfus said...

The sceptics aren't wrong, the problem is that the customer is always right.

The governments of the world want catastophic climate change to be true, so that is what Socialist Science is being paid to produce.

No conspiracy, just demand and supply.

Anonymous #5 said...

Oh right, because the government is run entirely by Socialists. And if you don't want to believe in climate change, then you have no chance of getting elected to higher office in America. After all, the proposed solutions to hypothetical climate change are remarkably popular and easy to sell to a broad audience. Even if they weren't, everyone knows that you can't win elections without raising money, and you can't raise money without making the Big Hippy lobby happy.

I would like to subscribe to your newsletter, I've never seen someone who cared so much about empiricism.

Alphy said...

I don't see the date on this post, but it is post-Climategate, right? Didn't those e-mails give hints that peer-reviewed papers would be given pressure to reject "non-consensus" papers?

I would also add that it's not too uncommon for a given group of people to become convinced that something is true, and reject anything that conflicts with what they accept is true--even if it has some sort of merit.

gappy said...

I mostly agree with the content of the post, but want to point out a factual inaccuracy. A. Sokal's "bunkum" is many things but not an indictment of peer review. Sokal's sham paper was submitted to "Social Text", which is not peer-reviewed. Check out his book Impostures Intellectuelles, or the material on his web site.