Noneconomists tend to think 'rational markets' is patently absurd, pointing to various asset bubbles such as the internet bubble, the recent housing bubble, or the 1987 stock market crash. That is, most people think extreme events are evidence against rational markets.
Malkiel and Shliefer once took opposite sides of the market efficiency debate in the Wall Street Journal. Dutch and Royal Shell trade on two separate exchanges. These different listings by law apportion a 60:40 split of cash flows and thus should trade as such, but indeed they vary by as much as 30% from this fundamental equivalence. Shleifer calls this a ‘fantastic embarrassment’ to the efficient markets hypothesis, yet Malkiel also notes it as being within his bands of reasonableness. To me, this highlights that much of this debate gets into semantics, and such debates are rarely fruitful (Wall Street Journal, 12/28/00).
To assert markets are irrational or inefficient, however, one needs to propose a measure of 'true value', and then show that actual market prices diverge from this. As classically worded by economists, any test of market efficiency is a joint test of a market model and the concept of efficiency. Thus, your test may merely be rejecting your market model, not efficiency. You may think this is unfair, but it's simple logic, and you have to deal with it. It is essential to have a specific alternative, because how do you know they are wrong unless you know the right answer? With hindsight, prices that were once really high, now not, were 'wrong', but one has to be able to go back in time and show the then-consensus was obviously wrong. Thus, if you propose, say, some metric of P/E, or dividend payout ratios, that is fine, but then presumably there will be some range of P/Es that, when breached, generate inevitable mean-reversion thus demonstrating the correctness of the P/E ratio. Actual arbitrage, in the form of strategies that generate attractive Sharpe ratios, are necessary, and this is very hard to do.
One big issue in tests of whether prices are 'right' or not is the Peso Problem. The term 'peso problem' has a long history, and I have seen the term attributed to several people, in any case it was first applied to the fact that the higher interest rate one received in the Mexican Peso for decades, was erased in a single day in 1977 when the Peso was devalued by about 45%. In 1982 Mexico did it again. Thus, decades of seemingly higher returns could have merely been the expected probability and size of these devaluations. As these probabilities are small, they are often not seen 'in sample', and the standard errors on these probabilities are sufficiently high that it is very difficult to see if they are sufficient to explain, or even over-explain, a certain return premium. That is, when you have a 1% or 5% chance, annually, of a 75% depreciation, the appropriate offset is 0.75% or 3.75%, a big difference.
Tom Rietz used the 'peso problem' 1988 to explain the anomalously high equity premium puzzle, then estimated around 6%. Big events aren't anomalies, but rather explanations in the rational markets paradigm. Recently, Robert Barro noted that historically, there has been about a 2% chance of a 15% to 45% GDP decline, which would probably cause equity markets to fall 90%. The implication is that many return premiums are really a mirage. Further, volatility is totally rational, not too high, because reasonable, rational people will disagree as to the specific probability, and as they move from a consensus of a 2% to a 5% probability of disaster, the price fluctuates wildly.
Such events are not proof against efficient or rational markets, but rather, supports it, because estimating the probabilities of these important events is clearly very difficult. A rational market should move a lot as people change their estimation that, say, the next Microsoft is extant in a set of internet stocks (with potential future market cap of $200B), or that a worldwide Depression is likely. The Peso Problem literature goes back to the 1980's at least, and fits within the rational market approach as one of the main reasons things that appear anomalous actually are rational. If you think extreme events invalidate rational markets, this implies one has a lot of certainty for the magnitude and probability of highly improbable events, which is not very compelling (eg, what is the probability of a second leg in the current financial crisis? 1%? 10%? 50%?).
Andre Shleifer and Larry Summers once wrote that “[i]f the efficient markets hypothesis was a publicly traded security, its price would be enormously volatile" —-too volatile, supposedly. Presumably Shleifer and Summers think economists are rational and understand that the rational consensus around a proposition can and does vary wildly around the truth. So why can’t market participants also be considered rational and yet have their collective opinions vary wildly over time and space? Truth is a very slippery concept, and whatever it may be for various propositions, it is something reasonable people can often agree to disagree, in aggregate and at different times.
In 2001 the New York Times had two articles by different authors on behavioral economics. The story was a cliche: a stolid conventional wisdom experiences a Kuhnian shift, lead by a small band of outsiders willing to flout traditional ways. The behavioralists reject "the narrow, mechanical homo economicus" and instead argue that " that most people actually behave like . . . people!" One articled noted "Some Economists [the behavioralists] Call Behavior a Key", implying that previously economists never were concerned with 'behavior'. This straw-man smack down has continued in the financial press to this day, and meanwhile, there are no canonical models of asset pricing based on behavioralist insights, merely explanations for well-known anomalies like momentum, size, and value, that were documented outside this literature.
Danny Kahneman, co-author of the Behavioralist Bible Judgement Under Uncertainty: Heuristics and Biases (a book published in 1982 about work mainly from the 1970s) went on to win the Nobel Prize in 2002. Herbert Simon, won the Nobel Prize in 1978 for his insight that humans have limited computing power, and so often satisfice in their optimization. In my dissertation back in 1994, I had to put 'behavioral economics' is scare quotes, but it has been part of conventional wisdom for at least a decade now. It's all grown up now, and shouldn't be judged on its potential anymore. One should apply behavioral biases to market 'data' (not anecdotes, or highly parochial experiments).
Crashes are interesting, but people's obsession with them highlights the hindsight bias more than a real-time, generalizable bias. Prices fluctuate more than we would like. But is it too much? The future is very uncertain, and in the US where so many prominent financial researchers work, we tend to forget we had a very fortunate 20th century (2-0 in World Wars!). Looking at history, where many countries have seen their equity indices get zeroed out (Hungary, Russia, China, Chechoslovakia, Poland), and some centuries are peaceful(13th in Europe) others horrific (14th in Europe), who's to say whether the stock market should be twice, or half, its current level with that sort of state space.