That decentralized, self-interested, people can collectively make such large errors seems irrational or corrupt to many, but they should remember that growing economies require people to be making things better, which means, new ways of doing things. New ideas are often wrong. Economics has gone onto intellectual cul-de-sacs many times (socialism, Keynesian macro models, input-output models, Hilbert spaces in finance, Arbitrage Pricing Theory, Kalman-filter macroeconomic models, etc.). Other scientific disciplines have their own mistakes, and political mistakes--stupid wars--are also common. These are rarely conspiracies, but rather, smart people making mistakes because the ideas that are true, important, and new, are really hard to discern, and tempting ones are alluring when lots of other seemingly successful people are doing them.
My Batesian Mimicry Theory posits that recessions happen because certain activities become full of mimics, entrepreneurs without any real alpha who got money from investors looking in their rear-view window of what worked and focusing on correlated but insufficient statistics. For example, people assumed a nationally diversified housing prices would not fall significantly in nominal terms, because they had not for generations; people assumed anything related to the internet would make them rich in the internet bubble, conglomerates would be robust to recession in 1970, that the 'nifty fifty' top US companies had Galbraithian power to withstand recessions in 1973, that cotton prices would not fall in 1837, etc.
As in ecological niches, there is no stable equilibrium when mimics arise to gain the advantages of those with a real, unique and costly, comparative advantage. Every so often there are too many mimic Viceroy butterflies, not enough real poisonous Monarch ones, and a massive cataclysm occurs as predators ignore the unpleasant after-effects and start chomping on all of them. The Viceroy population grows until this devastating event occurs, a species recession. Next time, it won't happen in butterflies, but rather, among frogs or snakes. They key is, some ecological niche is always heading towards its own Mayan collapse (distinct from the 2012 Mayan apocolypse).
The key to wealth creation is doing more with less--destroying jobs at the micro level and creating jobs at the macro level by reallocating capital and labor to more valuable pursuits. The computer got rid of things from typesetters, secretaries, to engineers working with slide-rules, but these people didn't stay unemployed, they did something else, making the economic pie bigger. This is antithetical to government and unions who think creating a permanent 'job' creates productivity--stability at the micro level and stagnation at the macro level. Wealth is created by having decentralized decision-makers focused on simple goal of making money, which means, they oversee transactions where revenues collected are greater than expenses paid. If externalities are properly priced (I know, most liberal think this never happens), this implies value is created. The continual improvements in method (ie, productivity, wealth creation) merely maintain profits in a competitive environment; to do nothing would see their profits eaten away by competitors would could easily copy what they did and just undercut their prices.
The key to this is having managers who keep their workers focused. A good example is a story I heard second-hand about a football player for Minnesota Vikings in the 1970s. Coach Bud Grant called this marginal player into a meeting, and said, 'Here's what I need you to do...'. The player, an articulate fellow quite confident in himself, interrupted with an explanation of why he wasn't doing better and suggestions about how to correct it, mainly focused what others were doing wrong. Grant cut him off: 'You don't understand. This isn't a negotiation. Do what I'm telling you, and you have a role here. Otherwise, you don't.' Hierarchies only work well when people have clearly defined goals, and managers who manage their direct reports singlemindedly.
Private firms can do this much more quickly and often than government, and are rewarded with investment and retained earnings to the degree they do it well. When the government wants to do something, like build a light-rail system, it instead satisfies all its stakeholders who have no financial downside, only veto power, and so the cost/benefit calculus is almost irrelevant. The probability that benefits will outweigh costs when not prioritized is negligible, as highlighted by the fact that companies have to work very hard to make this positive when all those other considerations are ignored.
Thus, Minneapolis's light rail, at the cost of $1.1B for 12 miles of track, takes me longer to go downtown than a car because it stops 19 times at places no one wants to go because these 'hubs' were then sold as development opportunities, and an unusual number of ex-city councilmen are part owners of coffee shops and stores near these stops. Ridership does not even cover their marginal costs. It could have worked if they had an express train that went non-stop from end to end, but doesn't because it was not designed with the goal of making money, only the hope.
Good companies like Facebook, Apple and Google, have this sense of really understanding their users. Lots of simple things that making going to their sites and getting what you want. Their inferior competitors are relatively ugly, cluttered, and clunky. These generally weren't genius ideas like the ideas needed to create the first transistor, or Cantor's diagonal argument, in that their competitors had similar raw competence in these field, but it did take people looking to do things better than others, and decisive people who could empathize with their customers created really great things.
Robin Hanson had a neat article about the Myth of Creativity, where he criticizes Richard Florida's vision of bohemian lead productivity:
This is a Star Wars vision of innovation: "Feel the force, Luke; let go of your conscious self and act on instinct." And it is just as much a fantasy as that celluloid serial. Innovation is no more about releasing your inner bohemian than it is about holding hands, singing Kumbaya, and believing in innovation.
In truth, we don't need more suggestion boxes or more street mimes to fill people with a spirit of creativity. We instead need to better manage the flood of ideas we already have and to reward managers for actually executing them.
Sure, it's good to punish fraudsters, and be wary of the stupid ideas that were passed off as brilliant in the prior cycle (eg, Angelo Mozilo winning the American Banker's Lifetime Achievement Award in 2006, celebrated by politicians on the right and left, prized by Fannie Mae, and Harvard, is now an example of the 'unregulated predatory private sector'). But this is like learning not to put one's hand on a hot stove--good to know, but old news to most. Our priority at the top level should be to get out of the way, and so government should focus on its essential but limited perennial tasks as opposed to creating some new engine of growth. Leave that for the millions of people making sure millions of small changes are constantly made to daily procedures. Such changes do not require vision from politicians, subsidies, or tax breaks, but are rather the natural by product of people trying to make a buck. It's the standard Hayek/Friedman view of macroeconomics, and it's still the best description of how the complex adaptive system of our economy works.