Sunday, July 27, 2008

Inferring Excellence from the Best

There are two ways to build an atheoretical statistical profit-making machine. In one, you look at a bunch of inputs--past returns, earnings misses, financial statement ratios--and then group into bins, and look at future returns. In another, you look at the best and worst performing stocks, and then try to identify their characteristics. I find the latter approach rather poor, because let's say you find that stocks with really high returns tend to have ratios of Accounts Payable/Sales that are above average. It may be that this ratio is associated with greater volatility, so that it generates both massive success and failure. From a portfolio, average, or expected basis, it is unclear whether this is good or bad. That's all an empirical issue, so you might as well be doing analysis from explanatory data to outcome from the beginning.

So I heard about this new book, Who Are You and What do You Want? and the authors noted they talked to all these really successful people, and then looked at their characteristics, as if this was a really great way to discover great habits. I'm not convinced. Say, you find that being really individualistic makes Steve Jobs a great CEO. What about all the other big dreamers? Perhaps, statistically, this is a bad strategy because for every Steve Jobs who happened to have an Apple, there are millions of cubicle drones who are despised and unrewarded for their seeming impracticality.

The problem is, 'success' is difficult to measure, so we have a tendency to choose people who are super successful, so that there is no debate over whether they are successful. Unfortunately, then the sample size is so small and skewed it's not valid for identifying a pattern.

But then again, the authors run a huge business consulting company that involves flattering CEOs so they send their subordinates to their little retreat. I suspect that's one of the main reasons it is a common way to structure such a book. The idea that interviewing CEOs is a way to find wisdom is very convenient.


Acad Ronin said...

As your penultimate paragraph points out, the problem with looking only at the successful, however one defines them, is not that the sample size is too small, it may or may not be small, but rather that one is selecting on the dependent variable. One needs to have both succesful and unsuccessful parties, and then find what variables one has chosen in advance, discriminate between them, if any do. If one data mines looking for discriminating variables, one must then test them on a hold-out sample. Otherwise, all one gets is best-selling but vacuous business books.

Anonymous said...

Another argument you could make to support your point is not just the sample size, but the inherent un-representativeness of outliers.

Anonymous said...

Yet another way to motivate this idea is with Bayes' theorem, pointing out that P(A|B) is not the same as P(B|A).

J. said...

...One needs to have both succesful and unsuccessful parties...

Of course. We meet the unsuccessful every day and everywhere, so we are all too familiar with its characteristics. we ourself may be of one of the unsuccessful parties, chances are we are. We dont need a counter-report documenting failures. That we can do it by ourselves.

On the other hand, successful people is exceedingly scarce, we may never meet a CEO for a long soul searching. What they say is intrinsically interesting.

Personally, I have met some successful people (nothing special, but I have lived long) and they have no idea how and why they made it. That is why I think Eric has a point, not because of the statistics.

Anonymous said...

again, confirmation bias, covered in the Black Swan.

karl k said...

What's "success?"

Seems to me the guy who is running a $3 million revenue tool and die factory employing 25 people for the last 10 years and netting for himself a couple hundred grand a year is a "success."

So too the guy who toiled for 30 years in a cube for Lockheed who has $750K in his 401K and his house paid for.

99.9% of the planet's population would DIE to be in those situations. It's not just about Bill Gates, Steve Jobs, and Warren Buffet.

Meanwhile, if you look at the super successful folks, there are some common threads.

--See potential of other ideas, get them cheap, and make them work (Gates, Jobs)

--Get lucky on your first deal (like the PE guy profiled in CNBC whose first deal netted him $15 million, but could have easily cratered).

--Take a simple formula in a business, apply it ruthlessly when you have the foundantional resources to do it (Welch, at GE).

Finally, it helps to be intelligent. Stupid people tend not to be successful, Forrest Gump notwithstanding. It's a combination of luck, opportunity, and hard work.