James Heckman won a Nobel Prize for his work on econometrics, statistics applied to economics. His latest work on education looks at the effects of programs on human capital.
In a recent Boston Review article on social mobility he highlights the results from two experiments in early childhood intervention that demonstrated significant benefits.
Charles Murray was one of several comentors Murray noted these programs were small, having about 60 kids in each, and so are probably random outliers among the many different programs being conducted. After all, one really great teacher undoubtedly can make a difference in such a small sample, but really great teachers, by definition, aren't easy to replicate. Heckman, as is his wont, responded rather angrily that
As Einstein noted, "common sense is nothing more than a deposit of prejudices laid down in the mind before you reach eighteen." That a great econometrician could dismiss the clear selection bias in a couple of 60-kid studies selected out of hundreds (thousands?) highlights that no amount of education or intelligence can overcome one's prejudices, or overcome one's common sense.
Charles Murray mischaracterizes the quality of the evidence on the effectiveness of early childhood programs. In doing so he suggests that my evidence is highly selective. The effects reported for the programs I discuss survive batteries of rigorous testing procedures. They are conducted by independent analysts who did not perform or design the original experiments. The fact that samples are small works against finding any effects for the programs, much less the statistically significant and substantial effects that have been found.A small sample will have more trouble demonstrating statistically significant results--it has low 'power'--so Heckman is technically correct. But it's not as if these two programs were the only ones generated since 1962; these are really order statistics, not simple statistics. I see job seekers with fabulous backtests all the time, and cherry picking winning algorithms applied to a large class of rules is the most common problem.
As Einstein noted, "common sense is nothing more than a deposit of prejudices laid down in the mind before you reach eighteen." That a great econometrician could dismiss the clear selection bias in a couple of 60-kid studies selected out of hundreds (thousands?) highlights that no amount of education or intelligence can overcome one's prejudices, or overcome one's common sense.
13 comments:
Proposition 1: Conditional on the sample size being small, the chance of a statistically significant finding for a given effect size is less than if the sample were larger, because power is lower. (This is what Heckman says.)
Proposition 2: Conditional on the effect size being small or non-existent, a small sample size is more likely to find a spuriously large effect, because of sampling variability.
So what do we make of the fact that the only studies to find such a large effect are tiny (compared to the Head Start studies, for example)? This seems more consistent with the "sampling variability" story than with the "HUGE effect size" story. Heckman conveniently fails to discuss sampling variability, though.
It's fantastic that you are getting ideas from this paragraph as well as from our dialogue made at this time.
my web site: erinmore
Presumably the 60 were not picked at random. The validity of the results depends of how they were chosen. If the selection algorithm is acceptable, then Heckman may be more than technically right. He may be right.
PS: You seem to take for granted a priori that there was a "clear selection bias". It's not necessarily so.
J: that's because there have been hundreds if not thousands of such programs since 1962, the rest of which did not work. 2 out of 100 will reject the null that they don't work at the 5% level.
I see.
The triumph of hope over reason. Because this "Hope" resonates with the prevailing biases of our time such a bias is in this case a good prestige enhancer.
Without even bothering to click on the link I’m going to guess that this article (and others about the study) conclude with something like:
“In light of this important study it is now abundantly clear that government must do (spend) more in the area of early childhood education if this country is to have a workforce capable of competing in today’s dynamic and highly competitive global marketplace.”
Just for the dummies (like me) you are saying that Heckman should have built a big meta-study to boost up the numbers and thus achieve a more meaningful measurement?
That's still work to do, yes?
Of course Heckman would realize that if hundreds of small studies show no effect but a few do, then the evidence is not there. I can't believe Heckman is makes the claims he does, knowing that there are many more studies that show no effect than there are studies that do.
Greetings from Colorado! I'm bored to tears at work so I decided to check out your website on my iphone during lunch break. I love the knowledge you provide here and can't wait to take a look when
I get home. I'm amazed at how fast your blog loaded on my mobile .. I'm not even using WIFI,
just 3G .. Anyhow, fantastic blog!
My website: make money online clickbank
I tend not to comment, however I browsed a few remarks
on "Great Minds Confabulate Like Small Minds".
I do have some questions for you if you do not mind.
Could it be just me or do a few of these remarks come across like they are coming from brain dead individuals?
:-P And, if you are posting at additional sites, I
would like to follow anything fresh you have to post. Could you make a list of the complete urls
of all your public sites like your twitter feed, Facebook
page or linkedin profile?
Here is my site ; amphora tobacco
Hello it's me, I am also visiting this web site daily, this web site is in fact fastidious and the visitors are truly sharing pleasant thoughts.
Here is my site :: van nelle tabak
Post a Comment