The issue of reproducibility in science in general, and psychological science in particular, has been getting a lot of attention. Rightly so. After all, the extent to which once can stand on the shoulders of giants depends on the giants having a solid (empirical) base. A new paper out in Science by the Open Science Collaboration has conducted replications of 100 psychological studies and found that only about 1/3 actually replicated the original findings. This result is obviously making a splash: I’ve already seen it commented in the Economist, NYT, Spiegel, and NZZ this morning. There may be a bad news/good news interpretation of these efforts though: On the one hand, it’s troubling that so few studies fail to replicate, on the other, psychologists are taking the reproducibility issue seriously by engaging in collaborative work and adopting open science practices…
Estimating the Reproducibility of Psychological Science
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects (Mr = .197, SD = .257) were half the magnitude of original effects (Mr = .403, SD = .188), representing a substantial decline. Ninety-seven percent of original studies had significant results (p < .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and, if no bias in original results is assumed, combining original and replication results left 68% with significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Be the first to leave a comment. Don’t be shy.