SIO 221C Data Analysis Laboratory


updated Fall 2014

Popular press readings on data analysis

When the popular press starts writing about scientific data analysis methods, maybe we should take this as a warning. Here are two articles of interest:
  1. The Truth Wears Off: Is there something wrong with the scientific method? by Jonah Lehrer, The New Yorker, 13 December 2010.

    The article discusses a phenomenon called the "Decline Effect": astonishing new results are published (often in high-profile journals with lots of media coverage). The results appear to be highly statistically significant, but in subsequent studies, no one ever replicates them as well. And after numerous additional investigations, the original astonishing results are essentially debunked. Why is this? Likely because the original results were in fact an anomaly. Perhaps 100 tests were carried out, and only the case that was significant at the 99% level was published. In fact it was just an outlier. Bayesian statistics might help us avoid this trap, or a healthy willingness to test and retest.

    In physical oceanography, examples of the decline effect often result from processes inferred from short time series that turn out not to persist as clearly once time series have been extended in length. Some examples: the Antarctic Circumpolar Wave, Southern Ocean EKE response to the SAM, (and help me think of some others....). To be fair, we learn a lot from preliminary studies, even if early inferences don't hold up to further scrutiny. Nonetheless when possible, let's strive for robust results.

    Note however that journalist Jonah Lehrer has had some integrity issues of his own. Wikipedia provides quite a thorough accounting. Perhaps that means that he's especially well qualified to discuss scientific integrity, or perhaps it means we should doubt his analysis too.

  2. Trouble at the lab, The Economist, 19 October 2013

    This article discusses shoddy statistics and makes a case for verifying published results from other authors and for publishing negative results (if only to provide full data for meta-studies that amalgamate the results of all published work.) While most of the anguish about science and statistics is directed to biomedical research, the issues undoubtedly matter to us too, and this underscores the urgency of us ensuring that we understand statistics.

While I'm at it, let me draw your attention to Nate Silver's book, The Signal and the Noise, which is a fairly readable popular introduction to Bayesian statistics (e.g. the sort of book I would give my brother for Christmas). You might argue with the details of his assessment of the IPCC, but in other regards, he's fairly persuasive.

Not surprisingly, true Bayesian statisticians quibble with the simplifications inherent in Nate Silver's book. One popular reading recommendation from the statisticians is Sharon Bertsch McGrayne's history of Bayesian statistics, The Theory that Would Not Die: How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines and Emerged Triumphant From Two Centuries of Controversy.