Carl Bergstrom and Jevin West were featured in this article in The Economist on debunking dodgy data.
ON JULY 2ND the American state of Georgia counted a total of 87,709 cases of covid-19. Fifteen days later the number had risen to 135,183. Yet the state government’s online heat map looked largely the same. There appeared to be no increase in the number of crimson red areas where the outbreak was most severe. How come?
As it turned out, the threshold for places to turn red had been lifted from 2,961 cases to 3,769. This example of misleading data visualisation was called out by Carl Bergstrom and Jevin West. It joined the ever-growing catalogue of “bullshit”, malign and otherwise, which they debunk for students at the University of Washington.
Out of that course they have spun “Calling Bullshit”, a helpful guide to navigating a world full of doubtful claims based on spurious data. Using clever anecdotes, nods to online culture and allusions to ancient philosophy, the book tells ordinary readers how to spot nonsense—even if they are not numerical whizzes. As well as sketching the difference between correlation and causality, the authors outline visualisation techniques and explain machine learning to arm people against assertions that seem, and so probably are, either “too good or too bad to be true”.
There is, alas, no shortage of material. In one of their examples, a widely shared scholarly article seems to show that musicians from genres such as rap and hip-hop die much younger than those who play blues or jazz. The researchers in question calculate that half of all hip- hop musicians are murdered—a classic case of a claim too bad to be true. Messrs Bergstrom and West show where they went wrong: the raw numbers are not incorrect, but the picture they paint is incomplete, because they discount performers who are still alive. As rap music only began in the 1970s, rappers who have already died tend to have done so younger than those from the more venerable genres cited in the article.
The ways of deceit and error with data are many—and the authors point them out ruthlessly. Their fellow scientists, the media, the “TED brand of bullshit”: no one is spared. They describe how the findings of a study can be manipulated to make them seem statistically important even when they are not, and how feeding an algorithm skewed inputs yields unreliable results. For instance, in 2017 two scientists sparked ethical concerns by claiming to have built an algorithm that could guess whether a person was gay or straight on the basis of pictures gleaned from a dating site. The paper, which The Economist covered at the time, failed to mention that their “gaydar” may have been responding to variations in how people choose to present themselves (make-up, poses and so on), rather than to authentic physical differences.
While charts depicting the life expectancy of musicians are hardly lethal themselves, purporting to discern a person’s character from dodgy variables is perilous. Amid the pandemic, misinformation about infection rates and the efficacy of drugs—often bolstered by sneaky graphics, as in Georgia—is a particular concern. Some scientists are bypassing the usual peer-review process. Meanwhile newsrooms are under ever-greater pressure to attract clicks. More and more bullshit is contaminating debate. Mr Bergstrom and Mr West picked a good time to expose it.