Before my new boyfriend Nate Silver was forecasting elections, he created a baseball stats system called PECOTA that used advanced statistical methods to predict future player performance. This was before Brad Pitt starred in Moneyball, so his fancy new stats were widely derided. But he has at least one new fan, Sports Illustrated writer Phil Taylor:
I have surrendered to the numbers. I will make no assessment, athletic or otherwise, without rigorous statistical analysis…. I reject your anecdotal evidence, your hunches, your wishes disguised as predictions. I will keep my gut instincts away from my brain and suggest you do as well.
And what has brought this swoon over Phil? My new boyfriend Nate Silver’s election-night success.
Silver not only forecast President Obama’s reelection, he did it with uncanny precision, calling which candidate would win each of the 50 states despite weeks of heckling from more than a few pundits… That’s like hitting every jumper in a three-point-shooting contest while opponents rain trash talk on your head.
Well, actually, no it isn’t. I’m not running down MNB Nate Silver’s skill or anything, because he’s my boyfriend and I support him unconditionally, but his miraculous achievement was the result of looking at polls that asked people which of several well-known choices they might make in a short time period coming up, and then believing the answer. There weren’t a lot of unpredictable factors. To be amazed by this is like asking people on your wedding invite if they want chicken or fish and then being impressed when the same number of chicken and fish dinners are ordered on the big day.
It’s really a very different creature than sabermetrics, aka nerdly analysis of baseball stats, where you’re doing a more straight-forward regressive analysis, trying to comb through figures to find ones that might be predictive of future performance. “His baseball predictions weren’t as spot-on as his election projections,” notes Taylor. Right! Because making a prediction of what people are going to do based on what they said they were going to do and making one based purely on past performance are two entirely different creatures. To paraphrase Pulp Fiction, “it ain’t the same ballpark, it ain’t the same league, it ain’t even the same sport.”
People who don’t necessarily understand numbers tend to think that one is pretty much as good as the other. This is the kind of thinking that leads to getting excited over the number of impressions a banner ad got, or the number of clicks a search term got, even if neither of those leads to revenue. But with any analysis it’s important to figure out which ones matter and which ones don’t. Then getting good results might not seem like such a miracle.
Picture by piratetuba