Last summer, the great mass of the American political punditocracy scoffed at the possibility that Donald Trump would win a single state in the Republican race. Now Trump has won the party’s presidential nomination.
Also
last summer, British bookmakers took bets at 5,000-to-1 odds that
Leicester City, a historically weak English soccer team, would win the
Premier League. The team has won, and it has reportedly cost the bookmakers 25 million pounds, or $36 million, the biggest loss on a single sports event in British gambling history.
Anyone
can simply be unlucky. But there are systematic biases and errors our
puny human brains tend to make repeatedly when we try to predict the
future, some of which are evident in the biggest political and sports
upsets of 2016.
It’s
fine and well to make fun of people (including myself) who were sure
that Mr. Trump had no shot at the presidency, or who offered overly
generous odds to try to bring bettors into a betting parlor. It’s even
possible that those projections were correct — that these outcomes
really were highly unlikely but that randomness prevailed (even a
million-to-one lottery ticket might hit).
But
it seems more likely that the odds were misjudged all along. And if
that’s the case, understanding the mental errors behind those mistakes
can be valuable for anyone who cares about forecasting and prediction
(which, one way or another, is pretty much all of us).
The
biggest is recency bias, a concept most often evoked in financial
markets by people who study the intersection of psychology and finance.
We tend to overweight recent history in thinking about the likelihood
that something might happen. People who have experienced a recent stock
market crash are more likely to think another one is imminent, for
example.
That
helps explain why, even when Mr. Trump entered the presidential race
and rocketed to the top of the Republican polls, so many smart people
remained so skeptical of the idea that he would win.
Remember
the 2012 Republican race, when a string of improbable candidates like
Michele Bachmann and Herman Cain briefly rose to the top of the polls,
while Mitt Romney, who had the support of establishment donors, made a
slow-and-steady path to the nomination. You could say the same of the
2004 Democratic race, when anti-establishment Howard Dean led in the
polls for months but John Kerry won the nomination.
But
it hasn’t always been that way. Wendell Willkie is the closest
counterexample. Amid a divided Republican party in 1940, Willkie, a
businessman who had never held public office and had been a Democrat
just a year prior, emerged from nowhere to claim the Republican
nomination. Barry Goldwater in 1964 was opposed by many Republican party
leaders.
And
in English soccer, in the last 21 years, only four teams with big
budgets and reputations — Chelsea, Manchester City, Manchester United
and Arsenal — had won. But before that, there are examples of teams
emerging to challenge the elite, like the 1995 champion Blackburn
Rovers.
Knowing
history in other words — and not just recent history — can help a
person avoid assuming that the way it has been recently is the way it
always is.
A
related lesson is to be aware of just how small the sample is when
looking to the past for guidance. American presidential elections take
place every four years, and English soccer championships once a year. A
basic lesson of sampling is that one should have less confidence in the
results the smaller the sample is, which in turn implies great modesty
about what might happen in the future.
But
it’s not just that the sample is small. It’s also that in these games —
running a presidential campaign or managing a soccer team — the rules
are always changing. This is a version of what Nassim Nicholas Taleb,
author of “The Black Swan,” called the “ludic fallacy.”
(It’s derived from Latin for “game.”) The idea is that it is a mistake
to apply the kind of simplified statistical model of a game to the
circumstances of real life.
In
a game, the rules are fixed. If you spin a roulette wheel, or even send
a baseball team out on the field to face an opponent, there are a set
range of things that can happen, and it’s relatively straightforward to
figure out the probability of each.
But
in politics, external events, media coverage, public opinion and the
strategies of opponents — even the number of opponents — are constantly
in flux in unpredictable ways. That means something that happened in a
past year may or may not be good evidence for what will happen this
year.
And
in sports, while an individual matchup might follow the usual rules of
probability, a team’s long-term strategy doesn’t. A smart owner or
manager can come up with new insights on how best to select and deploy
talent that gives an edge over rest of the league that simple odds won’t
predict. Here’s one analysis of the techniques Leicester City’s owners and managers used to prevail.
Another
example is the National Basketball Association’s Golden State Warriors,
who in the 2011-2012 season won only about third of their games. They
had been to the playoffs once in 18 years. Relying more on 3-point shots than the rest of the league — with the help of Stephen Curry, of course — they rose to be last year’s champion and set the record for regular-season wins this season.
Prediction
is hard. Complex systems with constantly changing rules, a limited
historical sample to analyze and our own psychological biases and short
historical memories make it harder. And that has rarely been more
evident than in the worlds of politics and sports in 2016.