move, from 6 percent to 1 percent in a longer-term projection (what happened
between 2000 and 2001) to have all your subsequent forecasts rendered
completely ineffectual in correcting your cumulative track record.
What matters is not how often you are right, but how large your cumulative
errors are.
And these cumulative errors depend largely on the big surprises, the
big opportunities. Not only do economic, financial, and political predictors
miss them, but they are quite ashamed to say anything outlandish to
their clients—and yet events, it turns out, are almost always outlandish.
Furthermore, as we will see in the next section, economic forecasters tend
to fall closer to one another than to the resulting outcome. Nobody wants
to be off the wall.
Since my testing has been informal, for commercial and entertainment
purposes, for my own consumption and not formatted for publishing, I
will use the more formal results of other researchers who did the dog work
of dealing with the tedium of the publishing process. I am surprised that
so little introspection has been done to check on the usefulness of these
professions. There are a few—but not many—formal tests in three domains:
security analysis, political science, and economics. We will no
doubt have more in a few years. Or perhaps not—the authors of such pa1
5 0 WE J U S T C A N ' T PREDICT
pers might become stigmatized by his colleagues. Out of close to a million
papers published in politics, finance, and economics, there have been only
a small number of checks on the predictive quality of such knowledge.
Herding Like Cattle
A few researchers have examined the work and attitude of security analysts,
with amazing results, particularly when one considers the epistemic
arrogance of these operators. In a study comparing them with weather
forecasters, Tadeusz Tyszka and Piotr Zielonka document that the analysts
are worse at predicting, while having a greater faith in their own
skills. Somehow, the analysts' self-evaluation did not decrease their error
margin after their failures to forecast.
Last June I bemoaned the dearth of such published studies to Jean-
Philippe Bouchaud, whom I was visiting in Paris. He is a boyish man who
looks half my age though he is only slightly younger than I, a matter that
I half jokingly attribute to the beauty of physics. Actually he is not exactly
a physicist but one of those quantitative scientists who apply methods
of statistical physics to economic variables, a field that was started
by Beno?t Mandelbrot in the late 1950s. This community does not use
Mediocristan mathematics, so they seem to care about the truth. They are
completely outside the economics and business-school finance establishment,
and survive in physics and mathematics departments or, very often,
in trading houses (traders rarely hire economists for their own consumption,
but rather to provide stories for their less sophisticated clients). Some
of them also operate in sociology with the same hostility on the part of the
"natives." Unlike economists who wear suits and spin theories, they use
empirical methods to observe the data and do not use the bell curve.
He surprised me with a research paper that a summer intern had just
finished under his supervision and that had just been accepted for publication;
it scrutinized two thousand predictions by security analysts. What it
showed was that these brokerage-house analysts predicted nothing—a
naive forecast made by someone who takes the figures from one period as
predictors of the next would not do markedly worse. Yet analysts are informed
about companies' orders, forthcoming contracts, and planned expenditures,
so this advanced knowledge should help them do considerably
better than a naive forecaster looking at the past data without further information.
Worse yet, the forecasters' errors were significantly larger than
the average difference between individual forecasts, which indicates herdTHE
SCANDAL OF P R E D I C T I O N 1 51
ing. Normally, forecasts should be as far from one another as they are
from the predicted number. But to understand how they manage to stay in
business, and why they don't develop severe nervous breakdowns (with
weight loss, erratic behavior, or acute alcoholism), we must look at the
work of the psychologist Philip Tetlock.
/ Was "Almost" Right
Tetlock studied the business of political and economic "experts." He
asked various specialists to judge the likelihood of a number of political,
economic, and military events occurring within a specified time frame
(about five years ahead). The outcomes represented a total number of
around twenty-seven thousand predictions, involving close to three hundred
specialists. Economists represented about a quarter of his sample.
The study revealed that experts' error rates were clearly many times what
they had estimated. His study exposed an expert problem: there was no
difference in results whether one had a PhD or an undergraduate degree.
Well-published professors had no advantage over journalists. The only
regularity Tetlock found was the negative effect of reputation on prediction:
those who had a big reputation were worse predictors than those
who had none.
But Tetlock's focus was not so much to show the real competence of
experts (although the study was quite convincing with respect to that) as
to investigate why the experts did not realize that they were not so good
at their own business, in other words, how they spun their stories. There
seemed to be a logic to such incompetence, mostly in the form of belief defense,
or the protection of self-esteem. He therefore dug further into the
mechanisms by which his subjects generated ex post explanations.
I will leave aside how one's ideological commitments influence one's
perception and address the more general aspects of this blind spot toward
one's own predictions.
You tell yourself that you were playing a different game. Let's say you
failed to predict the weakening and precipitous fall of the Soviet Union
(which no social scientist saw coming). It is easy to claim that you were excellent
at understanding the political workings of the Soviet Union, but
that these Russians, being exceedingly Russian, were skilled at hiding
from you crucial economic elements. Had you been in possession of such
economic intelligence, you would certainly have been able to predict the
demise of the Soviet regime. It is not your skills that are to blame. The
1 5 2 WE J U S T C A N ' T PREDICT
same might apply to you if you had forecast the landslide victory for Al
Gore over George W. Bush. You were not aware that the economy was in
such dire straits; indeed, this fact seemed to be concealed from everyone.
Hey, you are not an economist, and the game turned out to be about economics.
You invoke the outlier. Something happened that was outside the system,
outside the scope of your science. Given that it was not predictable,
you are not to blame. It was a Black Swan and you are not supposed
to predict Black Swans. Black Swans, NNT tells us, are fundamentally
unpredictable (but then I think that NNT would ask you, Why rely on
predictions?). Such events are "exogenous," coming from outside your
science. Or maybe it was an event of very, very low probability, a thousandyear
flood, and we were unlucky to be exposed to it. But next time, it will
not happen. This focus on the narrow game and linking one's performance
to a given script is how the nerds explain the failures of mathematical
methods in society. The model was right, it worked well, but the game
turned out to be a different one than anticipated.
The "almost right" defense. Retrospectively, with the benefit of a revision
of values and an informational framework, it is easy to feel that it was
a close call. Tetlock writes, "Observers of the former Soviet Union who, in
1988, thought the Communist Party could not be driven from power by
1993 or 1998 were especially likely to believe that Kremlin hardliners almost
overthrew Gorbachev in the 1991 coup attempt, and they would
have if the conspirators had been more resolute and less inebriated, or if
key military officers had obeyed orders to kill civilians challenging martial
law or if Yeltsin had not acted so bravely."
I will go now into more general defects uncovered by this example.
These "experts" were lopsided: on the occasions when they were right,
they attributed it to their own depth of understanding and expertise; when
wrong, it was either the situation that was to blame, since it was unusual,
or, worse, they did not recognize that they were wrong and spun stories
around it. They found it difficult to accept that their grasp was a little
short. But this attribute is universal to all our activities: there is something
in us designed to protect our self-esteem.
We humans are the victims of an asymmetry in the perception of random
events. We attribute our successes to our skills, and our failures to
external events outside our control, namely to randomness. We feel responsible
for the good stuff, but not for the bad. This causes us to think
that we are better than others at whatever we do for a living. Ninety-four
THE SCANDAL OF P R E D I C T I O N 1 5 3
percent of Swedes believe that their driving skills put them in the top
50 percent of Swedish drivers; 84 percent of Frenchmen feel that their
lovemaking abilities put them in the top half of French lovers.
The other effect of this asymmetry is that we feel a little unique, unlike
others, for whom we do not perceive such an asymmetry. I have mentioned
the unrealistic expectations about the future on the part of people
in the process of tying the knot. Also consider the number of families who
tunnel on their future, locking themselves into hard-to-flip real estate
thinking they are going to live there permanently, not realizing that the
general track record for sedentary living is dire. Don't they see those welldressed
real-estate agents driving around in fancy two-door German cars?
We are very nomadic, far more than we plan to be, and forcibly so. Consider
how many people who have abruptly lost their job deemed it likely
to occur, even a few days before. Or consider how many drug addicts entered
the game willing to stay in it so long.
There is another lesson from Tetlock's experiment. He found what I
mentioned earlier, that many university stars, or "contributors to top journals,"
are no better than the average New York Times reader or journalist
in detecting changes in the world around them. These sometimes overspecialized
experts failed tests in their own specialties.
The hedgehog and the fox. Tetlock distinguishes between two types of
predictors, the hedgehog and the fox, according to a distinction promoted
by the essayist Isaiah Berlin. As in Aesop's fable, the hedgehog knows one
thing, the fox knows many things—these are the adaptable types you need
in daily life. Many of the prediction failures come from hedgehogs who
are mentally married to a single big Black Swan event, a big bet that is not
likely to play out. The hedgehog is someone focusing on a single, improbable,
and consequential event, falling for the narrative fallacy that makes
us so blinded by one single outcome that we cannot imagine others.
Hedgehogs, because of the narrative fallacy, are easier for us to
understand—their ideas work in sound bites. Their category is overrepresented
among famous people; ergo famous people are on average worse at
forecasting than the rest of the predictors.
I have avoided the press for a long time because whenever journalists
hear my Black Swan story, they ask me to give them a list of future impacting
events. They want me to be predictive of these Black Swans. Strangely,
my book Fooled by Randomness, published a week before September 11,
2001, had a discussion of the possibility of a plane crashing into my office
building. So I was naturally asked to show "how I predicted the event." I
1 5 4 WE J U S T C A N ' T P R E D I CT
didn't predict it—it was a chance occurrence. I am not playing oracle! I
even recently got an e-mail asking me to list the next ten Black Swans.
Most fail to get my point about the error of specificity, the narrative fallacy,
and the idea of prediction. Contrary to what people might expect, I
am not recommending that anyone become a hedgehog—rather, be a fox
with an open mind. I know that history is going to be dominated by an improbable
event, I just don't know what that event will be.
Reality? What For?
I found no formal, Tetlock-like comprehensive study in economics journals.
But, suspiciously, I found no paper trumpeting economists' ability to
produce reliable projections. So I reviewed what articles and working papers
in economics I could find. They collectively show no convincing evidence
that economists as a community have an ability to predict, and, if
they have some ability, their predictions are at best just slightly better than
random ones—not good enough to help with serious decisions.
The most interesting test of how academic methods fare in the real
world was run by Spyros Makridakis, who spent part of his career
managing competitions between forecasters who practice a "scientific
method" called econometrics—an approach that combines economic theory
with statistical measurements. Simply put, he made people forecast
in real life and then he judged their accuracy. This led to the series of
"M-Competitions" he ran, with assistance from Michèle Hibon, of which
M3 was the third and most recent one, completed in 1999. Makridakis
and Hibon reached the sad conclusion that "statistically sophisticated or
complex methods do not necessarily provide more accurate forecasts than
simpler ones."
I had an identical experience in my quant days—the foreign scientist
with the throaty accent spending his nights on a computer doing complicated
mathematics rarely fares better than a cabdriver using the simplest
