methods within his reach. The problem is that we focus on the rare occasion
when these methods work and almost never on their far more numerous
failures. I kept begging anyone who would listen to me: "Hey, I am an
uncomplicated, no-nonsense fellow from Amioun, Lebanon, and have
trouble understanding why something is considered valuable if it requires
running computers overnight but does not enable me to predict better
than any other guy from Amioun." The only reactions I got from these
THE SCANDAL OF P R E D I C T I O N 1 5 5
colleagues were related to the geography and history of Amioun rather
than a no-nonsense explanation of their business. Here again, you see the
narrative fallacy at work, except that in place of journalistic stories you
have the more dire situation of the "scientists" with a Russian accent
looking in the rearview mirror, narrating with equations, and refusing to
look ahead because he may get too dizzy. The econometrician Robert
Engel, an otherwise charming gentleman, invented a very complicated statistical
method called GARCH and got a Nobel for it. No one tested it to
see if it has any validity in real life. Simpler, less sexy methods fare exceedingly
better, but they do not take you to Stockholm. You have an expert
problem in Stockholm, and I will discuss it in Chapter 17.
This unfitness of complicated methods seems to apply to all methods.
Another study effectively tested practitioners of something called game
theory, in which the most notorious player is John Nash, the schizophrenic
mathematician made famous by the film A Beautiful Mind. Sadly,
for all the intellectual appeal of these methods and all the media attention,
its practitioners are no better at predicting than university students.
There is another problem, and it is a little more worrisome. Makridakis
and Hibon were to find out that the strong empirical evidence of
their studies has been ignored by theoretical statisticians. Furthermore,
they encountered shocking hostility toward their empirical verifications.
"Instead [statisticians] have concentrated their efforts in building more sophisticated
models without regard to the ability of such models to more
accurately predict real-life data," Makridakis and Hibon write.
Someone may counter with the following argument: Perhaps economists'
forecasts create feedback that cancels their effect (this is called the
Lucas critique, after the economist Robert Lucas). Let's say economists
predict inflation; in response to these expectations the Federal Reserve acts
and lowers inflation. So you cannot judge the forecast accuracy in economics
as you would with other events. I agree with this point, but I do
not believe that it is the cause of the economists' failure to predict. The
world is far too complicated for their discipline.
When an economist fails to predict outliers he often invokes the issue
of earthquakes or revolutions, claiming that he is not into geodesies, atmospheric
sciences, or political science, instead of incorporating these
fields into his studies and accepting that his field does not exist in isolation.
Economics is the most insular of fields; it is the one that quotes least
from outside itself! Economics is perhaps the subject that currently has the
1 5 6 WE J U S T C A N ' T PREDICT
highest number of philistine scholars—scholarship without erudition and
natural curiosity can close your mind and lead to the fragmentation of
disciplines.
"OTHER THAN THAT," IT WAS OKAY
We have used the story of the Sydney Opera House as a springboard for
our discussion of prediction. We will now address another constant in
human nature: a systematic error made by project planners, coming from
a mixture of human nature, the complexity of the world, or the structure
of organizations. In order to survive, institutions may need to give themselves
and others the appearance of having a "vision."
Plans fail because of what we have called tunneling, the neglect of
sources of uncertainty outside the plan itself.
The typical scenario is as follows. Joe, a nonfiction writer, gets a book
contract with a set final date for delivery two years from now. The topic is
relatively easy: the authorized biography of the writer Salman Rushdie,
for which Joe has compiled ample data. He has even tracked down
Rushdie's former girlfriends and is thrilled at the prospect of pleasant interviews.
Two years later, minus, say, three months, he calls to explain to
the publisher that he will be a little delayed. The publisher has seen this
coming; he is used to authors being late. The publishing house now
has cold feet because the subject has unexpectedly faded from public
attention—the firm projected that interest in Rushdie would remain high,
but attention has faded, seemingly because the Iranians, for some reason,
lost interest in killing him.
Let's look at the source of the biographer's underestimation of the time
for completion. He projected his own schedule, but he tunneled, as he did
not forecast that some "external" events would emerge to slow him down.
Among these external events were the disasters on September 11, 2001,
which set him back several months; trips to Minnesota to assist his ailing
mother (who eventually recovered); and many more, like a broken engagement
(though not with Rushdie's ex-girlfriend). "Other than that," it was
all within his plan; his own work did not stray the least from schedule. He
does not feel responsible for his failure.*
The unexpected has a one-sided effect with projects. Consider the
* The book you have in your hands is approximately and "unexpectedly" fifteen
months late.
THE SCANDAL OF P R E D I C T I O N 1 5 7
track records of builders, paper writers, and contractors. The unexpected
almost always pushes in a single direction: higher costs and a longer time
to completion. On very rare occasions, as with the Empire State Building,
you get the opposite: shorter completion and lower costs—these occasions
are truly exceptional.
We can run experiments and test for repeatability to verify if such errors
in projection are part of human nature. Researchers have tested how
students estimate the time needed to complete their projects. In one representative
test, they broke a group into two varieties, optimistic and pessimistic.
Optimistic students promised twenty-six days; the pessimistic
ones forty-seven days. The average actual time to completion turned out
to be fifty-six days.
The example of Joe the writer is not acute. I selected it because it concerns
a repeatable, routine task—for such tasks our planning errors are
milder. With projects of great novelty, such as a military invasion, an allout
war, or something entirely new, errors explode upward. In fact, the
more routine the task, the better you learn to forecast. But there is always
something nonroutine in our modern environment.
There may be incentives for people to promise shorter completion
dates—in order to win the book contract or in order for the builder to get
your down payment and use it for his upcoming trip to Antigua. But the
planning problem exists even where there is no incentive to underestimate
the duration (or the costs) of the task. As I said earlier, we are too narrowminded
a species to consider the possibility of events straying from our
mental projections, but furthermore, we are too focused on matters internal
to the project to take into account external uncertainty, the "unknown
unknown," so to speak, the contents of the unread books.
There is also the nerd effect, which stems from the mental elimination
of off-model risks, or focusing on what you know. You view the world
from within a model. Consider that most delays and cost overruns arise
from unexpected elements that did not enter into the plan—that is, they
lay outside the model at hand—such as strikes, electricity shortages, accidents,
bad weather, or rumors of Martian invasions. These small Black
Swans that threaten to hamper our projects do not seem to be taken into
account. They are too abstract—we don't know how they look and cannot
talk about them intelligently.
We cannot truly plan, because we do not understand the future—but
this is not necessarily bad news. We could plan while bearing in mind such
limitations. It just takes guts.
1 5 8 WE J U S T C A N ' T P R E D I CT
The Beauty of Technology: Excel Spreadsheets
In the not too distant past, say the precomputer days, projections remained
vague and qualitative, one had to make a mental effort to keep
track of them, and it was a strain to push scenarios into the future. It took
pencils, erasers, reams of paper, and huge wastebaskets to engage in the
activity. Add to that an accountant's love for tedious, slow work. The activity
of projecting, in short, was effortful, undesirable, and marred with
self-doubt.
But things changed with the intrusion of the spreadsheet. When you
put an Excel spreadsheet into computer-literate hands you get a "sales
projection" effortlessly extending ad infinitum! Once on a page or on a
computer screen, or, worse, in a PowerPoint presentation, the projection
takes on a life of its own, losing its vagueness and abstraction and becoming
what philosophers call reified, invested with concreteness; it takes on
a new life as a tangible object.
My friend Brian Hinchcliffe suggested the following idea when we
were both sweating at the local gym. Perhaps the ease with which one can
project into the future by dragging cells in these spreadsheet programs is
responsible for the armies of forecasters confidently producing longerterm
forecasts (all the while tunneling on their assumptions). We have become
worse planners than the Soviet Russians thanks to these potent
computer programs given to those who are incapable of handling their
knowledge. Like most commodity traders, Brian is a man of incisive and
sometimes brutally painful realism.
A classical mental mechanism, called anchoring, seems to be at work
here. You lower your anxiety about uncertainty by producing a number,
then you "anchor" on it, like an object to hold on to in the middle of a
vacuum. This anchoring mechanism was discovered by the fathers of the
psychology of uncertainty, Danny Kahneman and Amos Tversky, early in
their heuristics and biases project. It operates as follows. Kahneman and
Tversky had their subjects spin a wheel of fortune. The subjects first
looked at the number on the wheel, which they knew was random, then
they were asked to estimate the number of African countries in the United
Nations. Those who had a low number on the wheel estimated a low number
of African nations; those with a high number produced a higher estimate.
Similarly, ask someone to provide you with the last four digits of his
social security number. Then ask him to estimate the number of dentists in
THE SCANDAL OF P R E D I C T I O N 1 5 9
Manhattan. You will find that by making him aware of the four-digit
number, you elicit an estimate that is correlated with it.
We use reference points in our heads, say sales projections, and start
building beliefs around them because less mental effort is needed to compare
an idea to a reference point than to evaluate it in the absolute {System
1 at work!). We cannot work without a point of reference.
So the introduction of a reference point in the forecaster's mind will
work wonders. This is no different from a starting point in a bargaining
episode: you open with high number ("I want a million for this house");
the bidder will answer "only eight-fifty"—the discussion will be determined
by that initial level.
The Character of Prediction Errors
Like many biological variables, life expectancy is from Mediocristan, that
is, it is subjected to mild randomness. It is not scalable, since the older we
get, the less likely we are to live. In a developed country a newborn female
is expected to die at around 79, according to insurance tables. When, she
reaches her 79th birthday, her life expectancy, assuming that she is in typical
health, is another 10 years. At the age of 90, she should have another
4.7 years to go. At the age of 100, 2.5 years. At the age of 119, if she
miraculously lives that long, she should have about nine months left. As
she lives beyond the expected date of death, the number of additional
years to go decreases. This illustrates the major property of random variables
related to the bell curve. The conditional expectation of additional
life drops as a person gets older.
With human projects and ventures we have another story. These are
often scalable, as I said in Chapter 3. With scalable variables, the ones
from Extremistan, you will witness the exact opposite effect. Let's say a
project is expected to terminate in 79 days, the same expectation in days
as the newborn female has in years. On the 79th day, if the project is not
finished, it will be expected to take another 25 days to complete. But on
the 90th day, if the project is still not completed, it should have about 58
days to go. On the 100th, it should have 89 days to go. On the 119th, it
should have an extra 149 days. On day 600, if the project is not done, you
will be expected to need an extra 1,590 days. As you see, the longer you
wait, the longer you will be expected to wait.
Let's say you are a refugee waiting for the return to your homeland.
Each day that passes you are getting farther from, not closer to, the day of
1 6 0 WE J U S T C A N ' T PREDICT
triumphal return. The same applies to the completion date of your next
opera house. If it was expected to take two years, and three years later you
are asking questions, do not expect the project to be completed any time
soon. If wars last on average six months, and your conflict has been going
on for two years, expect another few years of problems. The Arab-Israeli
conflict is sixty years old, and counting—yet it was considered "a simple
problem" sixty years ago. (Always remember that, in a modern environment,
