subject his modus to a rigorous examination by looking at empirical research
on the link between information and understanding. So this statement,
additional knowledge of the minutiae of daily business can be
useless, even actually toxic, is indirectly but quite effectively testable.
Show two groups of people a blurry image of a fire hydrant, blurry
enough for them not to recognize what it is. For one group, increase the
resolution slowly, in ten steps. For the second, do it faster, in five steps.
Stop at a point where both groups have been presented an identical image
and ask each of them to identify what they see. The members of the group
that saw fewer intermediate steps are likely to recognize the hydrant much
faster. Moral? The more information you give someone, the more hypotheses
they will formulate along the way, and the worse off they will be.
They see more random noise and mistake it for information.
The problem is that our ideas are sticky: once we produce a theory, we
are not likely to change our minds—so those who delay developing their
theories are better off. When you develop your opinions on the basis of
weak evidence, you will have difficulty interpreting subsequent information
that contradicts these opinions, even if this new information is obviously
more accurate. Two mechanisms are at play here: the confirmation
bias that we saw in Chapter 5, and belief perseverance, the tendency not
to reverse opinions you already have. Remember that we treat ideas like
possessions, and it will be hard for us to part with them.
The fire hydrant experiment was first done in the sixties, and replicated
several times since. I have also studied this effect using the mathematics of
information: the more detailed knowledge one gets of empirical reality,
the more one will see the noise (i.e., the anecdote) and mistake it for actual
information. Remember that we are swayed by the sensational. Listening
to the news on the radio every hour is far worse for you than reading a
weekly magazine, because the longer interval allows information to be filtered
a bit.
In 1965, Stuart Oskamp supplied clinical psychologists with successive
files, each containing an increasing amount of information about patients;
the psychologists' diagnostic abilities did not grow with the additional
supply of information. They just got more confident in their original diagnosis.
Granted, one may not expect too much of psychologists of the 1965
variety, but these findings seem to hold across disciplines.
THE SCANDAL OF P R E D I C T I O N 1 45
Finally, in another telling experiment, the psychologist Paul Slovic
asked bookmakers to select from eighty-eight variables in past horse races
those that they found useful in computing the odds. These variables included
all manner of statistical information about past performances. The
bookmakers were given the ten most useful variables, then asked to predict
the outcome of races. Then they were given ten more and asked to
predict again. The increase in the information set did not lead to an increase
in their accuracy; their confidence in their choices, on the other
hand, went up markedly. Information proved to be toxic. I've struggled
much of my life with the common middlebrow belief that "more is
better"—more is sometimes, but not always, better. This toxicity of knowledge
will show in our investigation of the so-called expert.
THE EXPERT PROBLEM, OR THE TRAGEDY OF THE EMPTY SUIT
So far we have not questioned the authority of the professionals involved
but rather their ability to gauge the boundaries of their own knowledge.
Epistemic arrogance does not preclude skills. A plumber will almost always
know more about plumbing than a stubborn essayist and mathematical
trader. A hernia surgeon will rarely know less about hernias than a
belly dancer. But their probabilities, on the other hand, will be off—and,
this is the disturbing point, you may know much more on that score than
the expert. No matter what anyone tells you, it is a good idea to question
the error rate of an expert's procedure. Do not question his procedure,
only his confidence. (As someone who was burned by the medical
establishment, I learned to be cautious, and I urge everyone to be: if you
walk into a doctor's office with a symptom, do not listen to his odds of its
not being cancer.)
I will separate the two cases as follows. The mild case: arrogance in the
presence of (some) competence, and the severe case: arrogance mixed with
incompetence (the empty suit). There are some professions in which you
know more than the experts, who are, alas, people for whose opinions
you are paying—instead of them paying you to listen to them. Which
ones?
What Moves and What Does Not Move
There is a very rich literature on the so-called expert problem, running empirical
testing on experts to verify their record. But it seems to be confus1
4 6 WE J U S T C A N ' T P R E D I CT
ing at first. On one hand, we are shown by a class of expert-busting researchers
such as Paul Meehl and Robyn Dawes that the "expert" is the
closest thing to a fraud, performing no better than a computer using a single
metric, their intuition getting in the way and blinding them. (As an example
of a computer using a single metric, the ratio of liquid assets to debt
fares better than the majority of credit analysts.) On the other hand, there
is abundant literature showing that many people can beat computers
thanks to their intuition. Which one is correct?
There must be some disciplines with true experts. Let us ask the following
questions: Would you rather have your upcoming brain surgery
performed by a newspaper's science reporter or by a certified brain surgeon?
On the other hand, would you prefer to listen to an economic forecast
by someone with a PhD in finance from some "prominent" institution
such as the Wharton School, or by a newspaper's business writer? While
the answer to the first question is empirically obvious, the answer to the
second one isn't at all. We can already see the difference between "knowhow"
and "know-what." The Greeks made a distinction between techn?
and epistèmê. The empirical school of medicine of Menodotus of Nicomedia
and Heraclites of Tarentum wanted its practitioners to stay closest to
techn? (i.e., "craft"), and away from epistèmê (i.e., "knowledge," "science").
The psychologist James Shanteau undertook the task of finding out
which disciplines have experts and which have none. Note the confirmation
problem here: if you want to prove that there are no experts, then you
will be able to find a profession in which experts are useless. And you can
prove the opposite just as well. But there is a regularity: there are professions
where experts play a role, and others where there is no evidence of
skills. Which are which?
Experts who tend to be experts: livestock judges, astronomers, test pilots,
soil judges, chess masters, physicists, mathematicians (when they
deal with mathematical problems, not empirical ones), accountants, grain
inspectors, photo interpreters, insurance analysts (dealing with bell curvestyle
statistics).
Experts who tend to be . .. not experts: stockbrokers, clinical psychologists,
psychiatrists, college admissions officers, court judges, councilors,
personnel selectors, intelligence analysts (the CIA's record, in spite of its
costs, is pitiful). I would add these results from my own examination of
the literature: economists, financial forecasters, finance professors, political
scientists, "risk experts," Bank for International Settlements staff,
THE SCANDAL OF P R E D I C T I O N 1 47
august members of the International Association of Financial Engineers,
and personal financial advisers.
Simply, things that move, and therefore require knowledge, do not
usually have experts, while things that don't move seem to have some experts.
In other words, professions that deal with the future and base their
studies on the nonrepeatable past have an expert problem (with the exception
of the weather and businesses involving short-term physical processes,
not socioeconomic ones). I am not saying that no one who deals with the
future provides any valuable information (as I pointed out earlier, newspapers
can predict theater opening hours rather well), but rather that
those who provide no tangible added value are generally dealing with the
future.
Another way to see it is that things that move are often Black
Swan-prone. Experts are narrowly focused persons who need to "tunnel."
In situations where tunneling is safe, because Black Swans are not
consequential, the expert will do well.
Robert Trivers, an evolutionary psychologist and a man of supernormal
insights, has another answer (he became one of the most influential
evolutionary thinkers since Darwin with ideas he developed while
trying to go to law school). He links it to self-deception. In fields where we
have ancestral traditions, such as pillaging, we are very good at predicting
outcomes by gauging the balance of power. Humans and chimps can immediately
sense which side has the upper hand, and make a cost-benefit
analysis about whether to attack and take the goods and the mates. Once
you start raiding, you put yourself into a delusional mind-set that makes
you ignore additional information—it is best to avoid wavering during
battle. On the other hand, unlike raids, large-scale wars are not something
present in human heritage—we are new to them—so we tend to misestimate
their duration and overestimate our relative power. Recall the underestimation
of the duration of the Lebanese war. Those who fought in the
Great War thought it would be a mere cakewalk. So it was with the Vietnam
conflict, so it is with the Iraq war, and just about every modern conflict.
You cannot ignore self-delusion. The problem with experts is that they
do not know what they do not know. Lack of knowledge and delusion
about the quality of your knowledge come together—the same process
that makes you know less also makes you satisfied with your knowledge.
Next, instead of the range of forecasts, we will concern ourselves with
the accuracy of forecasts, i.e., the ability to predict the number itself.
1 4 8 WE J U S T C A N ' T P R E D I CT
How to Have the Last Laugh
We can also learn about prediction errors from trading activities. We
quants have ample data about economic and financial forecasts—from
general data about large economic variables to the forecasts and market
calls of the television "experts" or "authorities." The abundance of such
data and the ability to process it on a computer make the subject invaluable
for an empiricist. If I had been a journalist, or, God forbid, a historian,
I would have had a far more difficult time testing the predictive
effectiveness of these verbal discussions. You cannot process verbal commentaries
with a computer—at least not so easily. Furthermore, many
economists naively make the mistake of producing a lot of forecasts concerning
many variables, giving us a database of economists and variables,
which enables us to see whether some economists are better than others
(there is no consequential difference) or if there are certain variables for
which they are more competent (alas, none that are meaningful).
I was in a seat to observe from very close our ability to predict. In my
full-time trader days, a couple of times a week, at 8:30 A . M . , my screen
would flash some economic number released by the Department of Commerce,
or Treasury, or Trade, or some such honorable institution. I never
had a clue about what these numbers meant and never saw any need to invest
energy in finding out. So I would not have cared the least about them
except that people got all excited and talked quite a bit about what these
figures were going to mean, pouring verbal sauce around the forecasts.
Among such numbers you have the Consumer Price Index (CPI), Nonfarm
Payrolls (changes in the number of employed individuals), the Index of
Leading Economic Indicators, Sales of Durable Goods (dubbed "doable
girls" by traders), the Gross Domestic Product (the most important one),
and many more that generate different levels of excitement depending on
their presence in the discourse.
The data vendors allow you to take a peek at forecasts by "leading
economists," people (in suits) who work for the venerable institutions,
such as J . P. Morgan Chase or Morgan Stanley. You can watch these economists
talk, theorizing eloquently and convincingly. Most of them earn
seven figures and they rank as stars, with teams of researchers crunching
numbers and projections. But the stars are foolish enough to publish their
projected numbers, right there, for posterity to observe and assess their degree
of competence.
Worse yet, many financial institutions produce booklets every year-end
THE SCANDAL OF P R E D I C T I O N 1 49
called "Outlook for 200X," reading into the following year. Of course
they do not check how their previous forecasts fared after they were formulated.
The public might have been even more foolish in buying the arguments
without requiring the following simple tests—easy though they
are, very few of them have been done. One elementary empirical test is to
compare these star economists to a hypothetical cabdriver (the equivalent
of Mikhail from Chapter 1): you create a synthetic agent, someone who
takes the most recent number as the best predictor of the next, while assuming
that he does not know anything. Then all you have to do is compare
the error rates of the hotshot economists and your synthetic agent.
The problem is that when you are swayed by stories you forget about the
necessity of such testing.
Events Are Outlandish
The problem with prediction is a little more subtle. It comes mainly from
the fact that we are living in Extremistan, not Mediocristan. Our predictors
may be good at predicting the ordinary, but not the irregular, and this
is where they ultimately fail. All you need to do is miss one interest-rates
