第 5 页

those incentives. All you need are some data.

In this case, the Chicago Public School system obliged. It made available a

database of the test answers for every CPS student from third grade through

seventh grade from 1993 to 2000. This amounts to roughly 30,000 students per

grade per year, more than 700,000 sets of test answers, and nearly 100 million

individual answers. The data, organized by classroom, included each student’s

question-by-question answer strings for reading and math tests. (The actual

paper answer sheets were not included; they were habitually shredded soon

after a test.) The data also included some information about each teacher and

demographic information for every student, as well as his or her past and future

test scores—which would prove a key element in detecting the teacher cheating.

Now it was time to construct an algorithm that could tease some conclusions

from this mass of data. What might a cheating teacher’s classroom look like?

The first thing to search for would be unusual answer patterns in a given

classroom: blocks of identical answers, for instance, especially among the harder

questions. If ten very bright students (as indicated by past and future test scores)

gave correct answers to the exam’s first five questions (typically the easiest ones),

such an identical block shouldn’t be considered suspicious. But if ten poor

students gave correct answers to the last five questions on the exam (the hardest

ones), that’s worth looking into. Another red flag would be a strange pattern

within any one student’s exam—such as getting the hard questions right while

missing the easy ones—especially when measured against the thousands of

students in other classrooms who scored similarly on the same test. Furthermore,

the algorithm would seek out a classroom full of students who performed far

better than their past scores would have predicted and who then went on to

score significantly lower the following year. A dramatic one-year spike in test

scores might initially be attributed to a good teacher; but with a dramatic fall to

follow, there’s a strong likelihood that the spike was brought about by artificial

means.

Consider now the answer strings from the students in two sixth-grade Chicago

classrooms who took the identical math test. Each horizontal row represents one

student’s answers. The letter a, b, c, or d indicates a correct answer; a number

indicates a wrong answer, with 1 corresponding to a, 2 corresponding to b, and

so on. A zero represents an answer that was left blank. One of these classrooms

almost certainly had a cheating teacher and the other did not. Try to tell the

difference—although be forewarned that it’s not easy with the naked eye.

Classroom A

112a4a342cb214d0001acd24a3a12dadbcb4a0000000

d4a2341cacbddad3142a2344a2ac23421c00adb4b3cb

1b2a34d4ac42d23b141acd24a3a12dadbcb4a2134141

dbaab3dcacb1dadbc42ac2cc31012dadbcb4adb40000

d12443d43232d32323c213c22d2c23234c332db4b300

db2abad1acbdda212b1acd24a3a12dadbcb400000000

d4aab2124cbddadbcb1a42cca3412dadbcb423134bc1

1b33b4d4a2b1dadbc3ca22c000000000000000000000

d43a3a24acb1d32b412acd24a3a12dadbcb422143bc0

313a3ad1ac3d2a23431223c000012dadbcb400000000

db2a33dcacbd32d313c21142323cc300000000000000

d43ab4d1ac3dd43421240d24a3a12dadbcb400000000

db223a24acb11a3b24cacd12a241cdadbcb4adb4b300

db4abadcacb1dad3141ac212a3a1c3a144ba2db41b43

1142340c2cbddadb4b1acd24a3a12dadbcb43d133bc4

214ab4dc4cbdd31b1b2213c4ad412dadbcb4adb00000

1423b4d4a23d24131413234123a243a2413a21441343

3b3ab4d14c3d2ad4cbcac1c003a12dadbcb4adb40000

dba2ba21ac3d2ad3c4c4cd40a3a12dadbcb400000000

d122ba2cacbd1a13211a2d02a2412d0dbcb4adb4b3c0

144a3adc4cbddadbcbc2c2cc43a12dadbcb4211ab343

d43aba3cacbddadbcbca42c2a3212dadbcb42344b3cb

Classroom B

db3a431422bd131b4413cd422a1acda332342d3ab4c4

d1aa1a11acb2d3dbc1ca22c23242c3a142b3adb243c1

d42a12d2a4b1d32b21ca2312a3411d00000000000000

3b2a34344c32d21b1123cdc000000000000000000000

34aabad12cbdd3d4c1ca112cad2ccd00000000000000

d33a3431a2b2d2d44b2acd2cad2c2223b40000000000

23aa32d2a1bd2431141342c13d212d233c34a3b3b000

d32234d4a1bdd23b242a22c2a1a1cda2b1baa33a0000

d3aab23c4cbddadb23c322c2a222223232b443b24bc

3d13a14313c31d42b14c421c42332cd2242b3433a3343

d13a3ad122b1da2b11242dc1a3a12100000000000000

d12a3ad1a13d23d3cb2a21ccada24d2131b440000000

314a133c4cbd142141ca424cad34c122413223ba4b40

d42a3adcacbddadbc42ac2c2ada2cda341baa3b24321

db1134dc2cb2dadb24c412c1ada2c3a341ba20000000

d1341431acbddad3c4c213412da22d3d1132a1344b1b

1ba41a21a1b2dadb24ca22c1ada2cd32413200000000

dbaa33d2a2bddadbcbca11c2a2accda1b2ba20000000

If you guessed that classroom A was the cheating classroom, congratulations.

Here again are the answer strings from classroom A, now reordered by a

computer that has been asked to apply the cheating algorithm and seek out

suspicious patterns.

Classroom A

(With cheating algorithm applied)

1. 112a4a342cb214d0001acd24a3a12dadbcb4a0000000

2. 1b2a34d4ac42d23b141acd24a3a12dadbcb4a2134141

3. db2abad1acbdda212b1acd24a3a12dadbcb400000000

4. d43a3a24acb1d32b412acd24a3a12dadbcb422143bc0

5. d43ab4d1ac3dd43421240d24a3a12dadbcb400000000

6. 1142340c2cbddadb4b1acd24a3a12dadbcb43d133bc4

7. dba2ba21ac3d2ad3c4c4cd40a3a12dadbcb400000000

8. 144a3adc4cbddadbcbc2c2cc43a12dadbcb4211ab343

9. 3b3ab4d14c3d2ad4cbcac1c003a12dadbcb4adb40000

10. d43aba3cacbddadbcbca42c2a3212dadbcb42344b3cb

11. 214ab4dc4cbdd31b1b2213c4ad412dadbcb4adb00000

12. 313a3ad1ac3d2a23431223c000012dadbcb400000000

13. d4aab2124cbddadbcb1a42cca3412dadbcb423134bc1

14. dbaab3dcacb1dadbc42ac2cc31012dadbcb4adb40000

15. db223a24acb11a3b24cacd12a241cdadbcb4adb4b300

16. d122ba2cacbd1a13211a2d02a2412d0dbcb4adb4b3c0

17. 1423b4d4a23d24131413234123a243a2413a21441343

18. db4abadcacb1dad3141ac212a3a1c3a144ba2db41b43

19. db2a33dcacbd32d313c21142323cc300000000000000

20. 1b33b4d4a2b1dadbc3ca22c000000000000000000000

21. d12443d43232d32323c213c22d2c23234c332db4b300

22. d4a2341cacbddad3142a2344a2ac23421c00adb4b3cb

Take a look at the answers in bold. Did fifteen out of twenty-two students

somehow manage to reel off the same six consecutive correct answers (the d-a-d-

b-c-b string) all by themselves?

There are at least four reasons this is unlikely. One: those questions, coming near

the end of the test, were harder than the earlier questions. Two: these were

mainly subpar students to begin with, few of whom got six consecutive right

answers elsewhere on the test, making it all the more unlikely they would get

right the same six hard questions. Three: up to this point in the test, the fifteen

students’ answers were virtually uncorrelated. Four: three of the students

(numbers 1, 9, and 12) left at least one answer blank before the suspicious string

and then ended the test with another string of blanks. This suggests that a long,

unbroken string of blank answers was broken not by the student but by the

teacher.

There is another oddity about the suspicious answer string. On nine of the fifteen

tests, the six correct answers are preceded by another identical string, 3-a-1-2,

which includes three of four incorrect answers. And on all fifteen tests, the six

correct answers are followed by the same incorrect answer, a 4. Why on earth

would a cheating teacher go to the trouble of erasing a student’s test sheet and

then fill in the wrong answer?

Perhaps she is merely being strategic. In case she is caught and hauled into the

principal’s office, she could point to the wrong answers as proof that she didn’t

cheat. Or perhaps—and this is a less charitable but just as likely answer—she

doesn’t know the right answers herself. (With standardized tests, the teacher is

typically not given an answer key.) If this is the case, then we have a pretty good

clue as to why her students are in need of inflated grades in the first place: they

have a bad teacher.

Another indication of teacher cheating in classroom A is the class’s overall

performance. As sixth graders who were taking the test in the eighth month of

the academic year, these students needed to achieve an average score of 6.8 to be

considered up to national standards. (Fifth graders taking the test in the eighth

month of the year needed to score 5.8, seventh graders 7.8, and so on.) The

students in classroom A averaged 5.8 on their sixth-grade tests, which is a full

grade level below where they should be. So plainly these are poor students. A

year earlier, however, these students did even worse, averaging just 4.1 on their

fifth-grade tests. Instead of improving by one full point between fifth and sixth

grade, as would be expected, they improved by 1.7 points, nearly two grades’

worth. But this miraculous improvement was short-lived. When these sixth-

grade students reached seventh grade, they averaged 5.5—more than two grade

levels below standard and even worse than they did in sixth grade. Consider the

erratic year-to-year scores of three particular students from classroom A:

5TH GRADE SCORE

6TH GRADE SCORE

7TH GRADE SCORE

Student 3

3.0

6.5

5.1

Student 6

3.6

6.3

4.9

Student 14

3.8

7.1

5.6

The three-year scores from classroom B, meanwhile, are also poor but at least

indicate an honest effort: 4.2, 5.1, and 6.0. So an entire roomful of children in

classroom A suddenly got very smart one year and very dim the next, or more

likely, their sixth-grade teacher worked some magic with a no. 2 pencil.

There are two noteworthy points to be made about the children in classroom A,

tangential to the cheating itself. The first is that they are obviously in terrible

academic shape, which makes them the very children whom high-stakes testing

is promoted as helping the most. The second point is that these students would

be in for a terrible shock once they reached the seventh grade. All they knew was

that they had been successfully promoted due to their test scores. (No child left

behind, indeed.) They weren’t the ones who artificially jacked up their scores;

they probably expected to do great in the seventh grade—and then they failed

miserably. This may be the cruelest twist yet in high-stakes testing. A cheating

teacher may tell herself that she is helping her students, but the fact is that she

would appear far more concerned with helping herself.

An analysis of the entire Chicago data reveals evidence of teacher cheating in

more than two hundred classrooms per year, roughly 5 percent of the total. This

is a conservative estimate, since the algorithm was able to identify only the most

egregious form of cheating—in which teachers systematically changed students’

answers—and not the many subtler ways a teacher might cheat. In a recent study

among North Carolina schoolteachers, some 35 percent of the respondents said

they had witnessed their colleagues cheating in some fashion, whether by giving

students extra time, suggesting answers, or manually changing students’

answers.

What are the characteristics of a cheating teacher? The Chicago data show that

male and female teachers are about equally prone to cheating. A cheating teacher

tends to be younger and less qualified than average. She is also more likely to

cheat after her incentives change. Because the Chicago data ran from 1993 to

2000, it bracketed the introduction of high-stakes testing in 1996. Sure enough,

there was a pronounced spike in cheating in 1996. Nor was the cheating random.

It was the teachers in the lowest-scoring classrooms who were most likely to

cheat. It should also be noted that the $25,000 bonus for California teachers was

eventually revoked, in part because of suspicions that too much of the money

was going to cheaters.

Not every result of the Chicago cheating analysis was so dour. In addition to

detecting cheaters, the algorithm could also identify the best teachers in the

school system. A good teacher’s impact was nearly as distinctive as a cheater’s.

Instead of getting random answers correct, her students would show real

improvement on the easier types of questions they had previously missed, an

indication of actual learning. And a good teacher’s students carried over all their

gains into the next grade.

Most academic analyses of this sort tend to languish, unread, on a dusty library

shelf. But in early 2002, the new CEO of the Chicago Public Schools, Arne

Duncan, contacted the study’s authors. He didn’t want to protest or hush up

their findings. Rather, he wanted to make sure that the teachers identified by the

algorithm as cheaters were truly cheating—and then do something about it.

Duncan was an unlikely candidate to hold such a powerful job. He was only

thirty-six when appointed, a onetime academic all-American at Harvard who

later played pro basketball in Australia. He had spent just three years with the

CPS—and never in a job important enough to have his own secretary—before

becoming its CEO. It didn’t hurt that Duncan had grown up in Chicago. His

father taught psychology at the University of Chicago; his mother ran an after-

school program for forty years, without pay, in a poor neighborhood. When

Duncan was a boy, his afterschool playmates were the under-privileged kids his

mother cared for. So when he took over the public schools, his allegiance lay

more with schoolchildren and their families than with teachers and their union.

The best way to get rid of cheating teachers, Duncan had decided, was to

readminister the standardized exam. He only had the resources to retest 120

classrooms, however, so he asked the creators of the cheating algorithm to help

choose which classrooms to test.

How could those 120 retests be used most effectively? It might have seemed

sensible to retest only the classrooms that likely had a cheating teacher. But even

if their retest scores were lower, the teachers could argue that the students did

worse merely because they were told that the scores wouldn’t count in their