Inside the multimillion-dollar essay-scoring business

Behind the scenes of standardized testing

There were a few other tricks to keep the numbers up. One was to send a wayward scorer off into a corner to study example papers long enough for the group's numbers to rebound. Another was to pair up a couple of bad scorers and make them decide together what to give a paper.

Or he could make the same announcement he'd heard from his supervisor back when he was a scorer.

"It's time we see more sixes," Farley would tell the group, which was code that his bell curve was off. "We're in trouble here, we need higher scores, give higher scores."

After three years working as a scorer, Dan DiMaggio says he's a skimming machine. "It's ugly," he says. "You just go as fast as possible."
Tony Nelson
After three years working as a scorer, Dan DiMaggio says he's a skimming machine. "It's ugly," he says. "You just go as fast as possible."

Though Farley and his fellow team leaders were fudging the numbers, even he was shocked when a representative from a southeastern state's Department of Education visited to check on how her state's essays were doing. As it turned out, the answer was: not well. About 67 percent of the students were getting 2s.

That's when the representative informed Farley that the rubric for her state's scoring had suddenly changed.

"We can't give this many 1s and 2s," she told him firmly.

The scorers would not be going back to re-grade the hundreds of tests they'd already finished—there just wasn't time. Instead, they were just going to give out more 3s.

No one objected—the customer was always right.

Eventually, Farley was hired away by a rival testing company and moved to the East Coast. As he saw standardized tests becoming more and more important to the fate of schools and kids, he got fed up, quit the industry, and decided to write a whistle-blowing book.

Making the Grades: My Misadventures in the Standardized Testing Industry, came out in 2009. Though the tell-all chronicles Farley's many misdeeds while scoring tests and supervising, he has nonetheless been invited back to work for the testing companies several times. The boom has just made his experience too valuable.

"They get paid money to put scores on paper, not to put the right scores on papers," he says. "They have a bottom line. Why anyone would expect anything else is beyond me."


PEARSON SPOKESMAN ADAM Gaber warns against taking the opinions of former scorers too seriously.

In an email, he characterized their concerns as "one-sided stories based upon people who have a very limited exposure and narrow point of view on what is truly a science."

Questar declined a request to visit their facilities, but reached by phone, Susan Trent, vice president of assessment services, said that the essays are scored as objectively as is possible.

"We're really insistent that readers understand they're dealing with kids," she says. "Decisions are being made about these kids based on these scores, and we're absolutely committed to getting them right."

She denies that graders are pressured to work too quickly and says that any evidence of scorer drift results in test re-scoring. She is also adamant that well-trained temps are the best way to score essays objectively.

"You do not have to be a teacher in order to score student response," adds Terry Appleman, vice president of performance assessment. "You have to have a good rubric and good training."

Asked what to make of the former Questar employees who felt they couldn't do a good job given their training and time constraints, Appleman quickly answers: "If they don't think they're qualified, it's not the job for them."

Most of the scorers interviewed for this story agree, but nearly all plan to return to the scoring center. They say they need the money. 

« Previous Page