Inside the multimillion-dollar essay-scoring business

Behind the scenes of standardized testing

One student wrote, "Martin Luther King Jr. was a good leader." With artfulness far beyond the student's age, the essay delved into King's history with the civil rights movement, pointing out the key moments that had shown his leadership.

There was just one problem: It didn't fit the rubric. The rubric liked a longer essay, with multiple sentences lauding key qualities of leadership such as "honesty" and "inspires people." This essay was incredibly concise, but got its point across. Nevertheless, the rubric said it was a 2. Puthoff knew it was a 2.

He hesitated the way he had been specifically trained not to. Then he hit, "3."

After three years working as a scorer, Dan DiMaggio says he's a skimming machine. "It's ugly," he says. "You just go as fast as possible."
Tony Nelson
After three years working as a scorer, Dan DiMaggio says he's a skimming machine. "It's ugly," he says. "You just go as fast as possible."

It didn't take long before a supervisor was in his face. He leaned down with a printout of the King essay.

"This really isn't a 3-style paper," the supervisor said.

Puthoff pointed out the smart use of examples and the exceptional prose. The supervisor just shook his head and pointed out how short the paragraphs were.

"You know, it's more of a 2," the supervisor repeated. "Not enough elaboration."

Puthoff quickly learned these were not arguments he could win. But as time went on, he found himself having more and more of them.

There were the students who wrote extremely well but whose responses were too short—in his mind he saw them, bored with the essay topic, hurrying to finish. Or the essays where the handwriting got rushed and jumbled at the end, then cut off abruptly—he imagined the proctor telling the frantic student to lay down his pencil on a well-written but incomplete response.

And there were the kids who just did what they wanted. Like the boy from Arkansas who, instead of writing about the most fun thing to do in his town, instead wrote a hilarious essay on why his town is terrible and how he wanted to burn it down and pee on the ashes.

"I wanted the kid to get the score they deserved," Puthoff says of his time in the business. "But they want to put them in boxes."

In defiance, Puthoff started giving creatively written essays an illicit score bump. His agreement numbers noticeably suffered.

The industry calls this "scorer drift," a well-documented tendency to begin deviating from the rubric over time. One case of scorer drift actually resulted in some 4,100 teachers failing the essay portion of their certification exams. The teachers successfully sued for $11.1 million.

What was different about Puthoff's scorer drift was that he was doing it on purpose.

"I'll bring them up, don't worry," he'd say of his agreement rate, then go back the next day and do the exact same thing.

"I know this kid is good," he'd tell himself. "I know this kid's a good writer."

   

TODD FARLEY TREATED his supervising position at a scoring company like a joke.

"At the time, testing wasn't that big," he says. "I never had to feel like I'm actually deciding someone's future. It was just silly."

Farley had started at the bottom rung of the testing industry in Iowa City. A part-time graduate student with bills to pay, he was more interested in partying and trying to become a writer than he was in getting a real job. So he took one scoring job after another for NCS.

"It was always a temporary gig," he remembers. "It was a lovely, slacker-y life."

Farley had no official training in teaching, education, or test writing, but the longer he remained at NCS, the more responsibilities he was handed. He took the offer to become a team leader because it paid a little extra money and got him out of scoring.

Teaching his first group of scorers, Farley walked them through the rubric the same way he'd been shown. He fielded the inevitable bombardment of confused questions as best he could, in particular from one man: Harry the laid-off refrigerator plant worker.

Even though Harry eventually passed his qualifying exams, he was a disaster. As Farley monitored Harry's scoring, he found himself walking back over to Harry repeatedly.

"Look," Farley would say. "You're giving this essay a 2 even though it's perfectly formatted."

Harry would nod. But a short time later, another ridiculous low ball from Harry would land on Farley's desk. Before long, Harry began to drag down the all-important agreement level.

Farley now understood the reasons why, when he'd been a scorer, his team leaders would tell the room he wanted to start seeing more 3s or 4s or whatever. Supervisors were expected to turn the test scores into a nice bell curve. If his room did not agree at least 80 percent of the time, the tests would be taken back and re-graded, wasting time and money. The supervisor would be put on probation or demoted.

When Farley complained to a fellow supervisor about his problem, she smiled wryly and held up a pencil.

"I've got this eraser, see," she told him. "I help them out."

So Farley simply began changing Harry's scores to agree with his peers'. The practice soon spread well beyond Harry.

"I'd just change a bunch of answers to make it look like my group was doing a great job," Farley says. "I wanted the stupid item to be done, and so did my bosses."

« Previous Page
 |
 
1
 
2
 
3
 
4
 
All
 
Next Page »
 
My Voice Nation Help
83 comments
essayshop.org
essayshop.org

Everyone has a different view on writing. When teachers grade essays, it`s not necessary that their grading is fair. Just because they may not like your style doesn’t mean it`s wrong or unlikable. All great essay writers faced criticism in the beginning but decided to stick with their writing styles and got their names in history books written in the same way. Creativity and uniqueness matters more than anything else when you are an essay writer.

ikilledtheshark
ikilledtheshark

The moneyed employing class has visited upon my generation a contrived denial of the chance to live up to our potential and contribute to society in a meaningful way. The atrocities that continue to be committed behind closed doors in secretive boardroom meetings are criminal at best under the RICO Act, and treasonous to present and future generations of humanity. Because the majority of government entities, as well as many "officials", have been complicit in these plans to block prosperity from millions of people, to obscure and obstruct the paths taken by former generations to reach the American dream, and  have provided cover to those foisting this reality upon us, they are equally guilty. But yet, there is more money in existence than at any prior time in history. Who is the authority here? The Declaration of Independence and US Constitution suggest that the power rests with the people.
It is plain to see, if you look past the distractions constantly hurtled at us, that the goalpost has been moved. Much like Lucy and Charlie Brown in the "Peanuts" cartoon, there isn't even a ball in play; that is, unless, like the more fortunate half of my peers, one starts out in life with advantages that others don't have. For the rest of us, it's a crap shoot. However, any more than two bouts with misfortune and you wind up in the boat with us. Instead, our children, your grandchildren and great-grandchildren have been abjectly sentenced to a destitute future full of uncertainty, instability, lack of good nutrition, safety, a sound education, health, and promising careers. Opportunities are limited to floating from temp. jobs to seasonal positions @ minimum wage (1/3 of a living) with no security. In the ancient Roman Empire, the "insulae" were poor people living in small, crowded apartments with only 2-3 rooms which housed their large families.
The broken promises, the failed policies (trickle-down, deregulation, free-market), the out and out LIES and deception, the smoke and mirrors from the warped views of PIGS in the Boardroom, in Congress, in the Whitehouse, on the Bench, in the Statehouses, etc.. is dismaying but it is all adding up. People need to start taking this seriously. Those named above are hard at work at ripping this country up and tearing it down. I know it; you know it.  It has been going on for 33 years, at a minimum, and needs to be brought to an end.
All hope is NOT lost.  Keep spreading and speaking truth to power, while the masses are continuing to awaken.

MrT51
MrT51

... If this isn't TRANSFER OF PUBLIC WEALTH TO CORPORATIONS, complete bastardization of the process of education, and selling our children's future for small change .. I don't know what is. The stink of this trend is what riled my guts and made me retire. Kids and teachers are jumped through hoops in order to enrich testing corporations. The fabrication of data, side-tracking the really brilliant kids, and cowing of teachers with poor test scores resulting from fraudulent testing. When the teachers and students were using the rubrics and scoring and kids were revising their work and constructing portfolios of their best creations, I thought we were really onto something. Authentic assessment of real student projects gathered for all to see. When repub legislators started complaining of the cost of university scoring and portfolio storage/transport, I just thought "and our kids aren't worth it?". They won - cancelled these all authentic assessments, literally dumped the portfolios and went to the corporate grading with repub legislatures awarding big corporate contracts and outsourcing the grading and assessment jobs out of the state. Suddenly we were pressed to drill kids for fill in the blanks and there was no time for portfolio projects or revising essays. It has degraded far more in the 8 yrs since I have left. My competent administrators have left. My colleagues who are left are demoralized and are "most-experienced=first to be laid off" status. The supposed anti-big givt Repubs have used big brother tactics to strip local schools boards and teachers of funding, self-govt, and any legitimate teaching of hard skills and inventiveness. Our dropout rate and teacher attrition rate speaks loudly of this. Nothing will change until parents vote out the repub privatizers of public schooling and their for-profit schemes - lining the pockets of their buddies who have just happened to start up private sector for profit education grifter schemes. Education, healthcare and other human services are not served by the profit-making ethic. I'd love to see a renaissance for the kids' sake but these Repubs are out for blood money and pure greed. This is not the crisis that the teacher-bashing reform movement of Michelle Rhee is even pointing out at present. One generation of this corporate takeover of our kids' futures and road map to the health of the national mind could be lost.

buster13
buster13

@MrT51 Agree whole-heatedly with everything said here. I am a recent retiree as well and a young one at that. I retired much earlier than I have ever intended but the inner turmoil and emotional toll of knowing the crimes being committed against our youth and future drove me to the brink. For those who remain in the field & are waking up at night sweating & worrying about whether they have properly prepared their students to pass the test only to have the grading of them be a farce? This realization will just be one more notch in every good teacher's inevitable decline into disillusionment , disgust and lunacy. 


I'm not sure that even one generation of students OR teachers will survive this and I am darn sure it will take MANY generations to overcome it. 

englishteacher
englishteacher

The worst thing is that scores on these kinds of tests are replacing experienced teachers' judgments about how their students' writing has developed over time.  Teachers in many schools and districts are being pressured to make their grades conform to scores on these kinds of tests, rather than the other way around.  

mariabronn
mariabronn

The most disgusting part of this story is the part where the scores are artificially inflated to make the proper bell curve with the total amount of results.  The kids whose papers were already graded got screwed, especially if someone argued in favor of a higher score but was overruled!  Then, kids who didn't do as good a job as they did got higher scores just so the testing results would appear to be accurate and valid, which is total crap!

I hope our elected officials responsible for this incredible mess are proud of themselves, because if they aren't, nobody else would be.  That's for sure.

Melissitzifun
Melissitzifun

There is really a negative slant on this story. I hope the majority of scoring centers aren't this bad. I certainly have never felt any pressure from my team leader or scoring director. I hope this is truly is a "one-sided story based upon people who have a very limited exposure and narrow point of view on what is truly a science."..... And it is true, I'm a temp and I do it for the money.... But it's a seasonal position.... The scoring centers really couldn't get the tests scored on time without using people who are qualified but out of work... The cream of the crop are not going to drop what they are doing (teaching) to sign on for a 3 month low-paying gig, Their hands are tied in some ways.....

Brian Pete
Brian Pete

Extended response evaluators Please Read

Gaxlion
Gaxlion

To gather a group of scorers a scoring business must meet the client state's request that the scorers have a certain accuracy rate, say the scorers can accurately score 70% or 80% of the trail paragraphs to the client states desired ratings. Since this is nearly impossible, the scoring business gives the potential scorers some of the answers during their qualifying tests and hints as to how to adjust other answers in order to get them qualified. I know, I'm a scorer and witnessed it first hand. Not one of the scorers in the group I was in failed to qualify. Education has been turned into a business. A business promises it's client a certain product then does what it has to to deliver. Is anyone surprised?

Jim
Jim

If you went to a reputable college anywhere in the world, standardized testing was involved as part of the requirement for admission. Our teachers, lawyers, doctors, engineers, nurses and many more "professionals" . . . all were required to take standardized . . . "admission to the profession" . . . tests. Who scores those tests? What is the argument here? If you don't test well you won't advance. It doesn't matter who scores the test or who designs the rubric. It's no different then a decision made in a court of law. If you can't make the case . . . you lose . . . unless you're OJ Simpson. Most people living in the United States know that a person who faces a jury trial, faces a group composed of their "peers". They follow a rubric called the rule of law. These people aren't required to know the name of a college (let alone have a college degree).

The wealthy people will most likely go to the "favored" schools. Most people will live in the "imperfect" world. People, including the educational psychologist you quoted in your article (to this person: GRE ring a bell . . . how about ACT or SAT), need to accomodate and adjust and quit eating sour grapes. Personally, I have a problem associating with people who over-compensate for a lack of ability to embrace the imperfection of humanity and make a decision to be happy.

RNelson
RNelson

So much for following the rubric. Our state's requires a 0-4 grade in five different areas (organization, student voice, sentence formation, etc). And each of those has several variables. But hey, your "rubric" is probably right.

never-to-return
never-to-return

Blame the teachers? Teachers can only work with what the parents send them. In ever-increasing numbers. Teachers are expected to essentially babysit forty kids whose parents either can't or won't raise them to be able to be taught anything in the first place. Whether they are single parents, or still-married parents who are both forced to work full-time just to make ends meet, or simply do not care or are depressed or on drugs. Trace it all back to the vast income disparity. Educate yourself a little about profit-mongering. Blame the teachers, indeed.

Jaydub1948
Jaydub1948

Living in a town like C-bus with lots of unemployed college grads, I've done plenty of seasonal work grading standardized tests. I have to agree with most of what was said in the article regarding the need to conform to the rubrics. It didn't take long to figure out what they wanted and score accordingly and with speed. It is quite true that the retired teachers, as well as unemployed librarians, could not "pull the trigger" as one of my team leaders would say.

20autismmom10
20autismmom10

While I hate to annoy people by holding up Canada as an example, we need to do what they have all along - use teachers and administrators each year to determine what a child should know and prepare, test and grade it themselves. This test-taking mania started with NCLB to help benefit Neil Bush, George W's brother, who started a business in "test-taking" materials, supplemental "educational" materials and has never had anything to do with the educational welfare of our nations' students. It was always about making a buck at the public's expense and look what's been created. Add to that the charter movement that proves to discriminate against special needs and English Language Learner students (how do you think they "claim" better test scores?) and you will see charters (as businesses) being handed these failing schools to then remove those students who are in need of the most help. The destruction of public education is a well-planned business maneuver to acquire public property and federal/state funding to use as they please for their discriminatory little school universe. It's a crime of national proportions, but we're all just lying down and letting them ruin and then give our public institutions away to the highest bidder.

MrT51
MrT51

You couldn't have outlined it better. And there are great models out there. Canada and Finland. The worst Is corporate. Look at corporate/for-profit healthcare. Three times the cost and the worst infant morality rate in the first world. Where is America's inventiveness that we desperately need to look to Europe, Taiwan, Australua for cutting edge education, health and gun policy. Is there brain death in the water? Where did this Corporate Taliban // Capitalist Fundamentalism begin. I know people worshipped Reagan for being the alzheimerist soap salesman for some "revolution". But we've just spiraled downward since his mumblings and Nancy's translations. I'd be confused if I hadn't seen a puppet show as a child.

audra.wulirecords
audra.wulirecords

@20autismmom10  And another thing... when did we stop holding back students that can't/don't/won't perform at a level high enough to merit advancing to the next grade? When I was a kid, we all lived in abject dread of not going to the next grade and that was enough motivation for most of us to keep it from happening...


Michael Hardy
Michael Hardy

We need to just drop an immense nuclear weapon of mass destruction on the whole standardized testing industry.

Guest
Guest

But test scores are all the fault of teachers.

okami
okami

I'm beginning to wonder about the education of the people in charge of this affair. . .

The Free School Apparent
The Free School Apparent

Once again, the curtain is parted and the wizard is shown to be a less than ordinary man.Is anything in our modern life free from the profit disease?

Ron Amundson
Ron Amundson

One solution to this is to require clawbacks as part of the contract with standardized testing companies based upon a students future performance. In other words, the testing industry would in effect be accountable for their methods of assessment (as well as fraud detection).

For example, if a student who does very well on the standardized tests, either MC or essay, and then said student proceeds to flunk out,or even get C's or D's while in high school or college years later, significant penalties would then be automatically and retroactively imposed upon the grading firm. It likely would be necessary to require bonds to be posted up front as part of said contract, as a firms viability cannot be predicted over the long haul. Granted, there are any number of factors well beyond the control of the assessment firms..but then again insurance actuaries do the same kind of thing nearly every day.

Granted, such an approach might prove to be too expensive to operate on shore, being it would likely equate to highly trained graders working together, and having enough time to achieve a level of accuracy such that their firm would not end up loosing their shirt years down the road (or go out of business today, as they could no longer afford the bond). On the other hand, having the assessment firm being at more than are arms length from those who would wish to influence outcomes might not be such a bad thing either.

I guess I;m half sarcastic,and half idealistic in this... it would throw such a huge wrench into the status quo, and many firms would go under, and likewise many would lose their jobs. On the other hand, to hold a school district accountable with teachers, staff, students,and parents at risk, all the while the test firms rake in the dough with so little accountability is not cool either.

Jwhyperion
Jwhyperion

As an employee of a company that produces standardized tests, I will say that this article needs to be read with caution. First, please consider the sources in the article. Please note that beside the inclusion of the very truncated responses from the testing companies, these companies were not given a voice. Let me share some of my professional knowledge with you.

First, standardized testing has been going on for hundreds of years. It's called tests, quizzes, and essays in the classroom. The individual school uses assessment testing as does the school district. It's always been done this way. What has changed is the importance placed upon the results of these standardized tests. Look to the government (of either political persuasion) for the federal requirement for statewide testing.

Having taught for 10 years and now having worked for a testing company for 5, I can assure you that educators have a HUGE say in what they're kids are tested on. In all the states, committees of teachers representing all areas of the state, review each and every test question and analyze the data from the field test to determine which items show validity and will be used on an operational test. In many states, the teachers actually write the test questions. I had no idea when I first started in this business how much impact the teachers have.

When it comes to essay testing, there will always be controversy, just as there will always be differing opinions in the classroom when a teacher gives you a "C" on an essay and you KNOW you should have gotten an "A". There are things called human error and human subjectivity. I agree that essay testing for a standardized test has many inherent problems. Again, however, know that in most states the teachers write these questions and create the rubrics for how they should be scored. Also, most importantly people should know that the essay portion of the state assessment does not dictate whethere a student passes or fails. A student's overall score is determined by both MC questions and the essay question with the emphasis in scoring on the MC questions. Not all states include essay questions as part of their assessment. Many use a short answer response that asks students to elaborate upon information gained from the text while bringing in their own experiences and opinions. In this case, if a student writes on a completely different topic they receive a zero just as they would on a math test if the added two numbers incorrectly.

There are of course inherent flaws in the system just as there are inherent flaws in most every system. Instead of just complaining, though, it would be nice if the detractors could use some of their effort to help come up with a solution. As my principal always said, "Don't come to me with a problem unless you have a proposed solution."

Captainfrogbert
Captainfrogbert

What a shock! A system developed by republicans in which the only criteria is shoveling billions in tax money into the pockets of big business with no consideration of the possible damage that could be done to the lives of thousands of innocent children. Glib corporate spokesmen with stock, facile responses to serious concerns. No oversight. Nothing but making the most money off the public dole and move on. Do lots of damage to America and her children in the process. Sounds like the ideal republican government program.

Rena McGee
Rena McGee

I don't know whether to laugh and feel smug (because I think the entire testing thing is a waste of time, or just shake my head.

Jenny
Jenny

MUST READ for English teachers.

Marin
Marin

It is not possible to objectively score essay questions, therefore no tests should have essay questions.

Sophia Green
Sophia Green

Hi cutie, Could you hit me up on--- RichFlirts.C'om---A dating club for successful, beautiful people.I am a smart&pretty gal. seeking a sweet man.pls Check out my username myshine,serious...

Bev
Bev

I have a teaching degree, and have done scoring for Pearson a few times. I am always frustrated by my inability to use absoloutes, when grading a writing sample. I strive to give each student my best response. All of the other scorers know that some kids life will be affected by our decisions. No matter where the scorer got his or her Bachealor degree, all seem to take the job seriously. Bev Schunk

Kathleenwicker
Kathleenwicker

As a teacher for 23 years in South Florida... I can tell you we experience the same problems. I've had students who could barely write their name - come back with a high score and students who were talented writers get the same score. Very inequitable. Bad for schools and more importantly bad for children.

Umar Anjum
Umar Anjum

The article shocked me, I knew there were issues with how the test were scored but I had no idea it was such a work-mill of questionable qualifications

Dharm
Dharm

This is where true benefit of advanced technology like natural language processing comes. Automated essay scoring has been proven to have similar performance compared to human scorers as much one human scorer compares to another. The mathematical model of cognitive and pedagogical processes is called latent semantic analysis.

Ed Sike
Ed Sike

Sounds dreadful, but what do you (or the complaining scorers) propose as an alternative? The people who set up this system and the politicians who fund it operate under godawful pressure from the media to change things, make things better, do it now, etc. The media should therefore accept some of the responsibility for the results and explore thoughtful alternatives, but I don't see any in this story.

Red5 C9
Red5 C9

The article is very one-sided. Where do you think the money comes from to pay for testing? The state of Arkansas Dept. of Education gets their money to pay from testing somewhere, right? Like probably their constituents and/or the Feds?Having multiple players in the industry promotes competition which is good to keep prices down.Hiring 300,600, or 1000 people to score student essays has to be a huge cash outlay for any company chosen to administer state level testing. Those companies that can do it in an efficient manner and keep costs down are providing a great service in an open market.Listen, objectively scoring essays from a couple hundred thousand students cannot be easy.Would everyone prefer not to evaulate our student population? Maybe we should just ask each student what they think they should get for a grade and give those out instead?Everyone wants perfection and everyone to be treated equally. Who can answer the best way to do that?

Esbtans
Esbtans

The testing system needs to be done away with. Tests should involve putting thoughts to practical uses that don't involve pencil in paper.

TV on the VCR
TV on the VCR

I did scoring for a season or two. Three things. Setting aside for the moment how well testing assess student performance, which is debatable, this article makes scoring essays sound much harder than it is.

The guy you quote at the beginning of the article must something serious going on. Even if you want to score the creative but off topic essays higher, the number of those are really limited - like maybe 5% or less. So even if you score those drastically different than anyone else, that is only 5% of the tests, you can still screw up another 15% before anyone cares about your "agreement rate."

And there are really only a limited number of categories - it is not that hard. Often times, like the ones in the print version of this article, there are only FOUR categories:0 - Student writes "Fuck you," draws a picture, writes "I hate my teacher," just plain doesn't put anything, or has a semi-literate scribble on the page that appears to contain a couple of off topic words.1 - Average, but on the low end.2 - Average, but on the high end.4 - Creative, interesting, well written, varied sentence structure, good use of details and adjectives, etc.

Even if there are six categories, it doesn't become that much more difficult. After you read 200 or 2000 essays on the same topic, you get a pretty good sense of what is good and what isn't. If you don't get a sense of it after that long, there is something seriously wrong.

Seeoscar
Seeoscar

Yeah, it's not surprising. The move to measure is good, but measuring writing is difficult. Standardizing and measuring en masse is, as the essay demonstrates, not only difficult but profitable.

Alan Scherstuhl
Alan Scherstuhl

Great, alarming piece. Good essays and good thinking are not made up of interchangeable parts.

 
Loading...