Inside the multimillion-dollar essay-scoring business

Behind the scenes of standardized testing

"I was like, 'Yeah, I have a degree, I can do that,'" she recalls.

On Indovino's first day, she drove out to Questar Assessment in Apple Valley, a beige warehouse, and followed the signs that said "Scoring Center" in bright red letters. During her brief interview, she'd been asked repeatedly if she was able to follow a "rubric"—a set of guidelines to assess the essays in as uniform a way as possible.

"I guess they've had bad experiences with English teachers," she says.

Todd Farley worked for 15 years in the testing industry before quitting to write a tell-all book about his former employer's scoring procedures
Todd Farley worked for 15 years in the testing industry before quitting to write a tell-all book about his former employer's scoring procedures

Inside Questar, Indovino took a seat in a room that looked like a classroom, crammed with as many computers and desks as could fit. It was here that the team leaders unveiled the scoring rubric, which was like a secret decoder ring for the job.

The rubrics are most often developed in conjunction with the state's department of education and its testing contractor. Currently, Minnesota contracts both its test writing and scoring to Pearson. Local teachers are included in the rubric-writing process, as well as test-writing academics called "psychometricians."

At first blush, the rubric seemed simple enough to Indovino. It was a chart with one- or two-sentence explanations of each number grade. Scorers are forbidden from taking the rubrics out of the Questar building or talking about them, but they generally look something like this:

6. An excellent response, the essay includes

• excellent focus and development

• excellent organization

• excellent language skills and word choice

• excellent grammar, usage, and mechanics

5. A good response, the essay includes

• good focus and development

• good organization

• good language skills and word choice

• good grammar, usage, and mechanics

4. An adequate response ...

On down to 1s, which were reserved for barely decipherable language.

As part of their training, Indovino and her co-workers read through pre-graded examples out loud, then discussed why each had been scored the way it was. The process quickly divided the room into two camps—the young, unemployed kids who were just there for a paycheck, and the retired teachers.

"The retired teachers would argue everything," says Indovino.

After two days of going through example papers, each scorer had to pass a qualifying exam. Indovino scored three sets of ten pre-scored papers. In order to be approved to work on the project, she had to pass two of the sets with at least an 80 percent "agreement rate" with the rubric. She did so with relative ease; most of the rest of the room passed on their second try.

Her first project was from Arkansas, an essay written by eighth-graders on the topic, "A fun thing to do in my town."

And that's where the troubles began.

Suddenly, she was being asked to crank through 200 real essays in a day. The scanned papers popped up on the screen and her eyes flitted as fast as they could down the lines. The difference between "excellent" and "good" and "adequate" was decided in a matter of seconds, to say nothing of the responses that were simply off the reservation. How do you score a kid who rails that his town sucks? What about an exceptionally well-written essay on why the student was refusing to answer the question?

All over the room, the teachers were raising their hands and disputing the rubric. Indovino preferred to keep her head down and just score the way she was told to.

"I was good at the bad system," she says.

Over the next several months, Indovino got to know her co-workers better. The young people were mostly laid off or in foreclosure. They came straight from paper routes and went off to waitressing jobs afterward.

They also made for a very dedicated workforce. Indovino says she saw her co-workers hung-over, extremely ill, and even fresh from surgery.

"I scored a full day without glasses on," Indovino says with a shrug. "I sat with my nose up to the glass all day. I couldn't read it."

When she eventually got a full-time job, Indovino quit scoring. Although she'd done well by the company's standards, following the rubric provided little sense of accomplishment.

"Nobody is saying, 'I'm doing good work, I'm helping society,'" she says. "Everyone is saying, 'This isn't right.'"

  

DAVID PUTHOFF WAS an experienced reader with Questar when he started getting the warnings that his job performance wasn't up to snuff.

"Your numbers are down a little bit," his supervisor said at the end of one day. "Make sure you bring those back up."

Most essays, depending on the criteria established in the state, are scored by two readers. As Puthoff and his fellow scorers whipped through their essays, their supervisor had their own eyes glued to a screen, keeping them apprised of whether Reader #1 agreed with Reader #2. If so, both got a 100 percent agreement score for that essay. If one differed by a point or so, the essay would be counted as "adjacent" agreement.

Puthoff had thus far been an agreement-rate superstar. He was consistently in the high 80s.

Then came the question from hell out of Louisiana: "What are the qualities of a good leader?"

« Previous Page
 |
 
1
 
2
 
3
 
4
 
All
 
Next Page »
 
My Voice Nation Help
83 comments
essayshop.org
essayshop.org

Everyone has a different view on writing. When teachers grade essays, it`s not necessary that their grading is fair. Just because they may not like your style doesn’t mean it`s wrong or unlikable. All great essay writers faced criticism in the beginning but decided to stick with their writing styles and got their names in history books written in the same way. Creativity and uniqueness matters more than anything else when you are an essay writer.

ikilledtheshark
ikilledtheshark

The moneyed employing class has visited upon my generation a contrived denial of the chance to live up to our potential and contribute to society in a meaningful way. The atrocities that continue to be committed behind closed doors in secretive boardroom meetings are criminal at best under the RICO Act, and treasonous to present and future generations of humanity. Because the majority of government entities, as well as many "officials", have been complicit in these plans to block prosperity from millions of people, to obscure and obstruct the paths taken by former generations to reach the American dream, and  have provided cover to those foisting this reality upon us, they are equally guilty. But yet, there is more money in existence than at any prior time in history. Who is the authority here? The Declaration of Independence and US Constitution suggest that the power rests with the people.
It is plain to see, if you look past the distractions constantly hurtled at us, that the goalpost has been moved. Much like Lucy and Charlie Brown in the "Peanuts" cartoon, there isn't even a ball in play; that is, unless, like the more fortunate half of my peers, one starts out in life with advantages that others don't have. For the rest of us, it's a crap shoot. However, any more than two bouts with misfortune and you wind up in the boat with us. Instead, our children, your grandchildren and great-grandchildren have been abjectly sentenced to a destitute future full of uncertainty, instability, lack of good nutrition, safety, a sound education, health, and promising careers. Opportunities are limited to floating from temp. jobs to seasonal positions @ minimum wage (1/3 of a living) with no security. In the ancient Roman Empire, the "insulae" were poor people living in small, crowded apartments with only 2-3 rooms which housed their large families.
The broken promises, the failed policies (trickle-down, deregulation, free-market), the out and out LIES and deception, the smoke and mirrors from the warped views of PIGS in the Boardroom, in Congress, in the Whitehouse, on the Bench, in the Statehouses, etc.. is dismaying but it is all adding up. People need to start taking this seriously. Those named above are hard at work at ripping this country up and tearing it down. I know it; you know it.  It has been going on for 33 years, at a minimum, and needs to be brought to an end.
All hope is NOT lost.  Keep spreading and speaking truth to power, while the masses are continuing to awaken.

MrT51
MrT51

... If this isn't TRANSFER OF PUBLIC WEALTH TO CORPORATIONS, complete bastardization of the process of education, and selling our children's future for small change .. I don't know what is. The stink of this trend is what riled my guts and made me retire. Kids and teachers are jumped through hoops in order to enrich testing corporations. The fabrication of data, side-tracking the really brilliant kids, and cowing of teachers with poor test scores resulting from fraudulent testing. When the teachers and students were using the rubrics and scoring and kids were revising their work and constructing portfolios of their best creations, I thought we were really onto something. Authentic assessment of real student projects gathered for all to see. When repub legislators started complaining of the cost of university scoring and portfolio storage/transport, I just thought "and our kids aren't worth it?". They won - cancelled these all authentic assessments, literally dumped the portfolios and went to the corporate grading with repub legislatures awarding big corporate contracts and outsourcing the grading and assessment jobs out of the state. Suddenly we were pressed to drill kids for fill in the blanks and there was no time for portfolio projects or revising essays. It has degraded far more in the 8 yrs since I have left. My competent administrators have left. My colleagues who are left are demoralized and are "most-experienced=first to be laid off" status. The supposed anti-big givt Repubs have used big brother tactics to strip local schools boards and teachers of funding, self-govt, and any legitimate teaching of hard skills and inventiveness. Our dropout rate and teacher attrition rate speaks loudly of this. Nothing will change until parents vote out the repub privatizers of public schooling and their for-profit schemes - lining the pockets of their buddies who have just happened to start up private sector for profit education grifter schemes. Education, healthcare and other human services are not served by the profit-making ethic. I'd love to see a renaissance for the kids' sake but these Repubs are out for blood money and pure greed. This is not the crisis that the teacher-bashing reform movement of Michelle Rhee is even pointing out at present. One generation of this corporate takeover of our kids' futures and road map to the health of the national mind could be lost.

buster13
buster13

@MrT51 Agree whole-heatedly with everything said here. I am a recent retiree as well and a young one at that. I retired much earlier than I have ever intended but the inner turmoil and emotional toll of knowing the crimes being committed against our youth and future drove me to the brink. For those who remain in the field & are waking up at night sweating & worrying about whether they have properly prepared their students to pass the test only to have the grading of them be a farce? This realization will just be one more notch in every good teacher's inevitable decline into disillusionment , disgust and lunacy. 


I'm not sure that even one generation of students OR teachers will survive this and I am darn sure it will take MANY generations to overcome it. 

englishteacher
englishteacher

The worst thing is that scores on these kinds of tests are replacing experienced teachers' judgments about how their students' writing has developed over time.  Teachers in many schools and districts are being pressured to make their grades conform to scores on these kinds of tests, rather than the other way around.  

mariabronn
mariabronn

The most disgusting part of this story is the part where the scores are artificially inflated to make the proper bell curve with the total amount of results.  The kids whose papers were already graded got screwed, especially if someone argued in favor of a higher score but was overruled!  Then, kids who didn't do as good a job as they did got higher scores just so the testing results would appear to be accurate and valid, which is total crap!

I hope our elected officials responsible for this incredible mess are proud of themselves, because if they aren't, nobody else would be.  That's for sure.

Melissitzifun
Melissitzifun

There is really a negative slant on this story. I hope the majority of scoring centers aren't this bad. I certainly have never felt any pressure from my team leader or scoring director. I hope this is truly is a "one-sided story based upon people who have a very limited exposure and narrow point of view on what is truly a science."..... And it is true, I'm a temp and I do it for the money.... But it's a seasonal position.... The scoring centers really couldn't get the tests scored on time without using people who are qualified but out of work... The cream of the crop are not going to drop what they are doing (teaching) to sign on for a 3 month low-paying gig, Their hands are tied in some ways.....

Brian Pete
Brian Pete

Extended response evaluators Please Read

Gaxlion
Gaxlion

To gather a group of scorers a scoring business must meet the client state's request that the scorers have a certain accuracy rate, say the scorers can accurately score 70% or 80% of the trail paragraphs to the client states desired ratings. Since this is nearly impossible, the scoring business gives the potential scorers some of the answers during their qualifying tests and hints as to how to adjust other answers in order to get them qualified. I know, I'm a scorer and witnessed it first hand. Not one of the scorers in the group I was in failed to qualify. Education has been turned into a business. A business promises it's client a certain product then does what it has to to deliver. Is anyone surprised?

Jim
Jim

If you went to a reputable college anywhere in the world, standardized testing was involved as part of the requirement for admission. Our teachers, lawyers, doctors, engineers, nurses and many more "professionals" . . . all were required to take standardized . . . "admission to the profession" . . . tests. Who scores those tests? What is the argument here? If you don't test well you won't advance. It doesn't matter who scores the test or who designs the rubric. It's no different then a decision made in a court of law. If you can't make the case . . . you lose . . . unless you're OJ Simpson. Most people living in the United States know that a person who faces a jury trial, faces a group composed of their "peers". They follow a rubric called the rule of law. These people aren't required to know the name of a college (let alone have a college degree).

The wealthy people will most likely go to the "favored" schools. Most people will live in the "imperfect" world. People, including the educational psychologist you quoted in your article (to this person: GRE ring a bell . . . how about ACT or SAT), need to accomodate and adjust and quit eating sour grapes. Personally, I have a problem associating with people who over-compensate for a lack of ability to embrace the imperfection of humanity and make a decision to be happy.

RNelson
RNelson

So much for following the rubric. Our state's requires a 0-4 grade in five different areas (organization, student voice, sentence formation, etc). And each of those has several variables. But hey, your "rubric" is probably right.

never-to-return
never-to-return

Blame the teachers? Teachers can only work with what the parents send them. In ever-increasing numbers. Teachers are expected to essentially babysit forty kids whose parents either can't or won't raise them to be able to be taught anything in the first place. Whether they are single parents, or still-married parents who are both forced to work full-time just to make ends meet, or simply do not care or are depressed or on drugs. Trace it all back to the vast income disparity. Educate yourself a little about profit-mongering. Blame the teachers, indeed.

Jaydub1948
Jaydub1948

Living in a town like C-bus with lots of unemployed college grads, I've done plenty of seasonal work grading standardized tests. I have to agree with most of what was said in the article regarding the need to conform to the rubrics. It didn't take long to figure out what they wanted and score accordingly and with speed. It is quite true that the retired teachers, as well as unemployed librarians, could not "pull the trigger" as one of my team leaders would say.

20autismmom10
20autismmom10

While I hate to annoy people by holding up Canada as an example, we need to do what they have all along - use teachers and administrators each year to determine what a child should know and prepare, test and grade it themselves. This test-taking mania started with NCLB to help benefit Neil Bush, George W's brother, who started a business in "test-taking" materials, supplemental "educational" materials and has never had anything to do with the educational welfare of our nations' students. It was always about making a buck at the public's expense and look what's been created. Add to that the charter movement that proves to discriminate against special needs and English Language Learner students (how do you think they "claim" better test scores?) and you will see charters (as businesses) being handed these failing schools to then remove those students who are in need of the most help. The destruction of public education is a well-planned business maneuver to acquire public property and federal/state funding to use as they please for their discriminatory little school universe. It's a crime of national proportions, but we're all just lying down and letting them ruin and then give our public institutions away to the highest bidder.

MrT51
MrT51

You couldn't have outlined it better. And there are great models out there. Canada and Finland. The worst Is corporate. Look at corporate/for-profit healthcare. Three times the cost and the worst infant morality rate in the first world. Where is America's inventiveness that we desperately need to look to Europe, Taiwan, Australua for cutting edge education, health and gun policy. Is there brain death in the water? Where did this Corporate Taliban // Capitalist Fundamentalism begin. I know people worshipped Reagan for being the alzheimerist soap salesman for some "revolution". But we've just spiraled downward since his mumblings and Nancy's translations. I'd be confused if I hadn't seen a puppet show as a child.

audra.wulirecords
audra.wulirecords

@20autismmom10  And another thing... when did we stop holding back students that can't/don't/won't perform at a level high enough to merit advancing to the next grade? When I was a kid, we all lived in abject dread of not going to the next grade and that was enough motivation for most of us to keep it from happening...


Michael Hardy
Michael Hardy

We need to just drop an immense nuclear weapon of mass destruction on the whole standardized testing industry.

Guest
Guest

But test scores are all the fault of teachers.

okami
okami

I'm beginning to wonder about the education of the people in charge of this affair. . .

The Free School Apparent
The Free School Apparent

Once again, the curtain is parted and the wizard is shown to be a less than ordinary man.Is anything in our modern life free from the profit disease?

Ron Amundson
Ron Amundson

One solution to this is to require clawbacks as part of the contract with standardized testing companies based upon a students future performance. In other words, the testing industry would in effect be accountable for their methods of assessment (as well as fraud detection).

For example, if a student who does very well on the standardized tests, either MC or essay, and then said student proceeds to flunk out,or even get C's or D's while in high school or college years later, significant penalties would then be automatically and retroactively imposed upon the grading firm. It likely would be necessary to require bonds to be posted up front as part of said contract, as a firms viability cannot be predicted over the long haul. Granted, there are any number of factors well beyond the control of the assessment firms..but then again insurance actuaries do the same kind of thing nearly every day.

Granted, such an approach might prove to be too expensive to operate on shore, being it would likely equate to highly trained graders working together, and having enough time to achieve a level of accuracy such that their firm would not end up loosing their shirt years down the road (or go out of business today, as they could no longer afford the bond). On the other hand, having the assessment firm being at more than are arms length from those who would wish to influence outcomes might not be such a bad thing either.

I guess I;m half sarcastic,and half idealistic in this... it would throw such a huge wrench into the status quo, and many firms would go under, and likewise many would lose their jobs. On the other hand, to hold a school district accountable with teachers, staff, students,and parents at risk, all the while the test firms rake in the dough with so little accountability is not cool either.

Jwhyperion
Jwhyperion

As an employee of a company that produces standardized tests, I will say that this article needs to be read with caution. First, please consider the sources in the article. Please note that beside the inclusion of the very truncated responses from the testing companies, these companies were not given a voice. Let me share some of my professional knowledge with you.

First, standardized testing has been going on for hundreds of years. It's called tests, quizzes, and essays in the classroom. The individual school uses assessment testing as does the school district. It's always been done this way. What has changed is the importance placed upon the results of these standardized tests. Look to the government (of either political persuasion) for the federal requirement for statewide testing.

Having taught for 10 years and now having worked for a testing company for 5, I can assure you that educators have a HUGE say in what they're kids are tested on. In all the states, committees of teachers representing all areas of the state, review each and every test question and analyze the data from the field test to determine which items show validity and will be used on an operational test. In many states, the teachers actually write the test questions. I had no idea when I first started in this business how much impact the teachers have.

When it comes to essay testing, there will always be controversy, just as there will always be differing opinions in the classroom when a teacher gives you a "C" on an essay and you KNOW you should have gotten an "A". There are things called human error and human subjectivity. I agree that essay testing for a standardized test has many inherent problems. Again, however, know that in most states the teachers write these questions and create the rubrics for how they should be scored. Also, most importantly people should know that the essay portion of the state assessment does not dictate whethere a student passes or fails. A student's overall score is determined by both MC questions and the essay question with the emphasis in scoring on the MC questions. Not all states include essay questions as part of their assessment. Many use a short answer response that asks students to elaborate upon information gained from the text while bringing in their own experiences and opinions. In this case, if a student writes on a completely different topic they receive a zero just as they would on a math test if the added two numbers incorrectly.

There are of course inherent flaws in the system just as there are inherent flaws in most every system. Instead of just complaining, though, it would be nice if the detractors could use some of their effort to help come up with a solution. As my principal always said, "Don't come to me with a problem unless you have a proposed solution."

Captainfrogbert
Captainfrogbert

What a shock! A system developed by republicans in which the only criteria is shoveling billions in tax money into the pockets of big business with no consideration of the possible damage that could be done to the lives of thousands of innocent children. Glib corporate spokesmen with stock, facile responses to serious concerns. No oversight. Nothing but making the most money off the public dole and move on. Do lots of damage to America and her children in the process. Sounds like the ideal republican government program.

Rena McGee
Rena McGee

I don't know whether to laugh and feel smug (because I think the entire testing thing is a waste of time, or just shake my head.

Jenny
Jenny

MUST READ for English teachers.

Marin
Marin

It is not possible to objectively score essay questions, therefore no tests should have essay questions.

Sophia Green
Sophia Green

Hi cutie, Could you hit me up on--- RichFlirts.C'om---A dating club for successful, beautiful people.I am a smart&pretty gal. seeking a sweet man.pls Check out my username myshine,serious...

Bev
Bev

I have a teaching degree, and have done scoring for Pearson a few times. I am always frustrated by my inability to use absoloutes, when grading a writing sample. I strive to give each student my best response. All of the other scorers know that some kids life will be affected by our decisions. No matter where the scorer got his or her Bachealor degree, all seem to take the job seriously. Bev Schunk

Kathleenwicker
Kathleenwicker

As a teacher for 23 years in South Florida... I can tell you we experience the same problems. I've had students who could barely write their name - come back with a high score and students who were talented writers get the same score. Very inequitable. Bad for schools and more importantly bad for children.

Umar Anjum
Umar Anjum

The article shocked me, I knew there were issues with how the test were scored but I had no idea it was such a work-mill of questionable qualifications

Dharm
Dharm

This is where true benefit of advanced technology like natural language processing comes. Automated essay scoring has been proven to have similar performance compared to human scorers as much one human scorer compares to another. The mathematical model of cognitive and pedagogical processes is called latent semantic analysis.

Ed Sike
Ed Sike

Sounds dreadful, but what do you (or the complaining scorers) propose as an alternative? The people who set up this system and the politicians who fund it operate under godawful pressure from the media to change things, make things better, do it now, etc. The media should therefore accept some of the responsibility for the results and explore thoughtful alternatives, but I don't see any in this story.

Red5 C9
Red5 C9

The article is very one-sided. Where do you think the money comes from to pay for testing? The state of Arkansas Dept. of Education gets their money to pay from testing somewhere, right? Like probably their constituents and/or the Feds?Having multiple players in the industry promotes competition which is good to keep prices down.Hiring 300,600, or 1000 people to score student essays has to be a huge cash outlay for any company chosen to administer state level testing. Those companies that can do it in an efficient manner and keep costs down are providing a great service in an open market.Listen, objectively scoring essays from a couple hundred thousand students cannot be easy.Would everyone prefer not to evaulate our student population? Maybe we should just ask each student what they think they should get for a grade and give those out instead?Everyone wants perfection and everyone to be treated equally. Who can answer the best way to do that?

Esbtans
Esbtans

The testing system needs to be done away with. Tests should involve putting thoughts to practical uses that don't involve pencil in paper.

TV on the VCR
TV on the VCR

I did scoring for a season or two. Three things. Setting aside for the moment how well testing assess student performance, which is debatable, this article makes scoring essays sound much harder than it is.

The guy you quote at the beginning of the article must something serious going on. Even if you want to score the creative but off topic essays higher, the number of those are really limited - like maybe 5% or less. So even if you score those drastically different than anyone else, that is only 5% of the tests, you can still screw up another 15% before anyone cares about your "agreement rate."

And there are really only a limited number of categories - it is not that hard. Often times, like the ones in the print version of this article, there are only FOUR categories:0 - Student writes "Fuck you," draws a picture, writes "I hate my teacher," just plain doesn't put anything, or has a semi-literate scribble on the page that appears to contain a couple of off topic words.1 - Average, but on the low end.2 - Average, but on the high end.4 - Creative, interesting, well written, varied sentence structure, good use of details and adjectives, etc.

Even if there are six categories, it doesn't become that much more difficult. After you read 200 or 2000 essays on the same topic, you get a pretty good sense of what is good and what isn't. If you don't get a sense of it after that long, there is something seriously wrong.

Seeoscar
Seeoscar

Yeah, it's not surprising. The move to measure is good, but measuring writing is difficult. Standardizing and measuring en masse is, as the essay demonstrates, not only difficult but profitable.

Alan Scherstuhl
Alan Scherstuhl

Great, alarming piece. Good essays and good thinking are not made up of interchangeable parts.

 
Loading...