Inside the multimillion-dollar essay-scoring business

Melissitzifun

There is really a negative slant on this story. I hope the majority of scoring centers aren't this bad. I certainly have never felt any pressure from my team leader or scoring director. I hope this is truly is a "one-sided story based upon people who have a very limited exposure and narrow point of view on what is truly a science."..... And it is true, I'm a temp and I do it for the money.... But it's a seasonal position.... The scoring centers really couldn't get the tests scored on time without using people who are qualified but out of work... The cream of the crop are not going to drop what they are doing (teaching) to sign on for a 3 month low-paying gig, Their hands are tied in some ways.....

teese02a

1 Like

@Melissitzifun , the point is that you should not be scoring these tests at all. These children should be evaluated by their teachers in their schools and all that tax payer money being spent on corporations getting rich paying you peanuts to do a job you are not qualified for should be going to educate kids not test them. Testing is not education.

Brian Pete

Extended response evaluators Please Read

Gaxlion

To gather a group of scorers a scoring business must meet the client state's request that the scorers have a certain accuracy rate, say the scorers can accurately score 70% or 80% of the trail paragraphs to the client states desired ratings. Since this is nearly impossible, the scoring business gives the potential scorers some of the answers during their qualifying tests and hints as to how to adjust other answers in order to get them qualified. I know, I'm a scorer and witnessed it first hand. Not one of the scorers in the group I was in failed to qualify. Education has been turned into a business. A business promises it's client a certain product then does what it has to to deliver. Is anyone surprised?

Jim

If you went to a reputable college anywhere in the world, standardized testing was involved as part of the requirement for admission. Our teachers, lawyers, doctors, engineers, nurses and many more "professionals" . . . all were required to take standardized . . . "admission to the profession" . . . tests. Who scores those tests? What is the argument here? If you don't test well you won't advance. It doesn't matter who scores the test or who designs the rubric. It's no different then a decision made in a court of law. If you can't make the case . . . you lose . . . unless you're OJ Simpson. Most people living in the United States know that a person who faces a jury trial, faces a group composed of their "peers". They follow a rubric called the rule of law. These people aren't required to know the name of a college (let alone have a college degree).

The wealthy people will most likely go to the "favored" schools. Most people will live in the "imperfect" world. People, including the educational psychologist you quoted in your article (to this person: GRE ring a bell . . . how about ACT or SAT), need to accomodate and adjust and quit eating sour grapes. Personally, I have a problem associating with people who over-compensate for a lack of ability to embrace the imperfection of humanity and make a decision to be happy.

Jim Herndonern

1 Like

I give his a 2.

RNelson

So much for following the rubric. Our state's requires a 0-4 grade in five different areas (organization, student voice, sentence formation, etc). And each of those has several variables. But hey, your "rubric" is probably right.

never-to-return

1 Like

Blame the teachers? Teachers can only work with what the parents send them. In ever-increasing numbers. Teachers are expected to essentially babysit forty kids whose parents either can't or won't raise them to be able to be taught anything in the first place. Whether they are single parents, or still-married parents who are both forced to work full-time just to make ends meet, or simply do not care or are depressed or on drugs. Trace it all back to the vast income disparity. Educate yourself a little about profit-mongering. Blame the teachers, indeed.

Jaydub1948

Living in a town like C-bus with lots of unemployed college grads, I've done plenty of seasonal work grading standardized tests. I have to agree with most of what was said in the article regarding the need to conform to the rubrics. It didn't take long to figure out what they wanted and score accordingly and with speed. It is quite true that the retired teachers, as well as unemployed librarians, could not "pull the trigger" as one of my team leaders would say.

Darryl Thomas

Yes, it's all about the money. Our country's future be damned.

20autismmom10

While I hate to annoy people by holding up Canada as an example, we need to do what they have all along - use teachers and administrators each year to determine what a child should know and prepare, test and grade it themselves. This test-taking mania started with NCLB to help benefit Neil Bush, George W's brother, who started a business in "test-taking" materials, supplemental "educational" materials and has never had anything to do with the educational welfare of our nations' students. It was always about making a buck at the public's expense and look what's been created. Add to that the charter movement that proves to discriminate against special needs and English Language Learner students (how do you think they "claim" better test scores?) and you will see charters (as businesses) being handed these failing schools to then remove those students who are in need of the most help. The destruction of public education is a well-planned business maneuver to acquire public property and federal/state funding to use as they please for their discriminatory little school universe. It's a crime of national proportions, but we're all just lying down and letting them ruin and then give our public institutions away to the highest bidder.

Accidentalfission

1 Like

Neil Bush's company was (is?) called Ignite IIRC and was in the business of selling NCLB test prep materials. This was a the same time that the Bush administration was paying columnists under the table to write positively about NCLB.

Does anyone remember this Did our "liberal media" ever comment on this?

Michael Hardy

We need to just drop an immense nuclear weapon of mass destruction on the whole standardized testing industry.

Carlos Mandelbaum

Disturbing.

Guest

But test scores are all the fault of teachers.

okami

I'm beginning to wonder about the education of the people in charge of this affair. . .

The Free School Apparent

Once again, the curtain is parted and the wizard is shown to be a less than ordinary man.Is anything in our modern life free from the profit disease?

Ron Amundson

One solution to this is to require clawbacks as part of the contract with standardized testing companies based upon a students future performance. In other words, the testing industry would in effect be accountable for their methods of assessment (as well as fraud detection).

For example, if a student who does very well on the standardized tests, either MC or essay, and then said student proceeds to flunk out,or even get C's or D's while in high school or college years later, significant penalties would then be automatically and retroactively imposed upon the grading firm. It likely would be necessary to require bonds to be posted up front as part of said contract, as a firms viability cannot be predicted over the long haul. Granted, there are any number of factors well beyond the control of the assessment firms..but then again insurance actuaries do the same kind of thing nearly every day.

Granted, such an approach might prove to be too expensive to operate on shore, being it would likely equate to highly trained graders working together, and having enough time to achieve a level of accuracy such that their firm would not end up loosing their shirt years down the road (or go out of business today, as they could no longer afford the bond). On the other hand, having the assessment firm being at more than are arms length from those who would wish to influence outcomes might not be such a bad thing either.

I guess I;m half sarcastic,and half idealistic in this... it would throw such a huge wrench into the status quo, and many firms would go under, and likewise many would lose their jobs. On the other hand, to hold a school district accountable with teachers, staff, students,and parents at risk, all the while the test firms rake in the dough with so little accountability is not cool either.

Jwhyperion

As an employee of a company that produces standardized tests, I will say that this article needs to be read with caution. First, please consider the sources in the article. Please note that beside the inclusion of the very truncated responses from the testing companies, these companies were not given a voice. Let me share some of my professional knowledge with you.

First, standardized testing has been going on for hundreds of years. It's called tests, quizzes, and essays in the classroom. The individual school uses assessment testing as does the school district. It's always been done this way. What has changed is the importance placed upon the results of these standardized tests. Look to the government (of either political persuasion) for the federal requirement for statewide testing.

Having taught for 10 years and now having worked for a testing company for 5, I can assure you that educators have a HUGE say in what they're kids are tested on. In all the states, committees of teachers representing all areas of the state, review each and every test question and analyze the data from the field test to determine which items show validity and will be used on an operational test. In many states, the teachers actually write the test questions. I had no idea when I first started in this business how much impact the teachers have.

When it comes to essay testing, there will always be controversy, just as there will always be differing opinions in the classroom when a teacher gives you a "C" on an essay and you KNOW you should have gotten an "A". There are things called human error and human subjectivity. I agree that essay testing for a standardized test has many inherent problems. Again, however, know that in most states the teachers write these questions and create the rubrics for how they should be scored. Also, most importantly people should know that the essay portion of the state assessment does not dictate whethere a student passes or fails. A student's overall score is determined by both MC questions and the essay question with the emphasis in scoring on the MC questions. Not all states include essay questions as part of their assessment. Many use a short answer response that asks students to elaborate upon information gained from the text while bringing in their own experiences and opinions. In this case, if a student writes on a completely different topic they receive a zero just as they would on a math test if the added two numbers incorrectly.

There are of course inherent flaws in the system just as there are inherent flaws in most every system. Instead of just complaining, though, it would be nice if the detractors could use some of their effort to help come up with a solution. As my principal always said, "Don't come to me with a problem unless you have a proposed solution."

RNelson

OK, I'm trying to be objective, but let me try to calmly take exception to some of your points.

"...standardized testing has been going on for hundreds of years." I'm not sure that's true, depending on how you define "standardized." Yes, a teacher or school may have decided their own standard, and tested accordingly, but if you mean standards that are set by an outside agency, in our case by the Arkansas Department of Education, then no, I'm sure it hasn't been hundreds of years.

I teach 7th grade, and we are required to give interim assessments throughout the year, to determine if our students are making progress towards a successful standardized test in the spring. Yes, we wrote the rubric, based on the state rubric written by teachers. I have a degree in English and I teach this stuff, and it took me 2-3 years to feel that I had a competent grasp on the rubric. It is denser than it looks. It still takes me 3-10 minutes per essay to evaluate them fairly. But I'm sure that temporary workers are better qualified than I to make these judgements. The thing in the article that disturbed me was the discomfort shown when retired teachers protested. Is it because they are wrong, or just unprofitable?

"...the essay portion of the state assessment does not dictate whether a student passes or fails. A student's overall score is determined by both MC questions and the essay question with the emphasis in scoring on the MC questions." Our state assessment, for literacy, as we are coached and reminded many times throughout the year, takes 67% of it's total score from the essay and open response (writing) portions. And while failing it in 7th grade will not (yet) keep you from passing to 8th, the End of Course literacy in 11th grade is being phased in as a requirement for graduation. I believe it will happen within the next 5 years. So please don't downplay these concerns.

I'm not naive, I understand that businesses exist to generate profit. And it doesn't have to be mutually exclusive of doing good, reputable work. I believe these companies want to do the best job they can - within the confines of the business model. But teachers are also doing their best, and many states are now tying my salary, my job performance, even my very security to the results of these tests. So while my students may not "fail" because Harry the laid-off refrigerator guy does a so-so job, my very job could depend on him. And on his ability to make nuanced decisions about the writing of a 12 year old.

I hope that we can find a way to increase student achievement, measured in multiple ways, for all. One that will work just as hard to reform what isn't best in test scoring as it does to fight what isn't working in teaching. We should all be accountable for changing young lives for the better.

msndis

"educators have a HUGE say in what they're kids are tested on"

If you've been at an education testing company for five years, I assume you know about the Common Core State Standards. With their national adoption, which is almost complete, there will be NO local/state control over what is on the standardized tests, nor the curriculum students use.

I do realize you need a job and you choose to continue working at the testing company despite the problems but I hope you also realize that the problems with the assessment process can affect students for the rest of their lives. In Washington State, if a scorer doesn't read a 10th grade essay well and gives a non-passing score because he/she didn't wear glasses, is hung over, is on pain meds from a recent surgery, etc., the student won't graduate from high school unless he/she can pass one of the re-take exams or an alternative. So, in my opinion, there is no room for error.

As a parent, I have never trusted the scoring of the state-mandated assessments and opted one daughter out of them for a few years (she had to take them this year as a graduation requirement). This article just reinforces my believe that these assessments have to go.

I have a question I'm hoping a commenter can answer - are the SAT writing tests also scored in scoring mills?

Leharper

No. I was a scorer for ETS. They are also graded on a rubric and there is certainly pressure to conform for the purposes of standardization. (This looks like a 4, but I think it is a 5, but I think everyone else is going to give it a 4 so I'll give it a 4.) Standardized tests are simply not good tools to measure divergent thing. Not on the multiple choice section, not in the essay section. That being said, I felt the ETS strove to be ethical.

Danthelawyer

What "their" kids are tested on; not what "they're" kids are tested on. Of course, one typo does not invalidate this writer's position. But it does throw his or her proofreading skills into question.

Gregor Samsa

Can you at least admit that you're a terrible writer yourself? "It's called tests, quizzes, and essays in the classroom." "Look to the government (of either political persuasion)..." Do you mean look at either political party? Writing effectively is a relatively rare skill even among the most highly educated. Perhaps testing should shift its focus away from writing and focus on skills that students are more likely to develop more robustly (and that scorers are more likely to have a knowledge of).

Patricia Trentacoste

You've offered a sincere response to “Inside the multimillion-dollar essay-scoring business.” In fact much of what you say is true, but beside the point, and consequently, no rebuttal. Here's why: You write that readers should ‘consider the sources’ of the accusation but this is a fallacious dodge frankly, because the qualifications of a critic are irrelevant to the substance of the criticism. Either the claims made by the article are true or false. Which claims in this article are false? You don't say. Instead you offer several asides which miss the point. For instance, you argue that the testing industry itself should have a say in an independent critique of its role in society. But clearly an internal audit is not appropriate for determining the impact of ST (standardized testing) on external parties with no such input, nor of the unethical practices within the industry (e.g., essay scoring mills). The burden of proof for ST advocates is to empirically show that the ST industry, including the essay scoring mills, are in fact improving the quality of education in America and promoting the flourishing of youth without using arbitrary or otherwise harmful methods or unnecessarily destroying a system that works when properly funded. What say you to the complaint that the real beneficiaries of the industry have turned out to be those who have profited from selling both the testing instruments and the remedial and tutorial tools made necessary by those very same testing instruments?

You claim 'standardized testing' has been going on for centuries. But you are clearly confusing testing constructed by a particular teacher of what s/he taught in his/her own classroom, with testing constructed by ST test writers (some of whom are teachers, some not) concerning subject matter that should have or could have been taught in a generic classroom. Your comments miss the central point that a test prepared by someone who's never met the child being tested, never sat in his or her desk—often elbow to elbow with too many classmates, never tasted the school lunch fare, never rode on that child's bus, never met that child's actual teacher, or understood his/her home life, is not qualified to assess that child's learning capacity and achievement. Worse, defense of ST practices as a means of withholding funding from innocent children treats equal educational opportunity as a privilege, not as a social good and a social right in a free civilized nation.

Ultimately, defense of ST's association with NCLB implies that what Mary scores on her test suffices to deprive Tommy of percapita educational funding. How can that be? NCLB and ST are designed in principle as assessment tools for abstractly quantifying a concrete qualitative experience—education. The task is impossible, yet the model is being foisted on the country via propaganda and vetted politicians as though it is actually working to uplift the social good. There is only one solution: abolish both NCLB and ST; do not prop them up with limp reforms, no matter how well intended. Unless of course, the agenda really is to destroy the social good for private profit… Patricia Trentacoste Full time adjunct philosophy professor

Captainfrogbert

What a shock! A system developed by republicans in which the only criteria is shoveling billions in tax money into the pockets of big business with no consideration of the possible damage that could be done to the lives of thousands of innocent children. Glib corporate spokesmen with stock, facile responses to serious concerns. No oversight. Nothing but making the most money off the public dole and move on. Do lots of damage to America and her children in the process. Sounds like the ideal republican government program.

L Abdo

The problem is all politicians, not just one party or another. Who helped that law get passed? Why is President Obama proposing a law that has many similarities? Complaining about a political party doesn't change anything. Reform of the entire political system and educational system are needed and people pointing fingers are delaying the process, hence the fiasco that is Washington, D.C.

Rena McGee

I don't know whether to laugh and feel smug (because I think the entire testing thing is a waste of time, or just shake my head.

LUV_AZ

Rena, and how do you propose we access students?

msndis

Hmmmm . . . maybe let their teachers assess them like they did when I was growing up. Also, if you want a nationally-normed test, how about the ITBS, there is nothing subjective in the scoring of that test and it is CHEAP!

Jenny

MUST READ for English teachers.

Marin

It is not possible to objectively score essay questions, therefore no tests should have essay questions.

P. David Albee

still waiting for superman...

Sophia Green

Hi cutie, Could you hit me up on--- RichFlirts.C'om---A dating club for successful, beautiful people.I am a smart&pretty gal. seeking a sweet man.pls Check out my username myshine,serious...

Bev

I have a teaching degree, and have done scoring for Pearson a few times. I am always frustrated by my inability to use absoloutes, when grading a writing sample. I strive to give each student my best response. All of the other scorers know that some kids life will be affected by our decisions. No matter where the scorer got his or her Bachealor degree, all seem to take the job seriously. Bev Schunk

Kathleenwicker

As a teacher for 23 years in South Florida... I can tell you we experience the same problems. I've had students who could barely write their name - come back with a high score and students who were talented writers get the same score. Very inequitable. Bad for schools and more importantly bad for children.

Umar Anjum

The article shocked me, I knew there were issues with how the test were scored but I had no idea it was such a work-mill of questionable qualifications

Dharm

This is where true benefit of advanced technology like natural language processing comes. Automated essay scoring has been proven to have similar performance compared to human scorers as much one human scorer compares to another. The mathematical model of cognitive and pedagogical processes is called latent semantic analysis.

Sabrina Stevens

But those computerized scoring models are based on the performance of these kinds of scorers. And agreement is not the same as accuracy-- two scorers might both low-ball or high-ball a student's work. Using computers may speed up the process and bring down costs, but it won't ensure that children's work is being accurately measured. This is to say nothing of the unchecked biases and assumptions (cultural, etc.) analysts bring with them as they build these new systems. Those biases and assumptions will be built into their mathematical/statistical models, privileging the performance of kids who happen to fit a given mold over that of kids who don't. Just because numbers have been attached to an end result does not mean that the process generating those results is by any means objective. That's pretty much impossible, in this case.

The key statement in all of this is that of the professor who notes that it's impossible to standardize in-depth performance measures. There will always be problems like these as long as we continue to try to standardize things that aren't standard, especially under conditions of intense political and social pressure.

Meaningful displays of intelligent performance require human judgment in order to be fairly assessed. So we have to ask ourselves: whose judgment do we trust more? That of trained educators who witness children's development over time, or that of rushed, faraway temporary employees whose only interaction with said children is during a 2-minute-or-less skimming of their responses on a "standardized" test?

HangoTango

lol, well you know what they say. If there is a buck to be made lol.

www.complete-privacy.eu.tc

Ed Sike

Sounds dreadful, but what do you (or the complaining scorers) propose as an alternative? The people who set up this system and the politicians who fund it operate under godawful pressure from the media to change things, make things better, do it now, etc. The media should therefore accept some of the responsibility for the results and explore thoughtful alternatives, but I don't see any in this story.

egc52556

"The media"? Are you joking? Those poor, poor politicians cringing under the whip of their "media" overlords...

Red5 C9

The article is very one-sided. Where do you think the money comes from to pay for testing? The state of Arkansas Dept. of Education gets their money to pay from testing somewhere, right? Like probably their constituents and/or the Feds?Having multiple players in the industry promotes competition which is good to keep prices down.Hiring 300,600, or 1000 people to score student essays has to be a huge cash outlay for any company chosen to administer state level testing. Those companies that can do it in an efficient manner and keep costs down are providing a great service in an open market.Listen, objectively scoring essays from a couple hundred thousand students cannot be easy.Would everyone prefer not to evaulate our student population? Maybe we should just ask each student what they think they should get for a grade and give those out instead?Everyone wants perfection and everyone to be treated equally. Who can answer the best way to do that?

"Those companies that can do it in an efficient manner and keep costs down are providing a great service in an open market."

...unless they cannot train scorers to handle responses that are unconventional or ignore that reality, pressure scorers to meet efficiency targets that hinder full consideration of each response, and essentially treat Writing skills as they would an objective set of questions.

I agree that everyone demands perfection, which is unrealistic. But, you CAN get very good and efficient results with smaller, better-trained groups of scorers in an environment that is amenable to thoughtful assessment. The cattle-call method is not the only way, just the cheapest and most convenient. States should look into this and reconsider the best, rather than most expedient, method to assess our students.

Esbtans

The testing system needs to be done away with. Tests should involve putting thoughts to practical uses that don't involve pencil in paper.

ipguy

How about math? Do we need to do away with math tests also?

msndis

I don't know if you've heard about the questions on the standardized math tests but a few years ago my oldest daughter took the Washington State 10th grade math test. One of the questions was about the factors that influence the cost of car insurance (she was 15 at the time and had never bought car insurance). Another question asked her to "describe" a math worksheet to her friend over the phone. What is correct in that instance? NO math was required on those questions and that was from a section that calculators could be used on. Pathetic!

I prefer that students are testing in their classrooms by their teachers. As a parent, that gives me a very good idea how they are doing in each subject. The state standardized tests are a joke and really need to go. In addition, they are extremely expensive and I, personally, prefer that the money they are sucking up be spent directly in the classroom.

tschuster62

@Carl Carl,

No, those questions were probably not written by teachers. Testing companies maintain that teachers have "input" in the development of test questions. What this amounts to is having teachers read lists of questions, they respond "This question is awful" and test companies say "Thank you for your input" and continue to use the question. Read "Making the Grades" by Todd Farley.

Carl

I know some educators in Washington state, and all their test questions are written by teachers in Washington state. From what the educators have told me, there are a lot of questions written and used that go through a testing process themselves that don't affect a student's grade. Once through this testing process the questions are deemed useful or not. There is also a series of state standards the questions must all adhere to, and no questions would appear that do not meet those standards in some way. My guess is that the questions you think are questionable A) fulfilled certain state standards and B) were possibly merely being tested to see if students respond as expected.

TV on the VCR

I did scoring for a season or two. Three things. Setting aside for the moment how well testing assess student performance, which is debatable, this article makes scoring essays sound much harder than it is.

The guy you quote at the beginning of the article must something serious going on. Even if you want to score the creative but off topic essays higher, the number of those are really limited - like maybe 5% or less. So even if you score those drastically different than anyone else, that is only 5% of the tests, you can still screw up another 15% before anyone cares about your "agreement rate."

And there are really only a limited number of categories - it is not that hard. Often times, like the ones in the print version of this article, there are only FOUR categories:0 - Student writes "Fuck you," draws a picture, writes "I hate my teacher," just plain doesn't put anything, or has a semi-literate scribble on the page that appears to contain a couple of off topic words.1 - Average, but on the low end.2 - Average, but on the high end.4 - Creative, interesting, well written, varied sentence structure, good use of details and adjectives, etc.

Even if there are six categories, it doesn't become that much more difficult. After you read 200 or 2000 essays on the same topic, you get a pretty good sense of what is good and what isn't. If you don't get a sense of it after that long, there is something seriously wrong.

RNelson

So much for scoring by the rubric. Our state's rubric DOES have 5 categories (organization, voice, sentence formation, etc.) with several variables that need to be assessed to determine the score in each. But hey, your "rubric" is probably better.