Inside the multimillion-dollar essay-scoring business

Behind the scenes of standardized testing

"I was like, 'Yeah, I have a degree, I can do that,'" she recalls.

On Indovino's first day, she drove out to Questar Assessment in Apple Valley, a beige warehouse, and followed the signs that said "Scoring Center" in bright red letters. During her brief interview, she'd been asked repeatedly if she was able to follow a "rubric"—a set of guidelines to assess the essays in as uniform a way as possible.

"I guess they've had bad experiences with English teachers," she says.

Inside Questar, Indovino took a seat in a room that looked like a classroom, crammed with as many computers and desks as could fit. It was here that the team leaders unveiled the scoring rubric, which was like a secret decoder ring for the job.

The rubrics are most often developed in conjunction with the state's department of education and its testing contractor. Currently, Minnesota contracts both its test writing and scoring to Pearson. Local teachers are included in the rubric-writing process, as well as test-writing academics called "psychometricians."

At first blush, the rubric seemed simple enough to Indovino. It was a chart with one- or two-sentence explanations of each number grade. Scorers are forbidden from taking the rubrics out of the Questar building or talking about them, but they generally look something like this:

6. An excellent response, the essay includes

• excellent focus and development

• excellent organization

• excellent language skills and word choice

• excellent grammar, usage, and mechanics

5. A good response, the essay includes

• good focus and development

• good organization

• good language skills and word choice

• good grammar, usage, and mechanics

4. An adequate response ...

On down to 1s, which were reserved for barely decipherable language.

As part of their training, Indovino and her co-workers read through pre-graded examples out loud, then discussed why each had been scored the way it was. The process quickly divided the room into two camps—the young, unemployed kids who were just there for a paycheck, and the retired teachers.

"The retired teachers would argue everything," says Indovino.

After two days of going through example papers, each scorer had to pass a qualifying exam. Indovino scored three sets of ten pre-scored papers. In order to be approved to work on the project, she had to pass two of the sets with at least an 80 percent "agreement rate" with the rubric. She did so with relative ease; most of the rest of the room passed on their second try.

Her first project was from Arkansas, an essay written by eighth-graders on the topic, "A fun thing to do in my town."

And that's where the troubles began.

Suddenly, she was being asked to crank through 200 real essays in a day. The scanned papers popped up on the screen and her eyes flitted as fast as they could down the lines. The difference between "excellent" and "good" and "adequate" was decided in a matter of seconds, to say nothing of the responses that were simply off the reservation. How do you score a kid who rails that his town sucks? What about an exceptionally well-written essay on why the student was refusing to answer the question?

All over the room, the teachers were raising their hands and disputing the rubric. Indovino preferred to keep her head down and just score the way she was told to.

"I was good at the bad system," she says.

Over the next several months, Indovino got to know her co-workers better. The young people were mostly laid off or in foreclosure. They came straight from paper routes and went off to waitressing jobs afterward.

They also made for a very dedicated workforce. Indovino says she saw her co-workers hung-over, extremely ill, and even fresh from surgery.

"I scored a full day without glasses on," Indovino says with a shrug. "I sat with my nose up to the glass all day. I couldn't read it."

When she eventually got a full-time job, Indovino quit scoring. Although she'd done well by the company's standards, following the rubric provided little sense of accomplishment.

"Nobody is saying, 'I'm doing good work, I'm helping society,'" she says. "Everyone is saying, 'This isn't right.'"

  

DAVID PUTHOFF WAS an experienced reader with Questar when he started getting the warnings that his job performance wasn't up to snuff.

"Your numbers are down a little bit," his supervisor said at the end of one day. "Make sure you bring those back up."

Most essays, depending on the criteria established in the state, are scored by two readers. As Puthoff and his fellow scorers whipped through their essays, their supervisor had their own eyes glued to a screen, keeping them apprised of whether Reader #1 agreed with Reader #2. If so, both got a 100 percent agreement score for that essay. If one differed by a point or so, the essay would be counted as "adjacent" agreement.

Puthoff had thus far been an agreement-rate superstar. He was consistently in the high 80s.

Then came the question from hell out of Louisiana: "What are the qualities of a good leader?"

« Previous Page
 |
 
1
 
2
 
3
 
4
 
All
 
Next Page »
 
My Voice Nation Help
82 comments
ikilledtheshark
ikilledtheshark

The moneyed employing class has visited upon my generation a contrived denial of the chance to live up to our potential and contribute to society in a meaningful way. The atrocities that continue to be committed behind closed doors in secretive boardroom meetings are criminal at best under the RICO Act, and treasonous to present and future generations of humanity. Because the majority of government entities, as well as many "officials", have been complicit in these plans to block prosperity from millions of people, to obscure and obstruct the paths taken by former generations to reach the American dream, and  have provided cover to those foisting this reality upon us, they are equally guilty. But yet, there is more money in existence than at any prior time in history. Who is the authority here? The Declaration of Independence and US Constitution suggest that the power rests with the people.
It is plain to see, if you look past the distractions constantly hurtled at us, that the goalpost has been moved. Much like Lucy and Charlie Brown in the "Peanuts" cartoon, there isn't even a ball in play; that is, unless, like the more fortunate half of my peers, one starts out in life with advantages that others don't have. For the rest of us, it's a crap shoot. However, any more than two bouts with misfortune and you wind up in the boat with us. Instead, our children, your grandchildren and great-grandchildren have been abjectly sentenced to a destitute future full of uncertainty, instability, lack of good nutrition, safety, a sound education, health, and promising careers. Opportunities are limited to floating from temp. jobs to seasonal positions @ minimum wage (1/3 of a living) with no security. In the ancient Roman Empire, the "insulae" were poor people living in small, crowded apartments with only 2-3 rooms which housed their large families.
The broken promises, the failed policies (trickle-down, deregulation, free-market), the out and out LIES and deception, the smoke and mirrors from the warped views of PIGS in the Boardroom, in Congress, in the Whitehouse, on the Bench, in the Statehouses, etc.. is dismaying but it is all adding up. People need to start taking this seriously. Those named above are hard at work at ripping this country up and tearing it down. I know it; you know it.  It has been going on for 33 years, at a minimum, and needs to be brought to an end.
All hope is NOT lost.  Keep spreading and speaking truth to power, while the masses are continuing to awaken.

MrT51
MrT51

... If this isn't TRANSFER OF PUBLIC WEALTH TO CORPORATIONS, complete bastardization of the process of education, and selling our children's future for small change .. I don't know what is. The stink of this trend is what riled my guts and made me retire. Kids and teachers are jumped through hoops in order to enrich testing corporations. The fabrication of data, side-tracking the really brilliant kids, and cowing of teachers with poor test scores resulting from fraudulent testing. When the teachers and students were using the rubrics and scoring and kids were revising their work and constructing portfolios of their best creations, I thought we were really onto something. Authentic assessment of real student projects gathered for all to see. When repub legislators started complaining of the cost of university scoring and portfolio storage/transport, I just thought "and our kids aren't worth it?". They won - cancelled these all authentic assessments, literally dumped the portfolios and went to the corporate grading with repub legislatures awarding big corporate contracts and outsourcing the grading and assessment jobs out of the state. Suddenly we were pressed to drill kids for fill in the blanks and there was no time for portfolio projects or revising essays. It has degraded far more in the 8 yrs since I have left. My competent administrators have left. My colleagues who are left are demoralized and are "most-experienced=first to be laid off" status. The supposed anti-big givt Repubs have used big brother tactics to strip local schools boards and teachers of funding, self-govt, and any legitimate teaching of hard skills and inventiveness. Our dropout rate and teacher attrition rate speaks loudly of this. Nothing will change until parents vote out the repub privatizers of public schooling and their for-profit schemes - lining the pockets of their buddies who have just happened to start up private sector for profit education grifter schemes. Education, healthcare and other human services are not served by the profit-making ethic. I'd love to see a renaissance for the kids' sake but these Repubs are out for blood money and pure greed. This is not the crisis that the teacher-bashing reform movement of Michelle Rhee is even pointing out at present. One generation of this corporate takeover of our kids' futures and road map to the health of the national mind could be lost.

englishteacher
englishteacher

The worst thing is that scores on these kinds of tests are replacing experienced teachers' judgments about how their students' writing has developed over time.  Teachers in many schools and districts are being pressured to make their grades conform to scores on these kinds of tests, rather than the other way around.  

mariabronn
mariabronn

The most disgusting part of this story is the part where the scores are artificially inflated to make the proper bell curve with the total amount of results.  The kids whose papers were already graded got screwed, especially if someone argued in favor of a higher score but was overruled!  Then, kids who didn't do as good a job as they did got higher scores just so the testing results would appear to be accurate and valid, which is total crap!

I hope our elected officials responsible for this incredible mess are proud of themselves, because if they aren't, nobody else would be.  That's for sure.

Melissitzifun
Melissitzifun

There is really a negative slant on this story. I hope the majority of scoring centers aren't this bad. I certainly have never felt any pressure from my team leader or scoring director. I hope this is truly is a "one-sided story based upon people who have a very limited exposure and narrow point of view on what is truly a science."..... And it is true, I'm a temp and I do it for the money.... But it's a seasonal position.... The scoring centers really couldn't get the tests scored on time without using people who are qualified but out of work... The cream of the crop are not going to drop what they are doing (teaching) to sign on for a 3 month low-paying gig, Their hands are tied in some ways.....

Brian Pete
Brian Pete

Extended response evaluators Please Read

Gaxlion
Gaxlion

To gather a group of scorers a scoring business must meet the client state's request that the scorers have a certain accuracy rate, say the scorers can accurately score 70% or 80% of the trail paragraphs to the client states desired ratings. Since this is nearly impossible, the scoring business gives the potential scorers some of the answers during their qualifying tests and hints as to how to adjust other answers in order to get them qualified. I know, I'm a scorer and witnessed it first hand. Not one of the scorers in the group I was in failed to qualify. Education has been turned into a business. A business promises it's client a certain product then does what it has to to deliver. Is anyone surprised?

Jim
Jim

If you went to a reputable college anywhere in the world, standardized testing was involved as part of the requirement for admission. Our teachers, lawyers, doctors, engineers, nurses and many more "professionals" . . . all were required to take standardized . . . "admission to the profession" . . . tests. Who scores those tests? What is the argument here? If you don't test well you won't advance. It doesn't matter who scores the test or who designs the rubric. It's no different then a decision made in a court of law. If you can't make the case . . . you lose . . . unless you're OJ Simpson. Most people living in the United States know that a person who faces a jury trial, faces a group composed of their "peers". They follow a rubric called the rule of law. These people aren't required to know the name of a college (let alone have a college degree).

The wealthy people will most likely go to the "favored" schools. Most people will live in the "imperfect" world. People, including the educational psychologist you quoted in your article (to this person: GRE ring a bell . . . how about ACT or SAT), need to accomodate and adjust and quit eating sour grapes. Personally, I have a problem associating with people who over-compensate for a lack of ability to embrace the imperfection of humanity and make a decision to be happy.

RNelson
RNelson

So much for following the rubric. Our state's requires a 0-4 grade in five different areas (organization, student voice, sentence formation, etc). And each of those has several variables. But hey, your "rubric" is probably right.

never-to-return
never-to-return

Blame the teachers? Teachers can only work with what the parents send them. In ever-increasing numbers. Teachers are expected to essentially babysit forty kids whose parents either can't or won't raise them to be able to be taught anything in the first place. Whether they are single parents, or still-married parents who are both forced to work full-time just to make ends meet, or simply do not care or are depressed or on drugs. Trace it all back to the vast income disparity. Educate yourself a little about profit-mongering. Blame the teachers, indeed.

Jaydub1948
Jaydub1948

Living in a town like C-bus with lots of unemployed college grads, I've done plenty of seasonal work grading standardized tests. I have to agree with most of what was said in the article regarding the need to conform to the rubrics. It didn't take long to figure out what they wanted and score accordingly and with speed. It is quite true that the retired teachers, as well as unemployed librarians, could not "pull the trigger" as one of my team leaders would say.

20autismmom10
20autismmom10

While I hate to annoy people by holding up Canada as an example, we need to do what they have all along - use teachers and administrators each year to determine what a child should know and prepare, test and grade it themselves. This test-taking mania started with NCLB to help benefit Neil Bush, George W's brother, who started a business in "test-taking" materials, supplemental "educational" materials and has never had anything to do with the educational welfare of our nations' students. It was always about making a buck at the public's expense and look what's been created. Add to that the charter movement that proves to discriminate against special needs and English Language Learner students (how do you think they "claim" better test scores?) and you will see charters (as businesses) being handed these failing schools to then remove those students who are in need of the most help. The destruction of public education is a well-planned business maneuver to acquire public property and federal/state funding to use as they please for their discriminatory little school universe. It's a crime of national proportions, but we're all just lying down and letting them ruin and then give our public institutions away to the highest bidder.

Michael Hardy
Michael Hardy

We need to just drop an immense nuclear weapon of mass destruction on the whole standardized testing industry.

Guest
Guest

But test scores are all the fault of teachers.

okami
okami

I'm beginning to wonder about the education of the people in charge of this affair. . .

The Free School Apparent
The Free School Apparent

Once again, the curtain is parted and the wizard is shown to be a less than ordinary man.Is anything in our modern life free from the profit disease?

Ron Amundson
Ron Amundson

One solution to this is to require clawbacks as part of the contract with standardized testing companies based upon a students future performance. In other words, the testing industry would in effect be accountable for their methods of assessment (as well as fraud detection).

For example, if a student who does very well on the standardized tests, either MC or essay, and then said student proceeds to flunk out,or even get C's or D's while in high school or college years later, significant penalties would then be automatically and retroactively imposed upon the grading firm. It likely would be necessary to require bonds to be posted up front as part of said contract, as a firms viability cannot be predicted over the long haul. Granted, there are any number of factors well beyond the control of the assessment firms..but then again insurance actuaries do the same kind of thing nearly every day.

Granted, such an approach might prove to be too expensive to operate on shore, being it would likely equate to highly trained graders working together, and having enough time to achieve a level of accuracy such that their firm would not end up loosing their shirt years down the road (or go out of business today, as they could no longer afford the bond). On the other hand, having the assessment firm being at more than are arms length from those who would wish to influence outcomes might not be such a bad thing either.

I guess I;m half sarcastic,and half idealistic in this... it would throw such a huge wrench into the status quo, and many firms would go under, and likewise many would lose their jobs. On the other hand, to hold a school district accountable with teachers, staff, students,and parents at risk, all the while the test firms rake in the dough with so little accountability is not cool either.

Jwhyperion
Jwhyperion

As an employee of a company that produces standardized tests, I will say that this article needs to be read with caution. First, please consider the sources in the article. Please note that beside the inclusion of the very truncated responses from the testing companies, these companies were not given a voice. Let me share some of my professional knowledge with you.

First, standardized testing has been going on for hundreds of years. It's called tests, quizzes, and essays in the classroom. The individual school uses assessment testing as does the school district. It's always been done this way. What has changed is the importance placed upon the results of these standardized tests. Look to the government (of either political persuasion) for the federal requirement for statewide testing.

Having taught for 10 years and now having worked for a testing company for 5, I can assure you that educators have a HUGE say in what they're kids are tested on. In all the states, committees of teachers representing all areas of the state, review each and every test question and analyze the data from the field test to determine which items show validity and will be used on an operational test. In many states, the teachers actually write the test questions. I had no idea when I first started in this business how much impact the teachers have.

When it comes to essay testing, there will always be controversy, just as there will always be differing opinions in the classroom when a teacher gives you a "C" on an essay and you KNOW you should have gotten an "A". There are things called human error and human subjectivity. I agree that essay testing for a standardized test has many inherent problems. Again, however, know that in most states the teachers write these questions and create the rubrics for how they should be scored. Also, most importantly people should know that the essay portion of the state assessment does not dictate whethere a student passes or fails. A student's overall score is determined by both MC questions and the essay question with the emphasis in scoring on the MC questions. Not all states include essay questions as part of their assessment. Many use a short answer response that asks students to elaborate upon information gained from the text while bringing in their own experiences and opinions. In this case, if a student writes on a completely different topic they receive a zero just as they would on a math test if the added two numbers incorrectly.

There are of course inherent flaws in the system just as there are inherent flaws in most every system. Instead of just complaining, though, it would be nice if the detractors could use some of their effort to help come up with a solution. As my principal always said, "Don't come to me with a problem unless you have a proposed solution."

Captainfrogbert
Captainfrogbert

What a shock! A system developed by republicans in which the only criteria is shoveling billions in tax money into the pockets of big business with no consideration of the possible damage that could be done to the lives of thousands of innocent children. Glib corporate spokesmen with stock, facile responses to serious concerns. No oversight. Nothing but making the most money off the public dole and move on. Do lots of damage to America and her children in the process. Sounds like the ideal republican government program.

buster13
buster13

@MrT51 Agree whole-heatedly with everything said here. I am a recent retiree as well and a young one at that. I retired much earlier than I have ever intended but the inner turmoil and emotional toll of knowing the crimes being committed against our youth and future drove me to the brink. For those who remain in the field & are waking up at night sweating & worrying about whether they have properly prepared their students to pass the test only to have the grading of them be a farce? This realization will just be one more notch in every good teacher's inevitable decline into disillusionment , disgust and lunacy. 


I'm not sure that even one generation of students OR teachers will survive this and I am darn sure it will take MANY generations to overcome it. 

MrT51
MrT51

No ones hands were tied. Republicans eager to transfer wealth to their corporate buddies who started up privatization schemes in states were Repubs had starved schools of $$, simplly dismantled a perfectly well-run assessment program at the local University of Michigan staffed with retired master teachers who did a professional job using tried and true essay topics, training programs and rubrics which would never have led to the nightmares in the rats maze described in this article. Our graders knew exactly what to do with the divergent but brilliant essays and other to be expected bizarre responses from kids on any given day. They weren't forced into being a "2". I participated and spearheaded much of this in our district. To go through the training made poor to mediocre teachers in all content areas much more skilled and therefore confident writers and evaluators who were becoming more likely to assign really good writing tasks. The repub-majority killed the whole test, it's evaluation/teacher training program and the public university grading service. It was "too expensive." Everything goes to temps in Minnesota now and the district pays twice as much. Greed, self-service, and corruption rules schools at present. Don't kid yourself that anyone's hands were tied. Greedy hands were untied by voters These legislators were voted in and still have control of our state education policy. The downward trend will continue until local control is returned and this fundamentalist almost communistic statist centralization is halted. That's why Egyptians are in the streets. They noticed everything being centralized and their democratic choice slipping away.

teese02a
teese02a

@Melissitzifun , the point is that you should not be scoring these tests at all.  These children should be evaluated by their teachers in their schools and all that tax payer money being spent on  corporations getting rich paying you peanuts to do a job you are not qualified for should be going  to educate kids not test them.  Testing is not education.

MrT51
MrT51

Thanks for offering the naked truth and for your honesty. We should not be surprised. Everything that is reduced to "business" is run by a bean-counter responsible for maximized production and profit. "Without vision, the people perish." comes in handy here. Profit is not a vision. It's death of the mind. Steve Jobs and any one of our most brilliant inventors and movers of civilization forward knew this and operated as such. Last year, the voters came perilously close to electing a bean counter to be President. Can you even imagine our spiral downward? People have to read this stuff and get mad. Get furious at what is being done to their children in the name of profit. I won't quote James Joyce when he lamented,"Ireland is an old sow that eats her own farrow." That would be too graphic and disturbing.

Tristan
Tristan

@Jim 
So.... basically, your point is that although politicians  and their business friends are ruining the education of kids in our country for their personal greedt, us whiny educators just need to get over it?  Following the same logic I gues  the next time someone burglarizes your home, breaks into your bank account or beats you up with a baseball bat, you won't complain but will instead "embrace the imperfection of humanity and make a decision to be happy"?

audra.wulirecords
audra.wulirecords

@Jim  Clearly a product of testing as education. Demonstrates low reading comprehension; inability to reason critically; fails to think logically; makes incorrect analogy(ies). Conclusion?


Possibly employed by testing company.

MrT51
MrT51

Sounds like you're stuck with a 14 year old's crush on Ayn Rand. Sorry. Hope you're better soon. In the meantime Google "colleges abandoning standardized test scores". Find articles like - http://www.huffingtonpost.com/2012/11/28/sat-act-not-required-colleges_n_2206391.html

Which states that 800 colleges no longer trust the tests as indicators of college success. Whole nations have thrown them out. So, take a big swig of STFU and chew on a Get Current biscuit. The real sour grapes is Ayn Rand's psychotic hatred of what commie-ism did to her daddy which caused her to hate America and ironically advocate for corporate Stalinism. Go figure. A truly batsh*t crazy bittersh*tter !

MrT51
MrT51

Well-said dude/dudette ! You sound like you'd be a good fighter in this fight. Hope you are able to be there for TGE kids. They are getting squashed in between.

Darryl Thomas
Darryl Thomas

Yes, it's all about the money. Our country's future be damned.

audra.wulirecords
audra.wulirecords

@20autismmom10  And another thing... when did we stop holding back students that can't/don't/won't perform at a level high enough to merit advancing to the next grade? When I was a kid, we all lived in abject dread of not going to the next grade and that was enough motivation for most of us to keep it from happening...


MrT51
MrT51

You couldn't have outlined it better. And there are great models out there. Canada and Finland. The worst Is corporate. Look at corporate/for-profit healthcare. Three times the cost and the worst infant morality rate in the first world. Where is America's inventiveness that we desperately need to look to Europe, Taiwan, Australua for cutting edge education, health and gun policy. Is there brain death in the water? Where did this Corporate Taliban // Capitalist Fundamentalism begin. I know people worshipped Reagan for being the alzheimerist soap salesman for some "revolution". But we've just spiraled downward since his mumblings and Nancy's translations. I'd be confused if I hadn't seen a puppet show as a child.

Accidentalfission
Accidentalfission

Neil Bush's company was (is?) called Ignite IIRC and was in the business of selling NCLB test prep materials. This was a the same time that the Bush administration was paying columnists under the table to write positively about NCLB.

Does anyone remember this Did our "liberal media" ever comment on this?

MrT51
MrT51

There's definitely a surfeit of profit-mongers. Is there a natural predator to solve this infestation?

MrT51
MrT51

You really have a hard time seeing what is right and only being half-sarcastic? Luckily, you or I may not have to do much. Corporate entities ("the smartest guys in the room") are even less intelligent than their commie commissars counterpoints and eventually sef-immolate. I don't know if paying back the devastated teachers and hampered children and dropouts will be as easy as compensating Bernie Madoff's victims but there's a reckoning coming. Missouri has already outlawed For-Profit Charters and when state officials must own up to the fact that colleges no longer use standardized testing because it is unreliable - and they've been r

Transferring taxpayer wealth into these failed companies to do the states children a disservice - - oh, there will be a day of reckoning. See 'em. Woukdnt wanna be 'em.

mazahn0613
mazahn0613

@Jwhyperion Let me get this straight-You want me to listen to your "professional knowledge", but your company does not want to listen to mine. I'm not listening.

mariabronn
mariabronn

@Jwhyperion You don't even know what STANDARDIZED testing is.  It's one-size-fits-all testing.  What teachers do in the classroom is ASSESSMENT, and they tailor it to fit THEIR students and to move THEM forward based on their ability, not some cookie-cutter crap.  That's the part in your little 'book' there where I stopped reading.  If you don't even know the difference, the rest of your post isn't worth my time.  You're welcome for the education in the correct terminology.

RNelson
RNelson

OK, I'm trying to be objective, but let me try to calmly take exception to some of your points.

"...standardized testing has been going on for hundreds of years." I'm not sure that's true, depending on how you define "standardized." Yes, a teacher or school may have decided their own standard, and tested accordingly, but if you mean standards that are set by an outside agency, in our case by the Arkansas Department of Education, then no, I'm sure it hasn't been hundreds of years.

I teach 7th grade, and we are required to give interim assessments throughout the year, to determine if our students are making progress towards a successful standardized test in the spring. Yes, we wrote the rubric, based on the state rubric written by teachers. I have a degree in English and I teach this stuff, and it took me 2-3 years to feel that I had a competent grasp on the rubric. It is denser than it looks. It still takes me 3-10 minutes per essay to evaluate them fairly. But I'm sure that temporary workers are better qualified than I to make these judgements. The thing in the article that disturbed me was the discomfort shown when retired teachers protested. Is it because they are wrong, or just unprofitable?

"...the essay portion of the state assessment does not dictate whether a student passes or fails. A student's overall score is determined by both MC questions and the essay question with the emphasis in scoring on the MC questions." Our state assessment, for literacy, as we are coached and reminded many times throughout the year, takes 67% of it's total score from the essay and open response (writing) portions. And while failing it in 7th grade will not (yet) keep you from passing to 8th, the End of Course literacy in 11th grade is being phased in as a requirement for graduation. I believe it will happen within the next 5 years. So please don't downplay these concerns.

I'm not naive, I understand that businesses exist to generate profit. And it doesn't have to be mutually exclusive of doing good, reputable work. I believe these companies want to do the best job they can - within the confines of the business model. But teachers are also doing their best, and many states are now tying my salary, my job performance, even my very security to the results of these tests. So while my students may not "fail" because Harry the laid-off refrigerator guy does a so-so job, my very job could depend on him. And on his ability to make nuanced decisions about the writing of a 12 year old.

I hope that we can find a way to increase student achievement, measured in multiple ways, for all. One that will work just as hard to reform what isn't best in test scoring as it does to fight what isn't working in teaching. We should all be accountable for changing young lives for the better.

msndis
msndis

"educators have a HUGE say in what they're kids are tested on"

If you've been at an education testing company for five years, I assume you know about the Common Core State Standards. With their national adoption, which is almost complete, there will be NO local/state control over what is on the standardized tests, nor the curriculum students use.

I do realize you need a job and you choose to continue working at the testing company despite the problems but I hope you also realize that the problems with the assessment process can affect students for the rest of their lives. In Washington State, if a scorer doesn't read a 10th grade essay well and gives a non-passing score because he/she didn't wear glasses, is hung over, is on pain meds from a recent surgery, etc., the student won't graduate from high school unless he/she can pass one of the re-take exams or an alternative. So, in my opinion, there is no room for error.

As a parent, I have never trusted the scoring of the state-mandated assessments and opted one daughter out of them for a few years (she had to take them this year as a graduation requirement). This article just reinforces my believe that these assessments have to go.

I have a question I'm hoping a commenter can answer - are the SAT writing tests also scored in scoring mills?

Danthelawyer
Danthelawyer

What "their" kids are tested on; not what "they're" kids are tested on. Of course, one typo does not invalidate this writer's position. But it does throw his or her proofreading skills into question.

Gregor Samsa
Gregor Samsa

Can you at least admit that you're a terrible writer yourself? "It's called tests, quizzes, and essays in the classroom." "Look to the government (of either political persuasion)..." Do you mean look at either political party? Writing effectively is a relatively rare skill even among the most highly educated. Perhaps testing should shift its focus away from writing and focus on skills that students are more likely to develop more robustly (and that scorers are more likely to have a knowledge of).

Patricia Trentacoste
Patricia Trentacoste

You've offered a sincere response to “Inside the multimillion-dollar essay-scoring business.” In fact much of what you say is true, but beside the point, and consequently, no rebuttal. Here's why: You write that readers should ‘consider the sources’ of the accusation but this is a fallacious dodge frankly, because the qualifications of a critic are irrelevant to the substance of the criticism. Either the claims made by the article are true or false. Which claims in this article are false? You don't say. Instead you offer several asides which miss the point. For instance, you argue that the testing industry itself should have a say in an independent critique of its role in society. But clearly an internal audit is not appropriate for determining the impact of ST (standardized testing) on external parties with no such input, nor of the unethical practices within the industry (e.g., essay scoring mills). The burden of proof for ST advocates is to empirically show that the ST industry, including the essay scoring mills, are in fact improving the quality of education in America and promoting the flourishing of youth without using arbitrary or otherwise harmful methods or unnecessarily destroying a system that works when properly funded. What say you to the complaint that the real beneficiaries of the industry have turned out to be those who have profited from selling both the testing instruments and the remedial and tutorial tools made necessary by those very same testing instruments?

You claim 'standardized testing' has been going on for centuries. But you are clearly confusing testing constructed by a particular teacher of what s/he taught in his/her own classroom, with testing constructed by ST test writers (some of whom are teachers, some not) concerning subject matter that should have or could have been taught in a generic classroom. Your comments miss the central point that a test prepared by someone who's never met the child being tested, never sat in his or her desk—often elbow to elbow with too many classmates, never tasted the school lunch fare, never rode on that child's bus, never met that child's actual teacher, or understood his/her home life, is not qualified to assess that child's learning capacity and achievement. Worse, defense of ST practices as a means of withholding funding from innocent children treats equal educational opportunity as a privilege, not as a social good and a social right in a free civilized nation.

Ultimately, defense of ST's association with NCLB implies that what Mary scores on her test suffices to deprive Tommy of percapita educational funding. How can that be? NCLB and ST are designed in principle as assessment tools for abstractly quantifying a concrete qualitative experience—education. The task is impossible, yet the model is being foisted on the country via propaganda and vetted politicians as though it is actually working to uplift the social good. There is only one solution: abolish both NCLB and ST; do not prop them up with limp reforms, no matter how well intended. Unless of course, the agenda really is to destroy the social good for private profit… Patricia Trentacoste Full time adjunct philosophy professor

L Abdo
L Abdo

The problem is all politicians, not just one party or another. Who helped that law get passed? Why is President Obama proposing a law that has many similarities? Complaining about a political party doesn't change anything. Reform of the entire political system and educational system are needed and people pointing fingers are delaying the process, hence the fiasco that is Washington, D.C.

ZoeBrain
ZoeBrain

@Jim Herndonern  No, a 1 due to incomplete sentences and ellipses. That's according to the rubric.
"Eating sour grapes" is not the correct use of the phrase. Acronyms are not explained.

Leharper
Leharper

No. I was a scorer for ETS. They are also graded on a rubric and there is certainly pressure to conform for the purposes of standardization. (This looks like a 4, but I think it is a 5, but I think everyone else is going to give it a 4 so I'll give it a 4.) Standardized tests are simply not good tools to measure divergent thing. Not on the multiple choice section, not in the essay section. That being said, I felt the ETS strove to be ethical.

buster13
buster13

@Patricia Trentacoste  4

cewagner
cewagner

Patricia Trentacoste,

Thank you.  I appreciated what you wrote.


MrT51
MrT51

"YOU LIE!" to quote the only political party disreputable enough to shame us in front of the rest of the world with sham for-profit education, healthcare, and foreign policy. Your "it's not just one party" whine is just embarrassing. Republicans are the party of Corporate Stalinism. Their for- profit ponzi schemes have driven us into the ground on all levels and their solution is more privatization and deregulation. The definition if insanity is doing the same thing over and over again expecting a different result. Which party is most insanely set on the "free" market?

epic60sman
epic60sman

@MrT51 "Complaining about a political party doesn't change anything" is the truth.  The role of the political parties in this country is to convince the people that there is some semblance of a Democratic process going on here,  It is all a big cover-up for the fact that real Democracy is extremely difficult to put into effect, as the Wall Street Occupiers found out.  The major force behind this Education Deform movement has come from organizations like the World Bank, the International Monetary Fund, and particularly the OECD.  OECD, for example, has been instrumental in installing and monitoring this program in 20 to 30 or more countries that owe money for development grants to the World Bank or the other major financial institutions that loans money to the these developing nations.  The Arab spring has made them all tremendously nervous about spreading education and the democratic anthem around.  'Lets keep them dumb' is the theme all these 'New World Order' enthusiasts are singing.  
BTW, you can, with veracity, blame the Republicans for opening the Public School doors to privatization of education but, in my state (Arizona) and I assume many others, the Tea Party Republicans are the only ones standing up to this federal-Industry takeover of education and the Common Core Standards and testing. 
 

 
Loading...