The robo-graders

[pendell]

seen here (http://www.scienceworldreport.com/articles/2368/20120330/robo-readers-new-teachers-helper-u-s.htm)



merican high school students are terrible writers, and one education reform group thinks it has an answer: robots.

Or, more accurately, robo-readers - computers programmed to scan student essays and spit out a grade.

The theory is that teachers would assign more writing if they didn't have to read it. And the more writing students do, the better at it they'll become - even if the primary audience for their prose is a string of algorithms.


Can't agree at all. Essay-writing is first and foremost a means for humans to communicate to each other. To my mind, it's reprehensible to assign essay writing to students that you have no intention of reading. What's to stop the kids from turning on a robot writer to turn in a perfect grade to your robot reader? After all, you've set the example by assigning work you have no intention of doing anything with, so why shouldn't they pass the work on themselves?

Respectfully,

Brian P.

[Elemental]

I can understand grading large scale standardised tests through a machine, particularly if they're the ones where you colour in boxes with 2B pencils.

But essays? That just won't work.
It's easy enough to write something to essay format, but it's the way it's written and what is written that is important.

Only a human can truly understand the writings of another and grade them adequately.


Edit: Which reminds me of a relevant story...
Every year, seniors in Queensland have to sit the QCS, which has a significant written component.
And as such, they get lots of teachers together to mark them all (bearing in mind that there are thousands to mark).

When we were preparing for it, Mrs. Weston, the teacher who was helping my group through the process, told me that the markers would often spend their lunch break telling each other about the really good things they had read.
It's part of the teaching process for the teacher to become proud of their students' results. This takes part of that away from them.
(Because, believe me, teachers are not in it for money)

[Manga Shoggoth]

The theory is that teachers would assign more writing if they didn't have to read it. And the more writing students do, the better at it they'll become - even if the primary audience for their prose is a string of algorithms.

Whut?

Without meaningful feedback from the teacher (a.k.a. marking the work properly) they will not improve, as reams of fanfiction throughout history has proved.

[Jack Squat]

We actually had something like this back when I was in 8th grade. You typed up an essay in a browser window, submitted it, and it's spit you out a score between 1 and 6. For the most part it just looked at word count, spelling, size of paragraphs, and repetitions of words.

It went over about as well as you expect - grades tended to be on the lower side due to repeating words (it counted instances of small words like "is" and "the", which are really hard to sub out), and after 3 tries, the scores were either fairly consistent or falling; I don't think anyone actually improved.

Of course, as bad as it was, I still got better feedback than when I was in 3rd grade and the teacher just threw out our 3 page papers as soon as we handed them in.

[Kneenibble]

seen here (http://www.scienceworldreport.com/articles/2368/20120330/robo-readers-new-teachers-helper-u-s.htm)



Can't agree at all. Essay-writing is first and foremost a means for humans to communicate to each other. To my mind, it's reprehensible to assign essay writing to students that you have no intention of reading. What's to stop the kids from turning on a robot writer to turn in a perfect grade to your robot reader? After all, you've set the example by assigning work you have no intention of doing anything with, so why shouldn't they pass the work on themselves?

Respectfully,

Brian P.

The quote you selected misrepresents the tone of the article completely, which is quite tongue-in-cheek and humourous and offers both pros and cons to the idea.

My objection to such a device is more that it would likely discourage creativity as much as it would discourage sloppiness.

[warty goblin]

They've been doing this for major standardized tests like the GRE for quite some time now.

Honestly, if their language heuristics are good enough, I can't really see the problem. Particularly for the sort of brutally dull writing standardized tests encourage, it's not like anybody's gonna do anything daring and creative there anyways. The difference between being evaluated by underpaid grad students and a machine is...well, I'm not gonna get upset about it.

This is clearly terrible for a longer, more research oriented, paper where a deep knowledge of the background material and scholarship is necessary*. But even for the sort of braindead response paper a lot of classes require, I find it quite believable a computer could grade it as well as a person.

*For now. Given development time, and massive computer power, I'd bet this problem can be done via a statistical analysis as well.

[Cyrano]

But Williamson argues that automated graders aren't meant to identify the next James Joyce. They don't judge artistic merit; they measure how effectively a writer communicates basic ideas.

Yeah, as someone who really likes writing, I completely disagree with this. Maybe we're currently at the state where we can make a computer that effectively grades how well one regurgitates basic ideas - I'm not really clear on how good we are at such things. But communication of an idea is not the same as regurgitating it, at all, any more than a beautiful, effective metaphor is as good as a lazily used cliche.


"The reality is, humans are not very good at doing [essays]," said Steve Graham, a Vanderbilt University professor who has researched essay grading techniques. "It's inevitable," he said, that robo-graders will soon take over.

That confuses me. How is a computer going to be better than a human at determining how effective something is at communicating an idea to a human[/]? I mean, yeah, it sucks that humans are totally fallible, and that they will subconciously mark essays on the neatness of the handwriting, and that one human being will consider one turn of phrase more effective than the next human being will, but those aren't problems that are solved by removing humans from the equation. Well, technically, they [i]are, they're just replaced by new and bigger problems - like exactly what is an essay being marked for? Grammatical accuracy? Mentioning lists of relevant words in sequence? Certainly not the effectiveness of communication, because by definition computer graders are ignoring that. Essays and grammar exercises are different things.

It's an interesting article and I worry I take it too seriously, but it's scary to me. Reading this is kind of like telling a musician that computer-generated arpeggios are going to replace human composed music because arpeggios tend to be an effective compositional technique. No matter how tongue-in-cheek it may be it's still fwightening.

[Whiffet]

This is a bad idea for several reasons. One is that the teacher needs to provide more feedback than simply, "You need to work on your grammar and sentence structure." Some of my peers in high school had horrible punctuation, but other problems were more widespread. If they were given newspaper articles and told to, say, write an essay comparing and contrasting them to something discussed in class, they would often just summarize the articles. My teachers had to work hard to get everyone to actually answer the questions. From what I understand, the robo-readers wouldn't be able to tell if someone failed to meet the demands of the question.

Another reason is this:

A prankster could outwit many scoring programs by jumbling key phrases in a nonsensical order. An essay about Christopher Columbus might ramble on about Queen Isabella sailing with 1492 soldiers to the Island of Ferdinand -- and still be rated as solidly on topic, Shermis said.
You know students would figure out the tricks to easy grades.

[Howler Dagger]

As a student who has used these programs, I can say I find them hilarious. We were doing an essay where the prompt was "Write about a time you did something you didn't think you could do." One of my friends got a 3 (out of four), and then added a string of randoms phrases and gibberish and got a better grade (a 4). I got a three, but my brother discovered that if you put your essay in a persuasive (as opposed to the narrative we were doing) about asking congress to make a new holiday, you could get it graded (as opposed to "ungradable", which amusingly happened to me on an essay where I would like to say it should have been). I had a three on the original prompt, but a four on the new prompt!

Also, the program I used DID give atual feedback, but it amounted to telling you what concepts (such as syntax, idea development) you did good at and what you did bad at.

[Anxe]

I'd have no problem with it if it was as good as a the thorough edits my father gives my writing. However, once it gets there isn't it a conversational level AI effectively?

[Teddy]

As a computer science student, I'd first of all like to point out that what the article actually revolves around is a competition to make as good auto-graders as possible. Since no illegal activities are involved, I see no problem with this. My standpoint regarding the use of auto-graders to grade actual writing assignments mostly boils down to the phrase "that depends", however.

This is pretty advanced computing we're talking about, so it's still in its cradle, but that's not to say that it is and will forever be impossible. After all, the human brain is not much more than a very advanced computer (I'd like to see a neural network (a type of computer mimicing the architecture of the brain) of 100 billion processors, which is roughly equal to the number of neurons in the human brain. That would be glorious).

To me, the best part of that article, however, is the first (and only) reader comment. Wut? That guy, I can't quite tell if he's serious or not (but I lean toward the latter. If it seems too absurd to be true, then it probably is too absurd to be true as well), but it's hillarious nevertheless.