How Can a Computer Grade Essays?

I was recently out of town on business in a city that I have never been to before. When I landed I grabbed my bags, headed to the rental car location and picked up my car. The attendant handed me a map and asked me if I needed directions anywhere. I declined and then, without even glancing at the map in my hand, I turned on my GPS, plugged in the address and let it guide me to my hotel.

As I listened to the turn-by-turn directions, I realized there was a time when I would have had to converse with another human being to get to where I needed to go. Now, it’s natural for me to decline any help from humans because I know I can rely on technology.

I am amazed by the leaps and bounds technology has taken in the past couple decades. Given how much technology has advanced for consumer products, like my GPS, I am equally as impressed by the advancements I have seen in regards to the products I work with. One in particular has stood out to me this year: IntelliMetric®.

Basically, IntelliMetric is an advanced artificial intelligence system that can read, evaluate and score written essays to determine a person’s writing skills.

Pretty cool, right? But you might be thinking, as I did when I first learned about it, “How can a computer possibly read and evaluate writing as well as a human being?”

Well, Vantage Learning (the creator of Intellimetric) does their homework to train the software to mimic expert human scorers. Here’s how they do it:

  1. Hundreds of essays are gathered on a specific topic.
  2. A rubric is developed and expert human scorers evaluate each essay based on the criteria.
  3. These scored essays are then used to train the software. IntelliMetric captures the pooled wisdom of many expert scorers and creates a unique solution for each stimulus or prompt, a process which is conceptually similar to prompt-specific training for human scorers.
  4. Once the training is complete, the system is tested with a fresh set of essays to validate the accuracy of the scoring. These essays are scored separately by IntelliMetric and by the experts.
  5. The results are analyzed to ensure the artificial intelligence engine is providing similar scores compared to the expert raters.

This painstaking process is done for every single writing prompt, allowing Intellimetric to be calibrated as needed for each individual question until it agrees with or exceeds expert human scoring. In fact, on average, IntelliMetric scoring comes within one point of expert raters over 97% of the time! Pretty impressive.

Automated Essay Scoring uses Intellimetric scoring, and we have received tremendous feedback regarding the ease of use and value of the information our clients are provided with. For educators, it has been extremely popular for admissions essays and classroom activities. Employers love it when they need to find out if job candidates have professional communication skills.

What do you think about using artificial intelligence to evaluate someone’s writing ability?

Add new comment

X