Are We Ready for Robots to Grade? (Chronicle)

Are We Ready for Robots to Grade? (Chronicle)

September 15, 2014

By Janet C. Walkow and Erin Reilly

Creating an ideal online-learning environment requires time and technology that can be challenging or beyond the reach of many faculty members. When we set out to create a MOOC for the College of Pharmacy at the University of Texas at Austin, we envisioned a unique learning environment that would challenge a global audience to think deeply about pharmaceutical issues. We pictured 20,000 essay-waving students taking our massive open online course. But wait: How could we grade that many open-response questions and evaluate that many students? Could automated graders fill the gap?

We recently tested this approach and were surprised by how closely the results from automated graders matched those from human ones. While automated grading wasn’t perfect, we believe it will be useful in the future, and we plan to use our experience to improve the process and reliability of results.

Our course, “Take Your Medicine,” launched in the fall of 2013. We had begun developing it nearly a year before, just after our university joined the nonprofit online-learning platform edX. At that time, one of us (Janet) was collaborating with Donna Kidwell, an online and gaming expert who brought her skills in e-learning pedagogy to the course. Donna’s excitement over the call for MOOC proposals spread quickly to Janet, whose reaction was, “Great! But what’s a MOOC?”

Much has changed in the MOOC world since we began planning our course, which explores how research innovations are developed into therapeutic medicines, as well as how to be a savvy consumer and patient. The course was designed to attract students from a broad range of backgrounds and interests, incorporating bite-sized video clips to engage them and assessment tools to gauge their progress. While students could not earn credit, they could get a certificate stating they had passed the course, which included an evaluation module.

We knew students’ engagement needed to be a key component of that. But we had one major question: How could we measure what they actually learned from the multiple online resources embedded in the course, like videos, animations, and lectures? Like most instructors, we had a strong desire to move beyond the traditional multiple-choice format and involve students in at least one written and graded assignment. We wanted them to engage with the course content in a way that was personal, challenging, and deeper than answering standard true-false questions.

Leave a Reply