AI in assessment: opportunities and limits

Students filling in written exam papers at their desks in a classroom

Assessment is one of the things teachers spend the most time on and talk about the least. Writing questions, building answer keys, marking papers and writing meaningful feedback for every student takes hours. That is why the idea that AI tools could lighten this load is so appealing. And at some stages they really do make things easier. But assessment is where we make decisions about students, and those decisions have consequences. So we need to talk about the limits as much as the opportunities.

The easiest place to use AI is drafting questions. Draft questions at different cognitive levels for a given learning objective, plausible distractors for multiple-choice items, possible student answers to an open-ended question or scenario ideas for a performance task are all things these tools can produce quickly. Writing distractors in particular is hard even for experienced teachers. Some of the distractors a tool suggests will be weak, but a few can be good options that capture students’ real misconceptions.

Still, you need to look at each of these drafts with an assessment expert’s eye. AI-generated questions have some recurring problems. The correct option is noticeably longer than the others, more than one option is correct, the stem gives the answer away, or the question tests recall rather than the objective. Then there is the plain problem of a wrong answer key. Especially in multi-step maths and science questions, do not accept the tool’s answer without solving the problem yourself.

Content validity is also your responsibility. An exam needs to represent the objectives it aims to measure in a balanced way. AI can give you questions that each look fine on their own, but planning which objective they measure, and with what weight, once they come together is your job. Preparing a table of specifications first and then asking AI for draft questions for specific cells is a healthy way to work.

Another useful area is producing variations of existing questions. Parallel items that test the same objective with different numbers or contexts can help when students need to retake an exam or when you teach several sections. Simplifying the wording of a question stem for students with reading difficulties, without changing the skill being measured, is another job these tools can help with. The thing to check is the same here too: whether the simplified or altered question still measures the same thing at the same level of difficulty.

AI in feedback

In my view, the most promising use of AI in assessment is not grading but formative feedback. When a student hands in a written draft, a problem solution or a project report, offering suggestions for improvement before giving a grade is one of the forms of assessment that contributes most to learning. In crowded classrooms, though, writing detailed feedback for every student is often not possible.

This is where AI can work as an intermediate layer. For example, based on a rubric you prepared, it can produce a draft listing the strengths of a student’s text and the points that could be improved. You read the draft, correct it, add your own observations and pass it on to the student. Your feedback time shrinks, and the final word is still yours.

You might also consider letting students get feedback from AI directly on their own drafts. In that case it is important to decide in advance what kind of feedback the tool will give. A tool that rewrites the text for the student does not help learning. Feedback that asks questions and points out gaps while leaving the solution to the student is far more valuable. Students also need to learn to question this feedback, because the tool can sometimes flag something correct as wrong.

Why grading is different

Having AI directly score open-ended answers or assignments is something to approach with much more caution. There are several reasons for this.

The first is consistency. If you give the same text to the same tool at different times, you may get different scores. A small change in your prompt can also change the scores. What we call reliability in assessment is exactly about this consistency, and trusting a tool without testing it in your own classroom is not the right thing to do.

The second is bias. These tools carry the patterns of the texts they were trained on. They may reward a particular writing style, vocabulary or structure and penalise a student who expresses the same idea in different language. Students who write in a regional variety rather than the standard language, who are learning the language of instruction as a second language, or who have learning difficulties are particularly at risk here.

The third is superficiality. These tools easily pick up a text’s fluency, organisation and length. They can be much weaker, however, at judging whether an argument really holds together, whether a solution is creative, or how far a student has come since last month. A long, tidy but empty answer can score high while a short but deep one goes unnoticed.

The fourth is accountability. When a student or a parent challenges a grade, “the AI gave it” is not an acceptable explanation. A grade is the teacher’s professional judgement, and the reasoning behind that judgement should be explainable. Even if you use AI in scoring, treat it as a preliminary assessment only and make the final decision for each student yourself.

There is also a data privacy side to this. Student papers contain personal data. Before uploading anything to an AI tool that reveals students’ names, numbers or identities, find out what your institution’s rules are. If the rules are not clear, at the very least work with anonymised texts.

AI is also changing assessment itself. We can no longer be sure that an essay or research assignment written at home reflects the student’s independent performance. Trying to solve this with AI detection tools may seem tempting, but the reliability of these tools is highly questionable and the harm to a student who is wrongly accused is serious. My suggestion is to redesign assessment rather than try to detect. In-class writing, process portfolios, oral presentations, short conversations in which students explain their work and assignments that show the development from draft to final product are more resilient to AI and richer in terms of learning.

AI in assessment is valuable to the extent that it frees teachers from tasks like drafting and first-round feedback so they can spend that time actually getting to know their students. The moment we hand decisions about a student over to a tool, we move away from the most basic principles of assessment.

See you soon.

Image: Narek75, CC BY-SA 4.0, Wikimedia Commons.