How can you tell whether an educational game works?
After using an educational game in class, the first impression most teachers walk away with is how the students reacted. If they got excited, did not want to stop and asked “can we play again?” at the end of the lesson, it is natural to think the game was a success. But those reactions tell us how much the game was liked, not how much it taught. In this post I want to talk about the signs you can look at to find out whether an educational game works.
First of all, you need to be clear about what “works” means. Did you use the game to help students learn a concept, to reinforce something they had already learned, or to raise their interest in the subject? Each of these calls for a different kind of evaluation. If the learning goal is vague, measuring the game’s success is impossible too. If you have a clear answer to “what should students be able to do after playing?”, you have already done half of the evaluation.
Fun and learning are not the same thing
Students’ engagement with a game is an important precondition for learning. It is hard to expect a student who is not engaged to learn anything from a game. But engagement is not learning itself. A student can happily play a game for hours and learn almost nothing about the target concept. With visually impressive games in particular, the excitement often comes from the novelty of the experience rather than from the content. With technologies such as virtual reality this novelty effect is even stronger. The excitement students show when they put on a VR headset for the first time comes from the headset itself.
In-game scores can be misleading as well. A high score does not always show that a student has learned the concept. Students are very good at finding a game’s loopholes. A student who finds the right answer by trying each option in turn, keeps pressing the hint button, or spots a pattern in the game and plays to it can move forward without understanding the concept. When you look at in-game scores, you also need to see how the student got them.
That is why I suggest checking learning outside the game too. Before playing, look at what students already know about the topic with a short activity, and repeat a similar activity after the game. The important thing is that the activity after the game should not be identical to the tasks inside it. If the student compared fractions using pizza slices in the game, can they compare fractions in a different context afterwards, for example with a measuring cup or on a number line? If they can carry the knowledge outside the game, we can speak of real learning.
Do not forget retention either. An assessment straight after the game shows what the student remembers at that moment. A short review activity a few weeks later tells you much more about how lasting the learning is.
Do not decide without a comparison
A question that is often skipped when evaluating educational games is this one: is the game better than what could be done with another method in the same amount of time? Students knowing the topic better after the game seems to show that the game is effective, but another activity done in the same time might have produced a similar result.
We saw this up close in our study of Keşfet Kurtul. In a quasi-experimental study on fractions with fourth graders, the academic achievement of students who played the game increased. However, a similar increase was seen in the comparison group, which used the method applied at the school, consisting mainly of mobile apps developed for fractions. In terms of achievement and engagement, the two methods had similar effects. The difference appeared in the social dimension of student engagement, where the game was more effective. Without a comparison group, it would have been very easy to credit the whole gain in achievement to the game.
You certainly do not need to run an experimental study in your classroom. But asking “what could I have done instead of the game, and how long would it have taken?” when evaluating one helps you make a more realistic decision. If setting up, explaining and playing a game takes most of a lesson, the game needs to offer something worth that time. On the other hand, if the game achieves the same learning with more students taking part or in a more positive classroom atmosphere, that is an important gain in its own right.
Watching students while they play gives you information that numbers cannot. When you observe, focus on what students do rather than on how much fun they are having. Are they thinking about the topic when they make decisions in the game, or just trying things at random? When they make a mistake, do they try to understand why? When they talk to each other, do they use the name of the concept? What students say to one another during play is a very valuable clue to what the game is making them think about.
It also helps to pick a few students, stand next to them and ask them to think aloud and explain what they are doing. A simple question like “why did you choose this door?” quickly reveals whether a student is using the concept or just guessing.
A short debrief after the game is an opportunity for both learning and evaluation. Ask students what they did in the game, where they struggled and how the game relates to the topic. If they only talk about the story and the characters and never mention the topic, you may conclude that the game did not deliver the learning you expected. If students can connect what happened in the game to the concept, that is a strong sign the game worked.
Keep in mind, too, that a game may not work the same way for every student. Students with more gaming experience can figure out the interface quickly and focus on the content, while less experienced students may spend most of their energy working out how to play. Rather than looking at the class average, looking at how different groups of students were affected gives you a clearer idea of who the game suits and under which conditions.
One practical suggestion is to write yourself a short note every time you use a game. Write a few sentences about which class you used it with, how long it took, how you prepared, where students got stuck and what you saw in the follow-up activity. When you use the same game next term or with another class, these notes will show you much more clearly how to position it. Over time you build your own small archive of evidence about which game really works for which learning outcome.
See you soon.
Image: Aigner Ronja, Kohlmeier Michelle, CC0, Wikimedia Commons.