The traditional assessment has had a good run, but it has outlived its usefulness, and it is time to focus on more modern ways of assessing skills that will lead to success on the 21st century work floor.
That is why many companies, such as Vodafone, Randstad & KLM have turned to using a more innovative approach to candidate evaluation – game-based assessments.
We turned to an expert in the field of neuroscience and game-based assessments, and a member of our advisory board, Dr. Marcia Goddard, to shed light on the topic and provide answers to the most common questions we’ve received when it comes to using game-based assessments in recruitment.
Can you explain the concept of game-based assessments and how they differ from traditional assessments?
Marcia: “Gamification means you add game mechanics, and game elements, to non-game situations. A traditional assessment is usually either just a bunch of boring questions about your personality, or it feels like you’re retaking your high school exams. Neither of those options is particularly engaging. Gamifying this experience basically means you change the rules. It’s no longer about answering a question or completing a test, it’s about reaching a goal or fulfilling a mission.
You turn the assessment experience into a game. “
What are the advantages and disadvantages of using game-based assessments compared to traditional assessments?
Marcia: “The association between traditional assessments and work floor performance is generally quite low. Not non-existent, but low. This means that the average assessment doesn’t do a very good job of predicting how well somebody will perform once you hire them for a specific role.
This has a few reasons.
- The first is that in general, people don’t really know what they’re looking for when they want to hire someone. I’ve seen organisations use digit span tests for sales roles. The digit span test measures working memory. Obviously, having a strong working memory is a good thing. But is it really a crucial skill to have as an account manager? I’m not so sure about that. So my first problem with assessments is that they aren’t used the right way.
- And secondly, as I mentioned, traditional assessments are boring, and therefore demotivating. When something is demotivating, people are not going to show you their top performance, which is what you want from an assessment. You want to see people’s potential.
This is why I believe in gamification. We know from years and years of research that using the right game mechanics can substantially enhance motivation because it triggers our brain’s dopamine system.
And we also know that games are engaging. The more engaged you are in an activity, the more likely you are to show your true, natural behaviours. We all know what happens during a game of Monopoly or Risk, right? People will show their true colours, which is why it sometimes ends up with the whole board game flying across the room…
Those games use the same game mechanics that are also used in game-based assessments. And since we want people to show their natural behaviour, as that is the most truthful representation of who they are, I believe gamification is the best way to conduct assessments.”
In terms of validity, how do game-based assessments compare to traditional assessments? Are game-based assessments as reliable or even more reliable and accurate in measuring candidates' abilities?
Marcia: “This is a tricky question for me to answer, because I have a fundamental issue with the concept of reliability and validity. Generally speaking we say a test is valid or reliable when repeated measurement in various groups produces the same results. High test-retest correlation, etc. However, the world is changing, and so are the people in it.
It takes, on average, about 5 years to officially validate a test. This means that scientific validation always lags behind changes in the real world. On top of that, once a test has been deemed valid and reliable, it keeps that label forever. This can be problematic. For example, we started using personality tests to make hiring decisions some time in the 1980s. We can have a whole discussion about whether or not that was a good idea to begin with, but the fact is the world today is nothing like the world was back then. However, tests that were deemed ‘valid’ back then, continue to keep that label.
So to answer this question: I believe game-based assessments are more reliable and accurate in measuring candidates’ abilities because, strictly looking at it from a holistic perspective, they come closer to real world 21st-century behaviour than traditional assessments. But there isn’t that much data yet to back that up.
This makes it particularly important for employers to ask assessment providers critical questions about their validation process. If they can’t explain it to you in a way that you understand, then there probably isn’t a very strong scientific basis for their assessment.
For example, every single one of Equalture’s game-based assessments goes through a rigorous validation process. Every assessment is based on an already existing, validated test of the skill or behaviour that is being assessed in the game. For example, the ferry game is based on the tower of London test and the racer game is based on the Wisconsin card sorting test.
These validated tests and assessments have been used in many different contexts for many years. This means that these tests were found to be effective in predicting an individual’s skill level on the skill being tested. Only if and when the correlation between Equalture’s assessment game and the original assessment is sufficiently high, will the game be added to their assessment portfolio. In other words: if a participant scores high on Equalture’s assessment game, they will also score high on the original test. Their continuous validation process ensures that Equalture’s assessment games are predictive of a participant’s true level of the skill being tested.
Most importantly, all of Equalture’s validation studies are available upon request. This level of openness helps potential clients make informed decisions about whether their assessment games are the right fit for them.”
Can you provide examples of industries or job roles where game-based assessments are a better fit than traditional assessments?
Marcia: “The world is changing rapidly, and that rate of change doesn’t show any signs of stopping. The work floor needs different skills and more diversity, which also leads to people who don’t fit into the norm set by these traditional assessments. From my perspective, the traditional assessment has had a good run, but has outlived its usefulness.
For example, for the longest time, conscientiousness was seen as an essential personality trait for ‘good employees’. Conscientious people were deemed dutiful, loyal, disciplined and organized. From an Industrial Revolution perspective, this was exactly what was needed in the workplace. However, work no longer looks the way it did when we were in the Industrial Revolution. We’ve entered the Digital Revolution, which is characterized by constant and continuous change. The way we work has, and will continue to change over time, with the rate of change seemingly increasing year by year.
This means you don’t just need people who are disciplined, people who think about all the consequences of a decision before they make it. You also need mavericks. People who are a bit more impulsive. People who take risks, and try new things. These traits are antithetical to the concept of conscientiousness, which means we need to broaden our perspective and look for different types of traits and skill sets.
Game-based assessments that are well thought-out, and measure the skills we need on the 21st-century workfloor, are the future (or the present, as far as I’m concerned).
In addition to that, many traditional assessments, particularly those in questionnaire format, are incredibly culturally biased. We live in a global society, and organizations are hiring people from all over the world. Traditional tests do not take into account differences in cultural perspectives, and may therefore disadvantage people from different (non-Western) countries.”
Are there any specific cognitive or psychological factors that game-based assessments can capture more effectively than traditional assessments?
Marcia: “All of them. Just think about it this way: what’s more effective? Having someone respond to the statement ‘I enjoy solving complex problems’ on a scale of 1 to 5, or giving them a complex problem to solve and assessing how they approach it?
And what is more likely to bring out someone’s best performance (which, again, is what we’re looking for when we use an assessment): keeping someone behind their laptop for 2 hours, completing mundane tasks like ‘how many of these numbers can you remember’, and ‘what should the next number in this sequence be’, or letting them go on an adventure in the Wacky World of King Abacus, where their goal is to find the Treasure of Mathematica? Just a random, not-all-thought-through example that I came up with in 30 seconds. Now imagine how much more engaging a regular test can be if people who actually know what they’re doing (i.e. game designers) spend time thinking about it!”
Is there a correlation between the age or novelty of an assessment method and its validity? In other words, does being an older or newer assessment method automatically determine its effectiveness or reliability?
Marcia: “No it doesn’t. However, the older the assessment, the more time people will have had to determine its validity and effectiveness. This is why I said that it requires a bit of a leap of faith to start using newer types of assessments. The evidence for game-based assessment is strong, but there currently is less data available than for traditional assessments. However, it is important to keep in mind that many of the older, traditional assessments were validated when the world looked a whole lot different than it does today. So that raises the question of how valid these assessments are today.
Regardless of how rigorous and technically-correct the validation process was when these tests were validated, the world has changed significantly. And employees work in the real world, not in an imagined scientific reality in which all variables remain stable. This is why I believe traditional assessments are past their expiration date, and it is time to focus on more modern ways of assessing skills that will lead to success on the 21st century work floor. This is where high quality game-based assessments take the cake. They are more engaging, more candidate-friendly, less biased, and less susceptible to social desirability bias than traditional assessments.”
What are the differences between various game-based assessments out there in terms of reliability, and scientific validation? What should you be mindful of when selecting a game-based assessment?
Marcia: “Honestly speaking, I’m not sure what differences there are, and that is mainly because most assessment providers hardly share anything about their validation process. If I were a client, that would be a massive red flag for me. I would want to know what the theoretical and empirical foundations of your assessment are, otherwise, I wouldn’t use them. I’m not saying they need to be officially and fully validated, as I mentioned above I don’t believe that is necessarily the best way to assess if an assessment will help you. However, I do want to know how the test was developed.
I would ask the provider what the theoretical framework behind the assessment is, and whether that is grounded in actual science (and not just ‘years of corporate experience’, for example).
I would also want to know what the validation process looks like. With the world continuously changing, is this validation process a one-off? Or does the provider continuously test its assessments within different populations (like Equalture does), to make sure the assessments are inclusive and unbiased?
And lastly, I would want them to be able to explain to me in very clear terms how the scoring system was developed. From the perspective of diversity, lower scores might sometimes be preferable. Is this something they keep in mind within their reporting, and what is the science behind the scoring system?”
The new standard in hiring
