Why you need a wide cast of experts to create a successful serious game (Serious Play Conference recap)
I would like to attribute most of my geographical knowledge to the game “Where in The World is Carmen Sandiego?” For those of you that have never heard of this educational gem, it’s a PC-based game that has you try to track a thief across the globe, where you learn about geography, history, and world cultures along the way. Not only was I hooked to this game as a child, but it helped me learn topics that I wasn’t intrinsically interested in. I’d like to thank Carmen Sandiego for helping me survive my torturous middle school geography classes.
Games like Where in The World is Carmen Sandiego? and gamification instructional strategies like leader boards or points systems have been used in K-12 educational curriculums for a while now. Gamification is making its way in other domains and is now revolutionizing learning in the private sector, healthcare and the military.
Last week I participated in the Serious Play Conference, and we discussed the impressive spread of gamification and serious games in education and training. The conference brought together experts (game designers, artists, engineers, educators, and researchers) to present and talk about the creation and use of games and simulations throughout various domains.
Throughout the sessions, I came across
These demonstrations provoked discussions about the purpose of virtual reality (VR), augmented reality (AR), and gamification in training and education. Specifically, one presenter brought up that companies often use these terms as buzzwords while not addressing the core reasons to use these platforms and methods.
Our team at QIC believes that implementing any new technology must depend on its effectiveness as a training tool and the cost of implementation. QIC is currently resolving this concern by developing a tool that guides comparative evaluations of training efficiency across training platforms (e.g., AR, VR, mobile).
I sat in several discussions that provided methods to game design. One session consisted of a workshop to develop games using Universal Design for Learning (UDL). UDL is a framework that focuses on inclusivity in learning and offers guidelines to ensure that all individuals have equal learning opportunities. These guidelines focus on
In other sessions we discussed how psychological concepts can be integrated into a game-based education and training to make it more memorable and engaging. For example, one presenter talked about the role of operant conditioning - learning that occurs through rewards and punishments for behavior- in fostering engagement and motivation.
Conferences like Serious Play are so important because they allow us to see what goes into developing a serious game through the eyes of experts in different fields. Industries often stop at integrating rewards and score boards into education and training, but that seriously limits their growth. The complexity of designing and developing serious games in education and training requires the teamwork of a wide cast of specialists.
Do you have a favorite serious game? If so, what made you enjoy it? We would love to know!
Who doesn't love a snide new way of insulting people? Enter the Dunning-Kruger effect. It's experiencing quite a lot of attention in the media these days as a means of saying "you're dumb AND you're too dumb to recognize how dumb you are." And hey, I totally get it. Insults are fun, and when you can insult people with science, it's even better. But there are some systematic ways that this effect is being slightly misinterpreted across social media these days. I'm going to clear them up for you because if you go on acting like you're an expert about this phenomenon when you actually aren't, well, guess what you're exhibiting?
There are two parts to this that I'm going to talk about:
First, let's tackle what the Dunning-Kruger effect is.
Dunning and Kruger (1999) found across a variety of tasks, when asked to evaluate their own performance, low performers (those in the bottom quartile) tended to think they performed a bit above average (around the 60th percentile) - see Figure 1. Those in the second and third quartile also overestimated their abilities, but the better people performed, the less 'off' their judgments were. Those in the top percentile actually thought they performed a teensy bit worse than they actually did. The pattern of the Dunning-Kruger effect has been replicated many times over and across many types of tasks.
The part of this that grabs attention is that low performers rate themselves as shockingly better than they really are. Note, however, that while the lowest performers rated themselves as above average, they did not rate themselves as "the best" - they rated themselves as "a bit above average." The people who rated themselves as the best were, in fact, the best: the top performing quartile also had the highest perceived ability. This runs contrary to the popular understanding of the effect, in which the most incompetent people think they are the best. They don't think they're the best, they just don't think they're the worst.
To sum up Part 1, the Dunning-Kruger effect:
Now for Part 2: the cause.
Part of this effect is likely a systematic bias that people have about believing that they are just generally crushing it. This is called the above-average-effect, better-than-average effect, or illusory superiority. People just rate themselves as above average at most things. This has been found a lot.
However, Dunning and Kruger argue that an additional factor is at play in this specific effect, and this explanation captures a lot of attention. Poor performers rate themselves as better than they are because of failed metacognition, which is to say, they lack insight into their abilities. Poor performers (or "the incompetent") lack the ability to know that they're not so hot. These poor performers suffer what's called a double burden: not only do their lack of skills lead them to produce the wrong answer, but it also prevents them from recognizing their error, thus inflating their impressions of themselves. According to Dunning and Kruger, the same knowledge is needed to do the task and recognize how you're doing. If you don't have that knowledge, you're bad at both. Poor performers think they did better than they did because they lack the ability to accurately assess themselves. As you become more competent, you gain both the skills needed to master the task and more accurately evaluate yourself, creating greater metacognitive calibration. The people at the top may have a tendency to think everyone knows as much as they do (the false consensus effect), thus making them think that other people did better. This slightly lowers their own self-estimation.
This idea of a lack of metacognition on the part of the incompetent has been pretty front-and-center lately. However, this interpretation has been subject to a bit of debate. As it turns out - as is so often the case - the original authors are not the only ones with thoughts about the cause of their reported pattern of results.
One alternate explanation is regression to the mean, and it's a big one. All people are going to make some errors about their performance. These errors will tend to pull toward the average. Therefore, you will see low performers rating themselves as higher than they are and high performers rating themselves as lower than they are. When this effect is statistically controlled for, the Dunning-Kruger effect is reduced, but it doesn't go away completely. So it's probably part of the effect, but not the whole effect.
One argument is that the 'above-average effect' + 'regression toward the mean' are sufficient to explain the Dunning-Kruger effect: no metacognitive explanation is necessary. Or at least, low-performers need not have poorer insight than high performers. Think of it this way: everyone may think that they're better than average, but some people actually are. Are those people better calibrated - do they have more insight - or do they have the same "I'm awesome" bias as everyone else, but they happen to be right? McIntosh et al. (2019) suggest that all three factors can contribute: a little bit of general above-average effect, a little bit of statistical quirks, and a little bit of low metacognition (with the last being a very small, and not even necessary, contributor).
While the debate about the cause of the Dunning-Kruger effect rages on, go ahead and use quirky psychological phenomena to insult people. I won't judge.
Last month I attended AIAA's Aviation Forum in Dallas, Texas. If you know me well, you know I love all things aviation from flying, to air traffic control, to airport design. The Aviation Forum is one of my favorite events of the year and this year was no different. Front and center throughout the week was the concept of Urban Air Mobility (UAM). If you're unfamiliar with the concept, think "The Jetsons," small aircraft zipping you from point to point throughout and between cities and suburban areas. Ready or not, here it comes!
While UAM may be in its infancy, there's no shortage of companies and organizations working to ensure its success, and for good reason. The UAM industry is estimated at a $500B market value. Critically though, if it's not done right the first time, it may be even further off than anyone realizes. The implementation of UAM is wrought with technical, logistical, policy, and infrastructure challenges. The biggest challenge however, is likely not with any of the components of the new system, but with society's acceptance (or lack thereof) of flying in small, unpiloted, automated aircraft. That's right. No pilot. You hop in, enter your destination, and off you go. I, for one, am all about it and can't wait until it becomes mainstream. That may be some time off, however. As automation continues to creep further into our lives, research, and probably your own personal experience indicates just how reluctant humans are to accepting automation and how long adoption into the mainstream can take.
But, if you think about it, elevators used to have operators and are now fully-automated. Passenger jets are about 95% automated but we still have at least 2 pilots at the controls. Trains or trams are becoming fully automated, particularly at airports. Do you think twice before you get on the tram shuttling you between terminals in Dallas, Orlando, Atlanta, or countless other airports across the globe? Cars and trucks are on their way to being fully-automated but I bet there are some steadfast concerns about getting into a fully-automated car or driving on the road alongside fully-automated cars. What is it that holds us back and prevents our unyielding trust in automation?! Failure. Accidents. Injury. Death. Ask Boeing. Those are simply the most headline-grabbing examples. Other examples you may not think of such as brand recognition, concern for dangerous or unruly passengers, privacy, presence or lack of a flight attendant, and cost are all part of the society's reluctance to accept UAM in the short term and hurdles that must be overcome. This is a great Market Study put together by our friends at Booz Allen Hamilton investigating such barriers to acceptance.
UAM has the opportunity to revolutionize the way we travel but it needs to be done right and it needs to be done well. It's entirely possible I won't even see UAM go mainstream in my lifetime, but it's certainly fun to consider the possibilities. What are you most concerned or excited about with automation becoming more prevalent in our everyday lives!?
Also… just look at how cool this is, https://lilium.com/.
Last month I attended InfoComm 2019 in Orlando, FL. InfoComm is the "largest audiovisual and integrated experience event in North America." Why did my bosses make me go? Well, at the time, we didn't know, but the goal was to explore how other industries are approaching the user experience through technological innovations and see if there is potential benefit for training, human factors intervention, and/or human performance assessment. Augmented and virtual reality (AR/VR) have become popularized by the entertainment industry, boosting interest and technological advancements, and look where they are being used now. I came across a few potential products, such as a carbon fiber mesh screen that generates holographic images with the use of a projector, which provides 3D visualizations without the need of glasses. Can you see the benefit?
On the exhibit floor, InfoComm offers free educational sessions at their Center Stage. There I learned about how a Fortune 500 copper mining company was using digital signage to connect its workforce in the field. Through its application, safety has improved and communication is more effective across the local, regional, and national levels. Their representative made an interesting comment about content novelty: one of the most effective ways to ensure employees pay attention to signs was to make sure the content was regularly updated and fresh. How can that concept be applied to the training industry?
Another session discussed how classrooms in over 60 schools across Manatee County, FL were revitalized with new displays and tablets. The concept was to update decade old-technology with modern equipment to better serve digital natives. The most important thing I got from this, which is something we at QIC champion as well, was that they weren't simply replacing physical white boards with digital ones, but were leveraging the technology to engage students by presenting the learning content in various ways, such as short videos or interactive exercises. The experience of learning is just as important as the content being learned, and this technology allowed them to create individual learning experiences for their students. How are you helping generate individualized learning experiences?
The takeaway is not to get stuck in your view of the world. There are many facets to a problem, each requiring a different perspective. Step outside and see what everyone else is up to, it will broaden your mind and fuel the spark for innovation.
These posts are written or shared by QIC team members. We find this stuff interesting, exciting, and totally awesome! We hope you do too!