Why Don't You Just Ask Them?

Julian Abich, Ph.D., Senior Human Factors Engineer

Why collect subjective data from the end-users when evaluating a training device? I was asked this question at a conference last month, almost as if subjectivity is a dirty word. The short answer is that if your users don't like the training device, they likely won't use it. If they don't use it, you won't get the objective data you may be looking for. The term "like" does not necessarily mean the training experience was pleasant. For instance, Soldiers, firefighters, surgeons, etc., must train under difficult, often intolerable conditions to prepare for the real-world challenges of their job. "Liking" the training, in such cases, means that the trainees see the value in preparing them to do their job successfully. When we collect subjective data, we also ask users if their expectations were met, what their emotional responses were from their experience, and most importantly, why. If we only gather objective data, such as the time to complete a task or the number of errors made, then we are only getting half the story. For example, why did it take so long to complete the task and why were certain errors made?

---

On a project for the U.S. Air Force, QIC's role was to conduct usability and user experience evaluations on a virtual flight simulator (Abich & Sikorski, 2022; Abich, Montalbano, & Sikorski, 2021). One of the most compelling responses I heard from an instructor pilot as he walked into the room was, "This looks like training." I reflected on that and thought, "What does it mean to look like training? How does the user's initial impression affect the feedback provided? How does the design impact their motivation to give the training device a chance? And why should any of this matter if the training device does what it's supposed to do?" The answers to these questions are subjective as end-users apply their unique expertise, experience, and expectations in evaluating the device. They are the ones that will have to use the devices and can see the value or potential shortcomings. If their lives depend on the skills, you can bet they will be particularly critical of any new training device. So next time you do a training device evaluation, embrace the subjective and don't worry about getting a little dirty. Otherwise, you may be missing out on some highly valuable feedback. ​

---