​In military training environments, critical incidents such as injury, friendly fire, and non-lethal fratricide may occur. Quantitative performance data from training and exercises is often limited, requiring more in-depth case studies to identify and correct the underlying causes of critical incidents. The present study collected Army squad performance, firing, and communication data during a dry-fire battle drill as part of a larger research effort to measure, predict, and enhance Soldier and squad close combat performance. Soldier-worn sensors revealed that some quantitatively rated top-performing squads also committed friendly fire and a fratricide. Therefore, case studies were conducted to determine what contributed to these incidents. This presentation aims to provide insight into squad performance beyond quantitative ratings and to underscore the benefits of more in-depth analyses in the face of critical incidents during training. Squad communication data was particularly valuable in diagnosing incident root causes. For the fratricide incident specifically, the qualitative data revealed a communication breakdown between individual squad members stemming from a non-functioning radio. The specific events leading up to the fratricide incident, and the squad’s response, will be discussed along with squad communication patterns among high and low-performing squads in the context of various critical incidents. We will examine how the conditions surrounding critical incidents and the underlying causes of those incidents can be recreated and manipulated in a simulated training environment, allowing instructors to control the incident onset and provide timely feedback and instruction.  

​Artificial intelligence (AI) can facilitate personalized experiences shown to impact training outcomes. Trainees and instructors can benefit from AI-enabled adaptive learning, task support, assessment, and learning analytics. Impacting the learning and training benefits of AI are the instructional strategies implemented. The co-learning strategy is the process of learning how to learn with another entity. AI co-learning techniques can encourage social, active, and engaging learning behaviors consistent with constructivist learning theory. While the research on co-learning among humans is extensive, human-AI co-learning needs to be better understood. In a team context, co-learning is intended to support team members by facilitating knowledge sharing and awareness in accomplishing a shared goal. Co-learning can also be considered when humans and AI partner to accomplish related tasks with different end goals. This paper will discuss the design of a human-agent co-learning tool for the United States Air Force (USAF) through the lens of constructivism. It will delineate the contributing factors for effective human-AI co-learning interaction design. A USAF maintenance training use case provides a context for applying the factors. The use case will highlight the initiative of leveraging AI to help close an experience gap in maintenance personnel through more efficient, personalized, and engaging support.

I have two younger brothers, and once a week, the Solberg kids have a phone call. We each hold management positions in the technology world, so we swap notes about work a lot. We may be one of the few families with a running joke about Kubernetes. After our call the other day, my brother Mike sent me this article from the National Bureau of Economic Research, and I have been geeking out over it ever since. From what I can tell, it’s one of the first studies on how generative AI can improve an organization’s effectiveness – and not just in terms of its bottom line.

Something I’ve been thinking about is how to train empathy. If you work in user experience, you understand how important empathy is to good design. Often, we end up working with solutions that suffer from “developer-centered design,” where features are built to check a box while minimizing the work for the development team. Also, we see “stakeholder-centered design,” where software development happens to impress someone with a pile of money. At the end of the day, if the people who need your solution can’t figure out how to use it, none of the rest of it matters.
​
Empathy means putting yourself in someone else’s shoes. More importantly, it involves caring about other people, which seems hard to come by these days. Wouldn’t it be great if we could make something that teaches people how to do that? For a long time, I wondered whether virtual reality could show you someone else’s perspective, and I still think it could. This study shows there may be a different way.

The study took place in a large software company’s customer support department. Working a help desk is a job where empathy is key to success. Not only do you have to be able to solve an irate and frustrated customer’s problem, you have to ensure they have a positive experience with you. In this research, customer support agents were given an AI chat assistant to help them diagnose problems but also engage with customers in an appropriate way. The assistant was built using the same large language model as the AI chatbot everyone loves to hate, ChatGPT. The assistant monitored the chats between customers and agents and provided agents real-time recommendations for how to respond, which agents could either take or ignore. As a result, overall productivity improved by almost 14% in terms of the number of issues resolved. Inexperienced agents rapidly learned to perform at the same level as more experienced ones. The assistant was trained on expert responses, so following its advice usually gave you the same answer an expert would give.
​
Here’s where it gets really interesting: a sentiment analysis of the chats showed that as a result of using the assistant, there was an immediate improvement in customer sentiment. The conversations novice agents were having were nicer. The assistant was trained to provide polite, empathetic recommendations, and over a short period of time, inexperienced agents adopted these behaviors in their own chats. Not only were they better at solving their customers’ problems, but the tone of the conversation was overall more positive. The agents learned very quickly how to be nice because the AI modeled that behavior for them. As a result, customers were happier, management needed to intervene less frequently, and employee attrition dropped.
​
The irony of AI teaching people how to be better human beings is palpable. Are the agents that used the assistant more empathetic? We don’t know, but from a “fake it until you make it” perspective, it’s a good start. That aside, this study is an example of how this technology could help people with all sorts of communication issues function at a high level in emotionally demanding jobs. Maybe we should spend a little more time thinking about how it could help many people succeed where they previously couldn’t and focusing less on how it’s not particularly good at Googling things.

An estimated 602,000 new pilots, 610,000 maintenance technicians, and 899,000 cabin crew will be needed worldwide for commercial air travel over the next 20 years (Boeing, 2022). That’s about 30,000 pilots, 30,000 maintenance technicians, and 45,000 cabin crew trained annually. Additionally, urban air mobility is creating a new aviation industry that will require a different type of pilot and technician. It's clear there is a demand across the entire aviation industry to turn out new personnel at an increased rate. The demand should not just be met by numbers, but also by knowledge and experience. But how can you train faster, cheaper, and yet still maintain (and exceed) current high standards? Is the answer extended reality (XR) technology? (Hint: that's part of the solution). These are the problems being tackled by the aviation community and discussed at the World Aviation Training Summit (WATS).

This year was the 25th anniversary of WATS. I've been attending and presenting at WATS for the past three years. In that time, I've seen the push for XR technology met with valid concerns for safety. If you know me, then you've heard me harp on the need for evidence-supported technology for training, so I can appreciate that stance. Technology developers have an ethical responsibility to conduct or commission the appropriate research before boasting claims of training effectiveness, efficiency, and satisfaction. There are many companies that came to the table with case studies showing the training value of their solution. Other companies want to do the same but may need more opportunities or research support. On the other side, some commercial airlines have been conducting XR studies internally, but the results tend to stay within the organization. The regulating authorities (e.g., FAA, ICAO, etc.) need the most convincing. They need to see the evidence and clearly understand where XR will be implemented during training. There may be an assumption that XR is the training solution but XR is just part of a well-designed, technology-enabled training strategy. It was great to hear many presenters convey this same message and reiterate the need for utilizing a suite of media suitable for developing or enhancing the necessary knowledge, skills, and abilities.

Extended reality (XR) technologies have been utilized as effective training tools across many contexts, including military aviation, although commercial aviation has been slower to adopt these technologies. While there is hype behind every new technology, XR technologies have evolved past the emerging classification stage and are at a state of maturity where their impact on training is supported by empirical evidence. Diffusion of innovation theory (Rogers, 1962) presents key factors that, when met, increase the likelihood of adoption. These factors consider the relative advantage, trialability, observability, compatibility, and complexity of the XR technology. Further, there are strategic approaches that should be implemented to address each of these innovation diffusion factors.
 
This presentation will discuss each diffusion factor, provide exemplar use cases, and outline evidence-backed considerations to improve the probability of XR adoption for training. Considerations will discuss various effects that may occur with the introduction XR technology, such as the novelty effect where improved performance initially improves due to new technology and not because of learning. Key questions will be presented that should be addressed under each diffusion factor that will help guide the information needed to support the argument for XR adoption. Importantly, the quality of research evidence to support XR implementation and adoption is critical to reducing the risk of ineffective training. Therefore, a discussion of research-related considerations will also be presented to ensure an appropriate interpretation of existing XR research literature. The goal is to provide the audience with an objective lens to help them determine whether XR technologies should be adopted for their training needs. 

Why collect subjective data from the end-users when evaluating a training device? I was asked this question at a conference last month, almost as if subjectivity is a dirty word. The short answer is that if your users don't like the training device, they likely won't use it. If they don't use it, you won't get the objective data you may be looking for. The term "like" does not necessarily mean the training experience was pleasant. For instance, Soldiers, firefighters, surgeons, etc., must train under difficult, often intolerable conditions to prepare for the real-world challenges of their job. "Liking" the training, in such cases, means that the trainees see the value in preparing them to do their job successfully. When we collect subjective data, we also ask users if their expectations were met, what their emotional responses were from their experience, and most importantly, why. If we only gather objective data, such as the time to complete a task or the number of errors made, then we are only getting half the story. For example, why did it take so long to complete the task and why were certain errors made?

On a project for the U.S. Air Force, QIC's role was to conduct usability and user experience evaluations on a virtual flight simulator (Abich & Sikorski, 2022; Abich, Montalbano, & Sikorski, 2021). One of the most compelling responses I heard from an instructor pilot as he walked into the room was, "This looks like training." I reflected on that and thought, "What does it mean to look like training? How does the user's initial impression affect the feedback provided? How does the design impact their motivation to give the training device a chance? And why should any of this matter if the training device does what it's supposed to do?" The answers to these questions are subjective as end-users apply their unique expertise, experience, and expectations in evaluating the device. They are the ones that will have to use the devices and can see the value or potential shortcomings. If their lives depend on the skills, you can bet they will be particularly critical of any new training device. So next time you do a training device evaluation, embrace the subjective and don't worry about getting a little dirty. Otherwise, you may be missing out on some highly valuable feedback. ​

Thankfully, at QIC, I report to the COO who meets with me weekly to ensure not only that my workload is balanced, but that I have the resources I need to be successful, including the flexibility necessary to maintain a work-life balance. I wish I could say this took having tough conversations to achieve, but the truth is, our company culture not only encourages us to be upfront and authentic, it requires it. 
 
While there is certainly more work to be done to support women, especially mothers, I am proud of the conversations I hear taking place around me. We are finally vocalizing so much of what has been deeply felt for too long. Looking ahead, I hope we can turn this one-day celebration into a year-round push for equality.  
​
If any of this interests you, check out this study from the LeanIn organization for more information and this site for ways to get involved. ​

2023 is here, and I’m not ready. As the project manager at QIC, I ensure that our project objectives are met using techniques rooted in the Agile methodology. No project has felt too daunting to manage nor a team too challenging to collaborate with. Yet here I am, staring January in the face, terrified of how I will manage my family and household. I need a plan. I need help. There’s going to be a crib-to-bed transition. Potty training. Swimming, ballet, soccer, gymnastics, skating, and karate classes. It’s. All. Too. Much. How to stem the hysteria? How to tackle this overwhelm?
​ 
I repeat I’m a capable, successful project manager. Can’t I use the same tools at home to bring order to the chaos? As it turns out, I’m late to this revelation. According to Bruce Feiler (TED talk), Agile was just what his household needed to “cut parental screaming in half.” I was simultaneously floored and elated. I had spent a decade working on mastering a framework that I could have been using in my personal life all along.

I will spare you the history and details of Agile, but if you’re interested, here is an excellent place to start. In short, the purpose of Agile is to make progress in an ever-changing environment, utilizing an empirical process to make decisions to ensure we move the needle in the right direction. As we understand at QIC, R&D contracts often come with many unknowns, having to meet tight timelines and budgets while leaving room for exploration, collaboration, and adaptation. Using Agile principles, our team meets at predictable regular intervals to regroup and connect to share progress and information to continue to improve and deliver value.
  
On the home front, my husband and I connect after the kids’ bedtime to have a warming beverage and look at the day ahead. This often involves laughing over who cried more, us or the kids, and ways to reduce tears. This is similar to the sprint retrospective when the team discusses ways to work better together to avoid recurring problems. It is also like the daily standup when the team shares what they will be doing for the day, enabling them to communicate roadblocks early, and assist where necessary. Granted, we may not have much progress involving our 2yo and 5yo kids in these discussions despite being key stakeholders; I would like to think that very soon they will be. Empowering them to participate in the process will hopefully lead to cooperation and a happy home life.

​Agile applied in the home is a compelling idea, one that I’m willing to try this year. If I fail, at least I’ll be failing fast! Would you try any of these approaches to household management in your personal life? What skills do you hone in your workplace that you could apply at home?  

Last week, Congress blocked a $400 million award to Microsoft for the purchase of nearly 7000 Integrated Visual Augmentation System (IVAS) systems for the Army. The award would have followed the Army’s investment of $125 million to develop the Hololens-based augmented reality headset. The issue? IVAS proved to be spectacularly problematic in user testing last year, with 80% of Soldiers reporting physical discomfort, eye strain, nausea, and other issues within the first three hours of use. Instead, they awarded a $40 million contract to develop yet another version of IVAS…to Microsoft, the same company that delivered the current system.  

On the one hand, as one who conducts these kinds of end user research, it’s gratifying to see the Army doing extensive user experience testing before deploying something this invasive and frankly, potentially dangerous. Using a Hololens, or any other augmented reality headset, in a controlled environment is one thing. Having your field of view occluded by constantly changing data streams while you’re being shot at is potentially a human factors nightmare. At the very least, if Soldiers don’t like it, they won’t use it. And it turns out, they don’t. In addition, the methodology of this testing has been criticized by the Inspector General.

User research is critically important when you’re developing a capability your users will interact with on a daily basis. The process should start before the prototype stage, though. The impetus for IVAS did not come from the boots on the ground; it’s the result of nearly 20 years of research and development into head-worn augmented reality. It’s not clear whether Soldiers were ever asked “What are your problems?” before they were asked “Do you like this thing we’ve made to solve them?”
​
We see this problem a lot working in the training technology space. We’re regularly asked to develop solutions without talking to the intended users until a prototype is developed. I get it. Their time is valuable. They’re hard to get a hold of. There’s a lot of bureaucracy involved. We don’t need to talk to them, we’re the experts. Yes, we’re the experts in how to apply the science behind why things work, and we’re the experts in designing the solution. We’re also the experts in evaluating the solution. But we can’t be the experts in how people do their jobs, the problems they have, the barriers to solving them, and their work environment without listening to them. Because it’s never as straightforward as it seems.
​
Where does IVAS go from here? Some would argue there are too many technical barriers to overcome right now. Others would argue that the contract should be recompeted instead of continuing to throw money at a sunk cost. I would argue the Army should take a step back and ask if this solution broadly is the right one to solve the Infantry’s problems today. ​