While failure is no fun, it is a part of life. Whether summitting mountains or managing projects, there are times when one must recognize the signs of failure, make the tough decision to call it a day, and learn from the experience to return smarter and stronger next time.
This past summer, I took a vacation to Colorado which was a blast. Three days on the agenda consisted of backpacking into the Chicago Basin, camping, and summiting 14ers (mountains over 14,000 feet). I had no prior mountaineering experience though was with someone who did, and we planned extensively. I had the right gear, was in good shape, and the conditions were ideal; I was feeling confident. On the second day, as I attempted a second summit, I began feeling fatigued, making mental errors, and falling behind. Though I tried to press on, I was aware that poor decisions and continued mistakes, especially near the summit, could be catastrophic (see Wickens, Keller, & Shaw, 2015). 

I made the difficult call to abandon the summit bid and hiked down to lower elevation. Watching the beautiful sunrise over the basin lakes, I began to reflect on the failure, thinking about it in terms of project management given a summit bid, like a project, is “a temporary endeavor undertaken to create a unique…result” (Project Management Institute, 2008, p. 5). Though a project manager (PM) never plans to fail, they do identify the risks and notice the potential signs of failure so that they can correct course. As a last resort, a responsible PM may decide it is best to call it quits before stakeholder and company losses become too great. 

Recognizing the Signs
In a Project Management Institute (PMI) published article "Managing Troubled Projects", Alaskar (2013) outlines signs that a project may be in jeopardy. Several of these apply to my summit bid:

• Delays in meeting milestones or completing deliverables
  • We left camp late. Scheduled departure was 4:00am and we left at 4:15am
• Project is unacceptably behind the planned schedule
  • The late departure put us immediately behind schedule and it did not get any better from there as we continued to lose time following a missed turn in the dark and several unplanned stops enroute to the summit
• High risk to the project's likelihood in delivering anticipated benefits
  • The delays and my increasing fatigue significantly threatened the likelihood of success, the obvious definition of success being reaching the summit 
• Critical and/or significantly growing technical issues with the project, and
• Frequent intervention required by the management team
  • My travel partner had substantial mountaineering experience, having summited 47 of the 58 Colorado 14ers to that point, making her pretty awesome and the de facto manager on this adventure. The more mistakes I made, the more she had to intervene, such as my hiking pole bottom falling out (I did not properly secure it having rushed to begin the hike) and her catching it while below me on the slope.
• Morale of the team has hit rock-bottom
  • We were both frustrated, with the rushed situation, the mistakes, and admittedly with each other. This contributed to low morale and there was little time to regroup.
    

Of course, in recognizing these signs, there is always hope for turning the project around. A PM certainly need not accept failure, and there is plenty of literature on how to save troubled or failing projects. Yes, I may have been able to eat a CLIF BAR® and rally, but that was not the case. So, in the rare and unfortunate event a project does fail, you can learn from that experience.     

Learning from Failure
In another PMI published article, Ranganath (2006) presents a learning from experience (LifE) cycle for project management that applies to dealing with project failure. The cycle involves:

• Planning the experience – identify milestones where it is important to assess risks and points of failure
• Living the experience – accept that failures and problems are as much a part of the experience as successes
• Reflecting on the experience – examine the symptoms and determine the probable causes of failure
• Learning from the experience – analyze the data to inform appropriate corrective action for the next time, document patterns that may be recognizable in the future, and establish change initiatives for improvement
• Acting on the experience – determine the effectiveness of the change initiatives through execution
• Reliving a new experience – move forward, not dwelling on the mistakes of the previous lived experience while continuing to monitor risks and look for opportunities to improve

My initial reaction was that I will not relive a mountain summit bid experience having learned to restrict my endurance activities to sea level. Upon further reflection, I am willing to attempt another summit though I need to better prepare, now having a more complete understanding of the work to be done and the potential points of failure. This preparation would include modifying my approach to physical training and trying to gain more exposure to heights. I’m hopeful that reliving the experience while better prepared will result in success. However, there is always the chance that, as John Popper aptly sang, “the mountains win again.” 

This semester QIC welcomes Natalie Paquette as a Human Factors Intern! Natalie is a Ph.D. student in the Human Factors and Cognitive Psychology program at the University of Central Florida (UCF). Natalie earned her MA in Applied Experimental and Human Factors Psychology in 2020 at UCF and her MA in Psychology focused on Cognitive and Behavioral Neuroscience at George Mason University in 2017. Her work has examined performance issues related to mismatched expectations, reliance on visual working memory, and the effect of restricted time intervals on error processing. Natalie’s interests include examining the neurophysiological and perceptual aspects of cognition and performance in various environments to determine optimal parameters for successful task completion.

This semester QIC welcomes ​Nicolas Uszak as a Human Factors Intern! Nicolas is a Ph.D. student at the University of Central Florida’s Human Factors and Cognitive Psychology Program. Nicolas has a Master’s in Applied Experimental Human Factors and a graduate certificate for Design Usability in Industrial Engineering, both from UCF. Previously he graduated summa cum laude with his B.A. in Psychology from Cleveland State University. His interests lie in motivation, situational awareness, automation, multi-tasking, vigilance, and machine learning. Nicolas is currently working on his dissertation involving situational awareness while operating automated vehicles. 

I've attended and presented at several conferences this year, such as the Human Systems Digital Experience, World Aviation Training Summit (WATS), and Applied Human Factors and Ergonomics (AHFE), and have noticed a simple yet powerful construct appearing over and over again…self-efficacy. Self-efficacy is "concerned with people’s beliefs in their capabilities to produce given attainments" (Bandura, 2006). In other words, it's the confidence  in the ability to exert control over one's own motivation, behavior, and social environment (Carey & Forsyth, 2009). Extensive evidence has shown self-efficacy to be a significant predictor across a variety of contexts and domains, such as college academic performance (Choi, 2005), pre-career pilot performance (Wilson, 2021), weight loss success (Armitage et al., 2014), health management (Arslan, 2012), second language skills (Raoofi, Tan, & Chan, 2012), and work burnout and engagement (Ventura, Salanova, & Lloren, 2015). While there are a host of other areas that have explored the role of self-efficacy, one of particular interest to me that has been gaining more attention is usability. 

Usability assessments typically focus on effectiveness, efficiency, and satisfaction, but research suggests the integration of self-efficacy can provide a robust assessment (Martin, 2007). As the preponderance of technological solutions continuously diffuses across all aspects of our personal and work lives, our dependency on them will impact our ability to complete tasks. Therefore, belief in our ability to complete a task with a technological solution should impact the way in which these solutions are designed. Plenty of evidence exists indicating the role of self-efficacy in the adoption of technology, such as mobile learning solutions (Bettayeb, Alshurideh, Al Kurdi, 2020), desktop virtual environments for learning (Makransky & Petersen, 2019), fitness devices (Rupp, Michaelis, McConnel, Smither, 2018), and medical support tools (Lindblom, Gregory, Wilson, Flight, & Zajac, 2012). 

Although some usability measures focus on or integrate the concept of self-efficacy, not all are implemented correctly based on Bandura's guidance (2006). One major flaw is using Likert-type bipolar ratings (e.g., strongly agree to strongly disagree) instead of unipolar ones  (e.g., 0  to 10). The issue is if you have zero confidence in your ability to complete a task, then negative ratings below zero make little sense and lead to skewed interpretations of the results. Further, when bipolar ratings are used, the mid-point (usually labeled as neither agree nor disagree) gets converted into a moderate-level of self-efficacy which is incorrect and not a true reflection of the construct (Bandura, 2012). Leveraging self-efficacy as a usability metric can provide valuable insight into the design and evaluation of technology, but it's critical that measures be developed, implemented, and interpreted appropriately.

​Bandura (2012) stated that "there is no single all-purpose measure of self-efficacy with a single validity coefficient." This indicates that it's expected for new measures of self-efficacy to be created for, as he puts it, "activity domains." Activity domains are the topic areas in which the tasks under evaluation are performed. For example, evaluating self-efficacy for driving a monster truck. Use these guidelines when developing your measures and scales for usability evaluations (or any evaluation for that matter):

• Use unipolar ratings with multiple gradations of strength (at least 0 - 10 rating)
• Use a set of questionnaire items within the scope of an activity domain
• Items should indicate the goals that should be achieved for the tasks
• Make sure the items have clearly defined levels of attainment or performance
• Items should be measured against varying levels of task demands or challenges to avoid ceiling or floor effects
• Avoid developing measures for specific performance on a specific task
• Use statements with "I can" as opposed to "I will" since the latter reflects intention, not self-efficacy
• Avoid items that request foretelling a future event, but instead focus on judging their belief in reaching a specific achievement
• When capturing collective efficacy of a group, be mindful of the level of interdependence of group members to meet a task goal
• Be prepared to develop multiple assessments of self-efficacy for complex, multidimensional activity domains

The major challenge with the use of artificial intelligence (AI) is that it is often difficult to explain how AI or machine learning (ML) solutions and recommendations come to be. Previously this may not matter as much because AI’s use was limited and its recommendations were confined to relatively trivial decisions. In the past few decades, however, AI use has become more pervasive and some of these AI solutions are impacting high-stakes decisions, so this problem has become increasingly important. Fueling the urgency are findings that AI solutions can be unintentionally biased, depending on the type of data used to train the algorithms. For instance, the algorithm used by Amazon to hire staff was found to be biased against women because the algorithm was trained on previous data that largely comprised resumes from male applicants (Shin, 2020). Algorithms used by COMPAS (i.e., Correctional Offender Management Profiling for Alternative Sanctions) to predict likelihood of recidivism were also found to be predict that black offenders were twice as likely to reoffend compared to white offenders (Shin, 2020). These are only two of many instances of algorithm bias.

These algorithms tend to be from “black box” models that are developed from powerful neural networks performing deep learning. These neural networks are usually what researchers have to use for computer vision and image processing which are less amenable to other AI/ML techniques. Researchers in the field of explainable AI (or XAI) have tried to shine a light into these “black boxes” in different ways. For instance, one popular way to help explain deep learning AI processing images is to use heat or saliency maps that show the regions/pixels of a picture that seem to highly influence the algorithm’s prediction (i.e., what the network is “paying attention to”). If these regions aren’t relevant to the algorithm’s task at hand, then the researcher may be looking at a biased algorithm. In her recent talk at a Deep Learning Summit, Rohrbach (2021) showed an example of wrong captioning by a black box model, mislabeling the woman sitting at a desk in front of a computer monitor as “a man” (Figure 1a). 

The saliency map showed that the network was attending more to the computer monitor than the person in the picture (Figure 1b), when it should have been focusing more on the person (Figure 1c). Presumably the bias arose because the training data contained more images of men sitting in front of computers than women doing the same. 

Saliency maps also provide a way to evaluate the “logic” of the algorithm even when the captioning seems appropriate (see Figures 2a & 2b). 

A major criticism of such saliency maps is that they only show the inputs of importance in deriving the algorithm. They do not reveal how these inputs are used. For instance, if two models had different captions or predictions but had very similar saliency maps, the maps would not explain how the models reached their different predictions (Wan, 2020).

Due to such limitations, other researchers such as Dr. Cynthia Rudin, propose that instead of making AI explainable, interpretable models should be developed instead. The difference being that interpretable models are inherently understandable, and the researcher is able to see how the model derives its solutions and algorithms, instead of merely trying to coax explanations from a model by reviewing what inputs were more influential than others after the algorithm has been developed (Rudin, 2021). Dr. Rudin’s work to make neural networks interpretable has received wide acclaim and she is a strong proponent of using interpretable models especially for high stakes decisions (Rudin, 2019). However, as she stated in the recent Deep Learning Summit, Dr. Rudin also admitted that developing such interpretable models takes more time and effort. This is because unlike black box models where researchers would not know if the model is working properly, interpretable models force researchers to work to troubleshoot the data when they see that the model is not working as it should – even if the solutions look ok (Rudin, 2021).

Nevertheless, there are some who still argue for the use of black box models that are low on explainability. Black box models are harder to copy and can give the company that developed them a competitive advantage. They are also easier to develop. Proponents of this view believe that the goal is not to explain every black box model but to identify when to use black box models (Harris, 2019). If a black box algorithm is able perform its task to a high degree of accuracy, we may not need to know exactly how it did it. Besides, what is a valid explanation to one may not be a valid explanation to another. These researchers argue that even physicians use things that they do not fully understand all the time, to include common drugs that have been shown to be consistently effective even though no one totally comprehends how they work in every patient. What is important is that that enough testing is done to ensure that the algorithm is dependable and suitable for its intended use (Harris, 2019).

When I first learned and read about AI and the problem of inexplicability some years back, I remember taking the position that the ends justified the means. That is, so long as the algorithm was accurate I’d be willing to sacrifice explainability. However, as I learn more about how people are using AI solutions for all sorts of important medical, hiring, and criminal justice decisions, I started to reconsider my position. After attending the Deep Learning Summit on Explainable AI (XAI) earlier this year, I am more inclined to think that it really depends on the application and type of decision involved. Just because a black box model has a sterling track record of highly accurate predictions in the past does not mean that it is not possible for the next prediction to be poor. The higher stakes the decision is, the better understanding we should have of the workings of the AI.
​
What do you think? For what applications or decisions would you accept a model that is unexplainable but have been consistently accurate?