Every year, I look forward to the Learning Guild’s DevLearn conference in Las Vegas. It’s a terrific opportunity to catch up with friends and colleagues, hear some great speakers, and check out the latest and greatest in L&D. A few months before the conference, they have a competition to design the logo for the conference tee shirt. There are some very talented artists in the DevLearn community, and I am by no means one of them. So, I never considered entering until this year. What changed? Did I take dozens of art classes and practice tirelessly? No. Just like everyone else, I started using generative AI tools on a daily basis. And so, when DevLearn announced the tee shirt contest this year, I found a golden opportunity to do two things I love doing at the same time: winning a contest and entertaining myself. Here are my submissions: The aesthetic is self-explanatory. Regardless, a good artist leaves their work open to interpretation.
Now, to be clear, I did not actually think I was going to win this contest, nor did I mean to. I wanted it to be very clear that I had used AI to create these images, so I left the typos and miscellaneous dots in. Besides, I think they provide an element of messiness that symbolizes the questionable decisions some of us make in Las Vegas. I was hoping to prove a point, and the Guild's leadership did exactly what I thought they would: I was told that my submissions were disqualified because I used AI to create them. This perspective is a little ironic given that according to the DevLearn Concierge GPT made with OpenAI's ChatGPT, 27 sessions at the conference are focused on applications of AI in L&D. Furthermore, there is an entire day-long AI and Learning Symposium on Tuesday, where you can learn how to use AI tools in instructional design. Despite the conference's clear - and appropriate - focus on using AI to get your work done, when it came down to accepting work created by AI, they had a contradictory position. There are myriad opinions on how, when, and where using AI tools (like Dall-E, in this case) to create things is acceptable. This time last year, I told people that the government, courts, and regulatory bodies had a responsibility to make rules and laws about this, and that hopefully we would get some clarity on the issue soon. Well, it's been a year, and we are no closer to resolving this debate. In the meantime, we are going to have to operate on a case-by-case basis, follow our guts, and do what we think is best. Here are a few things the Guild (or anyone having a contest) could do in the future:
Personally, I'd love to see these designs on a shirt, but I'll settle for a sticker. If you want one, too, find me Tuesday at the AI and Learning Symposium and Wednesday at 3:00 in room 122, where I'll be discussing research about trust in AI and automated systems. You can also get one from Build Capable's Sarah Mercier, who will be at the RISC booth showing off an amazing new product. See you in Dev Legas!
0 Comments
The other day, my co-workers and I were having a lively conversation about products that supposedly help us be more “grounded.” They were electricity-powered bed sheets, blankets, and pillowcases. The websites—yes, there was more than one manufacturer—for these products boast glowing testimonials and reviews. You can reconnect with nature by sleeping in bed sheets plugged in to "ground" you to the Earth’s electrons and neutralize your free radicals, drinking alkali water to prevent disease by neutralizing acid in your blood, and wearing copper bracelets to regrow joint cartilage in order to relieve arthritis pain. While these may work for some, there hasn’t been much scientific evidence of their efficacy, so individuals who swear by these health-promoting strategies and products are likely experiencing the placebo effect. “So what?” you ask. If the mechanism by which these work for some individuals isn’t the mechanism touted by their proponents, should we get upset about that? Maybe it’s a case of “no harm, no foul.” So long as it doesn’t hurt to try the grounding blanket and if someone can benefit from it, then what’s the fuss? Perhaps anything could have therapeutic benefits—if we haven’t seen any, maybe it’s because the “right” person hasn’t used it yet. Maybe the truth about grounding blankets and other similar treatments is not whether they work or not, but it’s about who you ask. Does this make the truth about the grounding blankets relative? I don’t think so. I think the absolute truth about these treatments is that they illustrate the potency of the mind’s belief in health and medicine. In other words, we must consider both objective data as well as the individual’s subjective experience. The challenge lies in capturing this subjective experience in a meaningful way. Unlike objective data which can be verified from a “third person’s” view, an individual’s subjective experience is only accessible from a “first person’s” view. Our subjective experiences can change over time with circumstances and even with our frame of mind. This makes subjective experiences elusive and hard to capture. Researchers have attempted to look for objective neural correlates with specific subjective experiences such as consciousness, pleasantness, attention, and agency, but most rely on interviews and subjective rating scales. Developers of rating scales try to design these scales so that different individuals perceive the anchors or points on the scale in the same way, which is a main assumption about such rating scales. While there are multiple rating scales for experiences that are somewhat easier to describe or define, such as pain or workload, there are many other subjective experiences that are not easy to define and involve an individual’s beliefs, expectations, and worldview, e.g., fear and joy, both of which can have a profound impact on our task performance and responses. Capturing the subjective experience is essential for understanding how users engage with products, treatments, and even technology. While objective data provides a solid foundation for evaluating efficacy, it is the subjective lens—shaped by individual beliefs, expectations, and even cultural or societal influences—that often drives our behavior and perceptions. Whether it’s grounding blankets or copper bracelets, understanding the complex interplay between mind and body reminds us that human experience is nuanced. All this contributes to the evolving challenges in human-centered design.
Can you recall a time when you completed a subjective rating scale but thought that it did not quite capture what you were experiencing? What did you wish it asked instead? The media hype behind extended reality (XR) technology promised it would revolutionize how we learn, play, and socialize. While there have been great strides to get mass adoption of this technology, it still has yet to reach market projections. This was one of the key messages I heard at AWE USA 2024, one of the largest spatial computing events in the world. Speaker after speaker kept talking about advancements in XR technology, new features, and software to make it easier for developers to create content. While all this is great, one thing that seemed to be an afterthought was: "What is the need they are trying to fill?" Before I go further, I am all for XR tech innovation, especially to improve training and human performance. However, I am also one of its biggest critics. Showing real value gained through appropriate technology applications is important to me. Finding a complementary match between tech and training needs requires a clear understanding of the use cases in which the tech will be applied. Over the past decade, my work has ensured that the right users get the right technology at the right time. Why are use cases important for XR adoption within the training domain? It all comes down to "knowing thy user." Use cases describe how a product, tool, or service will be used in real-world applications. They help set clear requirements that should guide technology development. They help identify all the steps a user will go through to achieve a goal, revealing opportunities for appropriate tech integration. They also help tech developers realize their initial ideas may not suit users. Therefore, they can provide the basis for pivoting to more effective solutions early in the design process, ultimately saving time, money, and resources during development. To generate use cases that accurately represent the expected user, work has to be done to discover key information. This includes:
Unfortunately, this part of user discovery is often skipped, but it is crucial to building effective, useful training tools. The "if you build it, they will come" attitude has proven unsuccessful in driving mass XR adoption for training. Why? The novelty of the technology soon wears off, and users are left wondering if they are truly benefiting from its use. As a technology developer, do you want your product to make changes in people's lives or collect dust on a shelf? Assuming it's the former, then take the time to develop clear use cases. I recently had a conversation with a five-year-old about Roy G Biv; not some guy named Roy, but the acronym for the colors of the visible spectrum (Red, Orange, Yellow, Green, Blue, Indigo, and Violet). This discussion grew from the classic child’s question, “Why is the sky blue”? The Roy G Biv conversation led to additional discussions about atmospheric backscattering, electromagnetic absorption versus reflection, and so forth. All these conversational threads shared a common origin and ending, “but why?” Eventually, cognitive fatigue got the best of me, and I relented by explaining that this is how the universe was created and that a higher authority should be consulted. This conversation stayed with me and caused me to speculate about the advantages that an AI-powered personal learning assistant might have or even accelerate a child’s education. Such an assistant would never become fatigued by a relentless series of questions and would always be available to pursue as many curiosity rabbit holes as the child desired. The potential advantages such a learning assistant could provide have spurred decades of discussion around this topic. What has been less widely discussed are the potential risks that AI-powered learning assistants might expose children to. The broad spectrum of risks includes everything from harmful misinformation to threats malicious cyber actors pose. In 2021, reports surfaced about the Alexa voice assistant encouraging a child to touch a penny to the exposed prongs of an electrical plug. Most people familiar with the fundamentals of safe electricity use will quickly recognize the extreme danger posed by such an action. Still, a child unaware of the potential threat might not realize the risk. A few weeks ago, Google AI advised users to put glue on pizza to prevent cheese from sliding off. This is another example of an ill-advised suggestion that would likely be humorously ignored by most adults familiar with the toxic potential of ingesting glue. Still, a young child might not recognize the danger. AI outputs are only as good as the data they ingest; no pun is intended. An older example of data poisoning a model was observed with Tay, the Microsoft chatbot, which became so vile and racist that it had to be taken offline after only 16 hours of being in service. These examples point to a few potential harms AI-powered learning assistants present if adults do not closely monitor their use.
The previous examples illustrate unintended model outputs leading to risk exposures, but what about model actions deliberately designed into the model? Deceptive design patterns (AKA dark patterns) describe product designs intended to benefit the designer but may harm the end user. A non-AI example might be defaulting to a monthly subscription rather than a one-time purchase of an item from an online store. An AI-powered learning assistant may remain in use with a particular user for several years, collecting immense amounts of highly sensitive data on that individual. This data will be precious to advertisers, criminals, political campaigns, and others. Seemingly innocuous interactions might be deceptive patterns designed to elicit highly personal and private information about that user for later resale. These previous examples derive from relatively benign motivations (selling someone something or getting them to vote a certain way). Still, it is essential also to consider the risk AI assistants pose if truly evil cyber threat actors gain control of them. The parents of a 10-month-old girl were horrified when they learned that a threat actor had breached their baby monitor and was actively watching their child. They discovered that the device had been compromised when they overheard a man screaming at their baby while she was sleeping in her crib. In 2023, 26,718 incidents of sextortion against children were reported. This crime usually involves convincing minors to commit sexual acts in front of a web camera and then using that compromising material to extort them. These reports involved relatively passive devices connected to the web. An AI-powered learning assistant designed to understand social and emotional cues can be easily repurposed to manipulate and exploit psychological and emotional vulnerabilities. There is a saying that there are no solutions, only tradeoffs. This concept especially applies to AI learning assistants. Such personalized assistants will undoubtedly usher in new and unanticipated benefits for children’s learning development worldwide and provide children across all socio-economic segments with unprecedented learning opportunities. However, UX and instructional designers must be mindful of the tradeoffs and carefully weigh the costs and the benefits when designing these technologies. This is a true crime story. Names have been removed to protect the innocent. One of my friends is an established author in the learning industry. Recently, she released her second book to great acclaim. One day, she checked her author page and to her shock and horror, an unauthorized book had appeared. "Sell Buy Create Relation Hip" was not written by her. Despite many complaints to Amazon, it was not taken down until very recently. However, this was not before three additional "fake" books showed up. Here's a snippet from one of them, "Learn to Use Time:" "My first job is to learn how to make the most of the time when the work is not yet full in order to enhance and broaden my knowledge as I will soon start working for a growing company. For instance, you should educate yourself on the political history of other nations as well as their economies, cultures, customs, and habits. Even if you have average intellect, you can develop the skill to dispel them if you have the information and know how to apply it to real-world situations. When I know I should do something but truly don't want to act, I think I have a lazy attitude. There is, in my opinion, no better phrase to describe this situation than what is said above. People who are lazy are those who lack the energy to work hard, who easily become depressed when faced with little challenges, and who often give up before attaining their objectives. Their motto is usually "I don't know how to do it" or "I can't do it," and they frequently encounter problems that have never been able to be addressed. Working hard and with a lot of enthusiasm will enable us to understand that there is no such thing as too much hard work. Can't do it, in my opinion, is just a lazy justification. People won't advance if they don't know what they want. Never lose up or express despair if you face challenges in your life or at work; instead, find ways to boost your confidence. physique and use it as a force for advancement." Who would do such a thing? A reasonable first reaction was to blame AI. After all, LLMs hallucinate, right? Besides, it fits the narrative about AI taking our jobs and leaving us with mass-produced, mindless garbage. However, I wasn't so sure. It's not typical for LLMs to write total nonsense unless you make them, and this was comically bad. I wanted to see whether I could solve this mystery using AI - fighting fire with fire, if you will. There are many tools that purport to detect AI in writing, and they are being used by some academic institutions to determine whether a student used a LLM to "cheat." Sometimes, they get it wrong, and false positives can have lasting consequences for students. I used this mystery as an opportunity to investigate for myself. OpenAI's ChatGPT enables you to create your own GPTs, and there are several devoted to detecting AI in writing. I ran the sample above through several of them. The most popular, Zero GPT Detector, said it was written by a human: "Based on the analysis, it's likely that the text provided was written by a human author. The linguistic cues such as variable sentence structure, nuanced expression of ideas, and a personal tone, including reflections and subjective opinions, support this conclusion. The text also contains context-specific advice and personal beliefs, which are more characteristic of human writing." Others leaned toward AI or ambivalence. Each gave its own justification based on linguistic style, syntax, and structure. However, none of them pointed out the obvious issue: the text makes zero sense. So much for the GPTs. Undaunted, I ran the snippet through demos of three AI detection websites: Scribbr, Quillbot, and Hive. The results were unanimous: there is no way AI is this dumb.
"The themes and style of the text might narrow down the possibilities to cultures that highly value education, have a formal approach to communication, and emphasize moral and ethical discussions about personal conduct. While these cultural aspects are prevalent in many Asian societies, they are not exclusive to them. However, given the linguistic features and content analysis, a background from an East Asian country like China, Korea, or Japan might be a plausible guess, but it could also potentially align with Eastern European backgrounds due to the emphasis on formal education and ethical labor." This was getting borderline racist, but I figured I'd throw everything I could at it. After incorporating word choice, literal translations, syntax and sentence structure, it came to the following conclusion: "Combining these linguistic cues with cultural context — emphasis on moral character, formal education, and a pragmatic approach to challenges — narrows the likely native languages to those where these elements are prominent. Given the formal style, emphasis on personal responsibility, and some specific types of errors, a native language such as Korean or Chinese appears plausible. These languages feature syntax and usage patterns that could lead to the types of errors and phrasings observed in the text, alongside cultural values that align with the themes discussed." So, we pull off the mask to find… a Korean and/or Chinese-speaking counterfeit scam artist! The emphasis on personal responsibility and moral character gave it away! Wait, what?
Obviously, this is not how forensic scientists determine authorship of mystery texts. We will never know whether these books were written by a lazy AI or are a product of an overseas underground fake book mill, or both. When it comes to making these determinations in the age of LLMs, we still have a lot of work to do. And if we're not careful, it's very easy to point the finger in the wrong direction. NFL organization report cards were released a couple months ago! For the past two years, the NFL Players Association (NFLPA) has surveyed active NFL players to assess various aspects of each NFL team’s organization. The purpose of this is to illuminate what the daily experience is like for players and their families on each team, to serve as a sort of “Free Agency Guide” for all players around the league (Tretter, 2024). In other words, players want to see what it’s like working for different organizations to help them decide where to work (and where to avoid). Luckily, these report cards are published for the public to see, and there were some interesting results. The categories that teams are graded on are: Treatment of Families, Food/Cafeteria, Nutritionist/Dietician, Locker Room, Training Room, Training Staff, Weight Room, Strength Coaches, Team Travel, Head Coach, and Team Owner. Teams are graded on these categories using a classic ten-point grading scale. Overall grades are weighted using a weighting scale that weighs more heavily on the grades of the Team Owner and Head Coach and less on the Dietician and treatment of Families (interesting weighting choice). You can find the full report card of each team here: https://nflpa.com/nfl-player-team-report-cards-2024. As an NFL fan, I find these report cards fascinating. I want to know how my team’s grades stack up with other teams. But it got me wondering: do these grades matter in terms of performance? Does the quality of the cafeteria food drive performance on the field? How do organizational benefits and workplace quality impact wins and losses? I took my football fan cap off, put on my research psychologist cap, and got to work to investigate these questions. I transformed letter grades to the ten-point grading scale and ran correlational analyses between organizational grades and NFL regular season win totals. Here are some of my findings.
These results are compelling. Overall, it appears that organizational benefits really do not impact wins and losses. However, an organization’s treatment of players’ families and the locker room quality do have an impact. If teams take care of players’ families, it may take a load off the player’s minds during games. If a player is worried about their family getting harassed by fans, not having a place to watch the game in comfort, and having to find a daycare away from the stadium for their kids, they may not perform at as high of a level. If the locker room is small and crowded and players don’t have a place to relax or recuperate between halves and before games, they may not be able to mentally and physically prepare to perform at their best during the game. Like a good data hygienist, I needed to explore the overall results a little more. When we plot out the comparison between organizations’ scores and their wins and losses, there is one particularly intriguing data point. It turns out that the team with the lowest average organizational score (unweighted) also had a very high number of wins. This team is none other than the 2024 Super Bowl Champions, Kansas City Chiefs. The Chiefs received F ratings for their Nutritionist/Dietician, Locker Room, and Training Staff and a staggering F- for their Ownership. Ouch. Additionally, they received only a D+ for their treatment of families (look out, Taylor Swift). The Super Bowl champion having the lowest overall rating seemed like a good reason to remove them from the analyses as an outlier. When we remove the Kansas City Chiefs from analyses, we find that there is a statistically significant relationship between NFL regular season wins and average organization grade (r = 0.41, p < 0.05), and the relationships strengthen between regular season wins and treatment of families (r = 0.38, p < 0.05) and locker room quality (r = 0.44, p < 0.05). It turns out that the quality of a workplace environment does impact team performance. However, this is among world-class athletes, who, for the most part, are getting paid millions of dollars every year. What does this relationship look like for other organizations? Over the past decade, we have seen a higher emphasis placed on glamourous workplace benefits, especially in the tech industry. Companies tout benefits ranging from professional chefs serving three meals daily, free laundry services, and even “pawternity leave” for new pet owners. But does this actually improve performance among their employees? Is the cost of providing glamorous benefits worth it for companies? Maybe companies should take a page out of the NFL’s book. Offer benefits that ensure employees’ families can be properly cared for and provide a functional work environment so employees can have physical and mental well-being in the office. The rest of it might just be fluff. That said, please, QIC, don’t take away my office snacks. References
Tretter, J. (2024, February 28). NFL team report cards 2024: For the players, by the players. NFL Players Association. https://nflpa.com/posts/nfl-team-report-cards-2024-for-the-players-by-the-players How many times have we been told to “put ourselves in someone else’s shoes” or “see things from the other person’s point of view?” According to the best-seller by Dale Carnegie, perspective taking is one of the principles for How to Win Friends and Influence People. It is not just negotiators or salespeople who have to practice this. We all try to do this when trying to understand our customers, staff, co-workers, bosses, friends, family, people we like, and people we don’t. Perspective taking happens when we imagine ourselves in the other person’s shoes. The thing is, our ability to take the other’s perspective relies on our imagination of what this other person is like and what we think we know about them, but this may not be accurate at all. Social psychology studies show that our reading of other people’s behaviors can be fraught with attribution bias, clouding our understanding of who the other person is. Some of our attempts at perspective taking can also be influenced by the stereotypes and biases we consciously or unconsciously have about different groups of people. When we have little information about the other person to go by, we may tend to overthink their intentions and read too much into things. When we have a lot of information to work with, we may still not select the correct information to focus on to understand what is most compelling for the other person at that particular time. How many well-meaning people have bought gifts that weren't really what the recipient wanted despite putting themselves in the other person’s shoes? I, for one, have done that for sure. It's not that there are no benefits to perspective taking at all. Perspective taking can help foster information elaboration that facilitates creativity in diverse teams and can help guard against automatic expressions of racial bias. There is also neuroscience research that suggests that exercises that included perspective taking can change the socio-affective and socio-cognitive brain networks in a positive way. However, putting ourselves in the other person’s shoes to understand them doesn’t always work because sometimes we really don’t know where the person is. A study showed that perspective taking did not necessarily lead to understanding the other person better, although it made the perspective taker feel more confident in their judgments. Interestingly, this confidence may hinder the perspective taker’s receptivity to learning and listening.
So while it’s good to put ourselves “in the other’s shoes” to understand them better, we need to recognize that our attempts to imagine what the other person is thinking and feeling can be obscured by our own bias when interpreting their behaviors, and/or the lack of accurate information. In addition to perspective taking, we should also just ask the other person about their views and listen unreservedly to them with an open mind.
What’s your perspective on this? Let's talk about error tolerance. I'm not talking about dealing with people who annoy you or what your parents practiced when they raised you (although similar principles probably apply). I'm talking about the flexibility of a system to continue to function in the presence of an error. Why would this be a good thing? Why do I want to use something possibly broken? In usability design, it's more about allowing users to achieve success, without being precise. Imagine if you misspelled something during your Google search and "Zero Search Results" appeared. Or if you were looking for a specific airline's website, but in return, you get a link to a mathematical definition. How quickly would you abandon the use of that tool? Has frustration kicked in? How would this impact businesses that depend on online traffic? (BTW, Google tried experimenting globally with Zero Search Results and, as you can imagine, angered many users). Error-tolerance can help your product be more usable.
What did we learn? If done well, error tolerance can keep the user happy when interacting with your product. It can help users succeed, even when they have never used your product. If error tolerance is overlooked, it can lead to catastrophic outcomes when the system fails. We also learned that planes can fly without doors.
The Women's Conference of Florida took place last month in Tampa, and I had the pleasure of attending. The last time I attended in 2019, I met and heard inspiring life stories from Abby Wambach, Monica Lewinsky, and Reshma Saujani. A lot has changed in the last four years, and I was curious to hear how the landscape had changed for women and the conversations around them. This year, I was excited to hear from Lauren Simmons, the wolfette of Wallstreet, Diane Obrist on leveraging our strengths, Mckinsey on the 9th annual report on Women at Work, and Katty Kay, US Special Correspondent for BBC. While I enjoyed several panels and discussions, the speaker that resonated most with me was Katty Kay. Already reading her newly released book, The Power Code, I was struck by the distinct shift in her messaging. It wasn't that women needed to change to have more positions of power but that the definition of power needed to change. She explained that traditionally, power was defined as 'power over,' a definition that most women are not inclined towards. However, when female leaders were asked what power meant to them, the definition was overwhelmingly 'power to,' a purpose-driven tool focused on what can be achieved. The research done by Katty and her co-author Claire Shipman indicates that the examination of best practices in the workplace and relationships at home can produce a less ego-driven world that is more impactful. In short, having women in power was great for society overall. She answered the question, what is the benefit of breaking the glass ceiling and achieving gender parity? All this made me think, how does having a female CEO shape the company? Here at QIC, we have all partaken in a stretch goals exercise to provide our leadership insights into what we would be excited to work on. Our CEO is using this information to guide the company's future to be inclusive of the goals of its employees. She uses her power to include her employees' aspirations in its vision. A more balanced, inclusive, and empathetic world where we leverage our strengths gives me hope for what's ahead. Last month QIC welcomed Dariel Tenf to the the team! Dariel is supporting QIC as a Data Scientist - Intern where he works with Research Psychologists and Human Factors Engineers to incorporate machine learning techniques into data processing, analysis, and visualization. Dariel holds an M.S. in Computer Engineering from the University of Central Florida with a specialization in Intelligent Systems and Machine Learning. Dariel’s work uses Natural Language Processing models in combination with more traditional machine learning models to gauge the success of a team-based effort based on communication and individual performance. Prior to joining QIC, Dariel designed a Toxic Comment Classifier, which was a machine learning system designed to read comments from social media and return whether the comment would be deemed as “toxic” based on the likelihood that the comment would cause someone to want to disengage from the conversation.
|
AuthorsThese posts are written or shared by QIC team members. We find this stuff interesting, exciting, and totally awesome! We hope you do too! Categories
All
Archives
November 2024
|