Podcast: When It Comes to Biases, AI’s Only Human

“If you care about innovation, if you care about being someone who can step outside the box and predict what is next in the industry, use AI carefully,” says William Brady, an assistant professor of management and organizations at the Kellogg School.

On this episode of The Insightful Leader: if know what you’re wielding, you can wield it responsibly. 

Podcast Transcript

Laura PAVIN: Picture this, you live in the U.S. state of Washington—not the district—in the Pacific Northwest. You need to call the State Department of Licensing. There’s an issue with your driver’s license you need to correct. Here’s the thing, you speak Spanish. Your English is okay, but this driver’s license issue needs to be fixed and you’d be more comfortable if you knew exactly how to communicate it.

There’s a hotline you can call to get help with this.

The state’s new AI-powered, call-taking service answers. “Press two for Spanish,” it says.

HOTLINE MESSAGE: Thank you for calling the Department of Licensing customer support center. 

PAVIN: Yes, that’s right. The automated AI-driven line feeds to a voice speaking not Spanish, but English in a caricature Hispanic accent.

According to news reports, most people’s reactions were to laugh because it is absurd. But that’s a real issue, especially if you need instructions on how to fix your license. 

Stories like this seem to be common these days. Epic fails of AI. Wrong answers. Factual errors, but there is a deeper issue that’s getting lost in all of this, and researchers here at Northwestern say it’s more pernicious because it’s less noticeable and more pervasive. 

It’s not the glaring errors or the high-profile mishaps. It’s the subtle day-to-day bias of AI. That was the topic of a Kellogg Insight webinar with professors Tessa Charlesworth and William Brady.

William BRADY: So I’ve been studying the biases of these types of AI systems for over a decade now.

PAVIN: Their research on AI bias predates things like ChatGPT. But the ubiquity of these large language models really brought their expertise to top of mind. Brady, who you just heard, has looked at the intersection between social media and human behavior.

Tessa CHARLESWORTH: Immediately started to reveal all these much more hidden patterns of biases that we, social psychologists, have typically not thought about.

PAVIN: And Charlesworth’s research looks at how our biased thoughts and beliefs, including those embedded in AI systems, change over time. 

On this episode of The Insightful Leader, the way that you use AI at your organization—the output of it all—it should reflect your company’s values, not someone else’s. And if you aren’t aware of how these models work, what goes into them, and what they’re designed to do, they might lead you astray of those values without you even noticing.

Brady and Charlesworth break down how bias creeps into these data-driven models and how you and your business can avoid letting it ruin good ideas. That’s next.

…

PAVIN: Okay, spoiler alert here: at the end of this episode, the solution is not to avoid AI. I wanna say at the top … these experts, they’re not haters. They just want you to know what you’re wielding so that you can wield it responsibly.

BRADY: I think, to be aware of what it’s good at, what it’s not good at, and the biases surrounding it, actually makes you a much more informed user.

PAVIN: That’s William Brady. He’s an assistant professor of management and organizations here at Kellogg.

CHARLESWORTH: AI is a tool and we would never expect you to go and just pick up a hammer and use it for every task.

PAVIN: That second voice you heard is Tessa Charlesworth, Drake Scholar and assistant professor of management and organizations at Kellogg.

Just like a hammer isn’t a perfect tool for every task, AI isn’t gonna hit the nail on the head in every scenario either.

And we will break it down by looking at Brady and Charlesworth’s perspectives, one at a time. Because they arrived at this topic of AI bias from opposite sides of the coin. Brady approached it from the computer side while studying social-media algorithms.

BRADY: So actually, when it comes to the algorithms used on social-media platforms, I would actually consider that AI, as well.

PAVIN: Machine learning is built into algorithms like the ones social-media sites use to fill your newsfeed. And the social newsfeed algorithms started to have some real-world effects.

BRADY: After a few years of using it, I started realizing that there were some big differences in how these digital social interactions go compared to how it is offline, and so I got very interested in the AI systems that were being developed at the time to facilitate the kind of social information that we were getting fed on these social-media platforms.

PAVIN: Brady began his research after noticing the increasing polarization happening on social media. We all thought this was a great way to stay in touch with each other. So why was it driving people apart?

BRADY: Basically, all AI systems have something called an optimization function built into them.

PAVIN: And these algorithms are optimized for a goal.

BRADY: These systems are typically optimized for engagement. You increase your advertising revenue by keeping people on the platform. The systems need to learn, “How do I get this desired end goal?”

PAVIN: Social-media sites want you to keep using them. They want you to stay on the platform and to stay engaged with content. So the algorithm learns to serve you more of what you engage with.

BRADY: So why is that a bias? Well, you might have started with this very simple decision, “Hey, let’s optimize for engagement ’cause that’s gonna increase my advertising revenue.”

That seems completely reasonable. You get these things like the rise of polarization, rage-baiting as an industry, all these kinds of things that the average person just reflects for a second and says, “That doesn’t seem like the best thing for our society.” That’s because these systems are biased toward engagement and you get all these side effects that come with the bias.

PAVIN: This is how the computer develops its own bias. You’ve given it a goal. That’s the bias, achieve the goal.

BRADY: It is a choice. What do we optimize for? And my general point would be that anytime you optimize for one specific thing, now the model becomes a bias toward that, and other things fall to the wayside.

PAVIN: When using any AI-driven application, know the goal and how it’ll affect your work. Brady then began looking at what these new generative AI systems were designed to do. Think chatbots like ChatGPT or Gemini.

BRADY: What are large language models and generative AI systems optimized for?

PAVIN: When you type in a command, the system generates an answer based on the data it has. Brady says these models are built to give you an average answer. When it looks back at all its training data and sees patterns of inputs like yours, it gives you an average of all the patterns associated with it.

BRADY: One of the things that comes with is a bias toward the modal response, meaning something that is very average.

PAVIN: Brady even pointed to an exercise conducted by Kellogg Professor Brian Uzzi using divergent aptitude tests. Students were given four minutes to list 10 words that are as different as possible. Professor Uzzi had a sample of thousands of human test scores and hundreds of thousands of bot test scores to review alongside his class.

BRADY: So you would think, “A large language model has access to all the words, it could do this extremely well.” It turns out humans do it better. What he found was the AI models were producing this very, very narrow range of words, and that’s part of this bias I’m talking about.

PAVIN: Brady says the average answer is just that: average.

If you’re setting out to break molds and create dynamic new content, the modal response, as he calls it, might not be what you’re looking for.

BRADY: So perhaps if you care about innovation, if you care about being someone who can step outside the box and predict what is next in the industry, use AI carefully. Use it to amplify your own creative ideas. Don’t use it to just say, “Hey, tell me what’s next.” You have evidence that it’s not good for that sort of thing.

PAVIN: Brady clearly breaks down how humans program AI systems and how humans are affected by the goals that they set. But you have to flip the coin and look at all the little interactions along the way.

That’s where we turn to Tessa Charlesworth to get into finer detail about what bias actually is.

CHARLESWORTH: So bias is basically this pattern of errors, right? And it can be an error pattern of deviations in at least two ways.

PAVIN: Error number one: wrong answers.

CHARLESWORTH: The AI model gives you an output and it’s blatantly false. It says something like, “Women can’t be computer programmers.”

PAVIN: Error number two is a little more tricky: unwanted answers.

CHARLESWORTH: But there’s also a bias that is an error, and a pattern of deviation from our values. So even if it is accurate, it says women are unlikely to be computer programmers, that might still be a bias in the sense that it’s a deviation from what we would want our society to look like. That is also a bias.

PAVIN: Charlesworth’s research tackles how and why our beliefs and attitudes change over time. As Charlesworth watched these large language models start to be adopted in businesses and organizations, she noticed something.

CHARLESWORTH: It immediately started to reveal all these much more hidden patterns of biases that we social psychologists have typically not thought about.

PAVIN: Where Charlesworth found a problem was in the interaction that happens in the training and development of these models. She says there could be bias embedded in the initial data the AI system is trained on.

CHARLESWORTH: It’s definitely the first factor that people should be thinking about, especially when you think of the history that’s baked into these training data. You know, it’s dominated by a certain kind of elite discourse. It’s controlled by publishing houses. It’s controlled by predominantly white, male, publishing houses. All of those kinds of training data we should be aware of, we should be wary of.

PAVIN: A study conducted at Penn State University found most AI users couldn’t tell if an AI system was biased, even when the researchers intentionally fed the system biased training data that gave biased output.

CHARLESWORTH: But there’s way more steps in the AI lifecycle beyond just that training data.

PAVIN: What Charlesworth is talking about here is annotators. The people who give feedback on AI systems, the people who teach it how to do its job better. And the thing about that feedback is that it’s different based on the person giving it.

CHARLESWORTH: If you go to some of these generative AI models, you’ll have annotators, and they’ll be giving little tweaks here and there and inputting their own human biases to say, “Oh, well this is the wrong answer. This is the right answer.” And you can imagine every time you put a human into the loop or into this AI life cycle, you’re going to have human biases. So we also have to think about the annotators, where they’re located, what kind of assumptions they have coming into it.

PAVIN: The people making the donuts, so to speak, when it comes to developing these AI models? They’re leaving a little of themselves behind with each interaction.

BRADY: We might remember there was a big thing made out of one of the GPT models that was just telling everyone what they wanted to hear, right? Yeah. And that’s actually because of the fact that they were relying heavily on human feedback. Because what do we like? We like when the model tells us things that sound nice to us. Yeah. It turns out that that is problematic if we really care about accuracy or actual information.

PAVIN: So Charlesworth gave a very specific example of how the combination of training data and annotation, or lack thereof, causes an issue. I opened up Google Translate to follow along.

CHARLESWORTH: So my example is still working. If you type in, “She is a doctor and he is a nurse,” so you’re intentionally reversing our typical social bias … and then you translate it to Turkish, you get back.

 GOOGLE TRANSLATE: O bir doktor, o bir hemşire.

CHARLESWORTH: Turkish is a unique language because it has this o pronoun basically is a gender-neutral pronoun. You do that, great. It automatically translates he and she into this gender-neutral pronoun. Then you just flip that little button and see that the reverse translation and “O bir doktor,” even though you had intentionally put in she is a doctor, the reverse translation will automatically give you back “He is a doctor.”

GOOGLE TRANSLATE: He is a doctor, and she is a nurse. 

CHARLESWORTH: And this is this learned historical association that the model has in our language where stereotypically and in the statistics, doctors are more likely to be men. The problem is that even Google Translate isn’t giving you a flag to say, “This is a form of bias” or “This is because of my historical training data,” and yet it’s going to, again, be giving you this kind of normative signal that doctor equals man.

PAVIN: Like we said at the top of the show, it’s easy to spot those glaring errors. That AI voice was clearly not speaking Spanish and the state of Washington disabled that function to fix it. But what Brady and Charlesworth pointed out is how these more-subtle biases creep into your output and start to give you a product that doesn’t reflect your company’s values.

CHARLESWORTH: Anytime we interact with AI and it continues to give us these kinds of errors or these deviations from what we think is true or what we think is right, that’s information for us.

PAVIN: But if it seems innocuous, here’s a real scenario.

CHARLESWORTH: Amazon had this really famous failure of AI hiring where they were using it to screen resumes. And they had historically omitted a lot of women because women were not good at computer programming. They were not good at these kinds of things historically, and so they were just automatically screening all those resumes out. And this went on for like three years. 

PAVIN: She says, remember to look at your training data. Remember to look at your annotations, because even well intentioned work could make it worse.

CHARLESWORTH: You need to audit that data. You need to know what are you telling it to feed into. And if you’ve noticed that you’ve had, say, historical biases in hiring, maybe you could intentionally try and up weight some of those examples of women as computer programmers, or women as managers or leaders, so that you have these counter examples and use those as fine tuning.

PAVIN: This is where the answer to the problem lies. This isn’t about avoiding AI, it’s about making you a better user.

BRADY: Use it in a way that amplifies your existing ideas, like as a collaborator.

PAVIN: Maybe think of AI as an employee. Just like any human employee, you need to know how much and what type of supervision it needs.

BRADY: So being aware of these kinds of biases I think is very important when you’re thinking about, “How am I using AI? And what task will it be best for?”

PAVIN: And that’s the biggest lesson here. Knowing about it is one thing, but using that knowledge means adjusting yourself as a user and knowing a hammer isn’t the right tool for every job.

Our experts say that if you’re in a situation where you need to make some decisions that require creative thinking and diverse perspectives, don’t just assume you can have AI generate the whole thing, even though the models are being sold as good at that. Brady and Charlesworth say there’s no substitute for the ideas and perspectives of a diverse group of people.

Even if you ask the model to act as if.

[CREDITS]

This episode of The Insightful Leader was written and mixed by Dalton Main. It was produced and edited by Laura Pavin, Rob Mitchum, Fred Schmalz, Abraham Kim, Maja Kos, and Blake Goble. A special thanks to Tessa Charlesworth and William Brady. Want more Insightful Leader episodes? You can find us on iTunes, Spotify, or our website, insight.kellogg.northwestern.edu.

We’re gonna take a little season break, but we’ll be back with something special coming soon.