Podcast: When AI Becomes a TA

Specifically, Martin asked students in his operations management class to interact with a chatbot he built to help reinforce course concepts, like a real teaching assistant would. Students liked it—a success that might pique the interest of business leaders looking for a low-stakes use case to adapt for their teams.

“All of us are still not aware of all the ways it can actually help. And I think this is the real impact of this project here,” Martin said.

On this episode of The Insightful Leader, we hear about Martin’s AI TA and what leaders can learn from his experiment. Plus, a student tells us what it was like to use the bot.

Podcast Transcript

LAURA PAVIN: In October last year, New York City Mayor Eric Adams made a big announcement that he said would help the business community.

MAYOR ADAMS: We’re launching the next phase of MyCity with the MyCity business site and ChatBot. For the first time business owners and entrepreneurs will be able to direct their questions to an AI Powered Chatbot.

PAVIN: The city said it fed the AI more than 2,000 NYC Business webpages and articles, presumably making it an expert on all things local law and municipal policy. The intent was to make accessing this kind of information easier for businesses so, you know, they wouldn’t have to wait on hold or scour city websites for answers.

But some recent reporting showed that the AI didn’t always give the right information. Sometimes it told businesses things that were completely false. Like, it said it was legal to fire an employee who complained about sexual harassment or who didn’t disclose a pregnancy. To be fair, the site is pretty clear on the fact that it’s a beta version and might be incorrect. But still. It was enough to get computer scientists and ethicists to raise alarms.

The site, as I’m recording this, is still up though.

[Music]

Welcome to The Insightful Leader. I’m Laura Pavin. I don’t know about you, but something I’ve heard over and over again from thought leaders is, “Get your hands dirty with generative AI!” “Play around with it!” “Don’t fear the inevitable!” But for me, I think Mayor Adams’s attempt shows that there is a way to kinda mess this up.

And it makes you wonder, is there a way to experiment with this technology responsibly? Like, you know, without all the high-stakes ramifications?

On today’s show, there’s a professor here at Kellogg who’s been thinking pretty deeply about all this stuff. And in fact, he’s found a way to bring ChatGPT into his classroom to help students learn rather than do all their work for them.

It’s an interesting look at the value AI could offer when we think a little bit deeper about what purpose we want it to serve.

That’s next.

PAVIN: Sébastien Martin is a professor here at Kellogg. And at the beginning of last quarter, he was getting ready to teach a course on operations management.

SÉBASTIEN MARTIN: Operations management is about how everything is done in the company, how a company is organized. So for example, if it’s manufacturing, how do you actually make stuff? If it’s a tech company, how do you set up all the algorithms, apps, websites in order to serve your customer? How people are organized, who do you hire?

PAVIN: All the nuts and bolts stuff. And as he was doing his standard teaching prep, he had an epiphany: maybe AI could help.

MARTIN: I realized that the technology, the generative AI technology, was almost there so that it could be used potentially in a productive way for teaching and learning. And so I decided to just try it, and I built essentially an AI TA, an AI teaching assistant.

[Music cue]

PAVIN: An AI teaching assistant. I know, it sounds kind of like science fiction. But when you see it, it looks normal enough. It’s a chatbot that looks kind of similar to ChatGPT, because it’s powered by OpenAI, which makes ChatGPT. The landing page is a lot of white space and a text box at the bottom where you can write prompts. The unique thing about Martin’s version, is that it knows a whole lot about Operations Management 430.

MARTIN: I always [laughs], before every class I write examples of prompts the students can ask.

Martin would often prompt the bot in class so students could see just how specific this thing’s knowledge was. One of the things he asked it was to write a Haiku about Operations Management 430. Here’s what that turned up:

Operations dance,

Lean tools in harmony sway,

Knowledge blossoms wide.

MARTIN: Right now we’re covering lean operations, so the AI knows that.

PAVIN: But it’s not just a party-trick tool. It has a job to do.

MARTIN: You can view it as a modified Chat GPT, that gives it a more precise personality and has access also to a lot of specific information about the class. It’s GPT with a job, if you want.

PAVIN: To do this job, Martin fed it the class syllabus, homework assignments, and announcements. He transcribed his lectures and fed it those. All the same stuff a human TA would need. And then he gave it instructions, like you would give to a human. So no code; just plain words. For example:

MARTIN: “We have completed classes 1 to 11, and the next class is class 12.”

PAVIN: He says this is how it knows where in the agenda the class is.

Martin also told the bot exactly who it was and how it should act. For example, he wrote, “you are an AI TA, and a TA is generally a young PhD student. Avoid bullet points when possible and use straightforward, informal language with short answers inviting a dialogue. Don’t hesitate to make jokes!”

And so, with all of this knowledge, the bot can do its job. You can ask it to remind you what was covered in class five, and it’ll tell you bottlenecks were covered in that class. And it’ll share the recording links and the slides, in case you want to practice. It could also quickly create practice quizzes and tell you what you got right and wrong and help clarify things you might be confused about.

It’s all very impressive and felt seamless. I wanted to know if students felt the same. The ones this was supposed to help. Did they feel like they learned more because of the bot? Or was it just kind of a cool toy?

I got in touch with one student.

LAUREN PINDERSKI: My name’s Lauren Pinderski, and I go to Kellogg in the evening weekend program, but I go to classes during the week.

PAVIN: Lauren is a current Kellogg MBA student. She also works full time doing forecasting for a pharmaceuticals company. Before Martin’s class, Lauren said she really hadn’t used Chat GPT or AI much at all because it just wasn’t something she thought to use. But when Martin unveiled the AI TA in his class and she started using it, she was like, “hey, this is really cool.”

PINDERSKI: I was super surprised the first time I used it because I would answer one of its questions and then it would reply to me by saying what I said but better and then adding things onto it. So literally it was shocking how much of a human-like conversation it was. I wasn’t expecting that.

PAVIN: Lauren said Martin hadn’t just made the AI available to his students, he intentionally wove it into the curriculum. Like, for example, he would have them complete assignments prior to each lecture that required them to chat with the AI.

PINDERSKI: So it would ask us about class concepts, can you apply this to something at your current or previous role? Or it would ask us if we were in that unit, quantitative questions, and we were able to either answer the question—and it would say, yes, you got it right, or it would say, not quite, look at this.

PAVIN: One time, it told her she forgot to take the square root of something in an answer she gave. And it explained big picture ideas to her.

PINDERSKI: I would use it and say, “Hey, this is my understanding of this concept. Can you fill in any blanks that I’m missing here?” So it just gave me a more holistic view of what I was learning, and I thought that was really helpful.

[Music]

PAVIN: Students like Lauren had a lot of good things to say about the bot when Martin surveyed them. It helped them grasp class concepts better. They liked the quizzes. And it seemed to help international students because it could speak multiple languages, and, like, that feels like a eureka moment, right? Like, it’s really hard to master a new language, let alone learn really complex topics in it. That’s a really valuable discovery.

All in all, the bot did a good job helping students and not getting in the way. And that’s a win. For the students, for their teachers who want them to learn, and for all of us other people looking for a success story to point to so we have some use case to reference in the event we want to implement it on our teams.

So what do we think worked about Martin’s experiment? Well, for one thing, there wasn’t a lot of pressure around it. Martin didn’t give students a big, scary letter grade based on how they used it. He simply asked that they use it and that they send a copy of their interactions along so he could see that they tried.

Lauren again.

PINDERSKI: The AI pre-class work that we had to do, it was all pretty much pass/fail. I think that they were worth two points each, and I think as long as you made an effort with the assignment, you would get the full credit.

PAVIN: One student said the low-stakes approach helped them focus on absorbing the material rather than worrying if they had an incorrect answer or if they sounded silly. This was a bot, after all, not a human who will judge you.

So what could Martin’s experience tell us about getting our hands dirty with generative AI? Here’s Martin’s perspective on it:

MARTIN: Sometimes it’s about nudging and just creating a tiny process around it and extra information. And that’s all you need to really unlock. And so it’s really about just showing at this point, the AI thing just arrived and all of us are still not aware of all the ways it can actually help. And I think this is the real impact of this project here.

PAVIN: In other words, this is about low-stakes trial and error. You don’t want an NYC bot situation on your hands.

So as you consider implementing generative AI into your organization, it’s probably smart to think small. At least at first. Identify some needs your team has, and think through some low-stakes ways of using generative AI to help out. Then survey everyone who has been using it: Is it doing what you’d hoped? What else could it be doing?

Now, before you get too excited, I do think it’s important to say here incorporating the AI TA hasn’t saved Martin time. Not by a long shot. This was a ton of work to put together because he had to fine-tune it and update it constantly. He had to tell it everything he told students in class, for which there wasn’t an existing transcription. But the nice thing about the AI was that it didn’t necessarily require that he know a specific coding language, or even the first thing about coding at all. He just gave it really specific instructions, the way he might give them to a human TA. And I think that opens the door for a lot of leaders who might not have the tech chops, but who absolutely know how to give really specific instructions.

If you’re trying to find ways to bring AI to your company or your team, Martin’s use case offers one example of how it can work and catch on with your team.

Martin’s student Lauren, for instance, has become somewhat of an AI ambassador at her pharmaceutical company, encouraging her colleagues to give Chat GPT a try.

PINDERSKI: One of my bosses at my job today, she was talking to me about, Hey, I’m trying to convince our senior vice president to do something that she doesn’t think is necessarily a good idea, but I think it’d be really helpful. And I was like, if you want help putting your thoughts into very good words, why don’t you just put it in Chat GPT and it’ll help you write it out? And she was like, I don’t know, I have never done that before. And I was like, lemme just show you how it works. So I pulled it up and I showed her and she’s like, oh my gosh, I didn’t realize I could do that.

[Music fades up]

CREDITS

PAVIN: This episode of The Insightful Leader was written by Andrew Meriwether. It was produced and edited by Laura Pavin, Jessica Love, Fred Schmalz, Maja Kos, and Blake Goble. It was mixed by Andrew Meriwether. Special thanks to Sébastien Martin. Want more The Insightful Leader episodes? You can find us on iTunes, Spotify, or our website: insight.kellogg.northwestern.edu. We’ll be back in a couple weeks with another episode of The Insightful Leader Podcast.