962: Wharton Prof Ethan Mollick on Why Your AI Strategy Is Already Obsolete

0:00Jon Krohn:What if the employees, secretly saving 70 % of their time with AI, don't want anyone to know about it and they have good reason to hide? Welcome to the Super Data Science Podcast. I'm your host, Jon Krohn. It is not hyperbole to say that today's guest, Ethan Mollick, is one of the most sought-after AI experts in the world. Ethan is an associate professor at the University of Pennsylvania's prestigious Warden Business School, where he co-directs the Generative AI Lab. His book, Co-Intelligence, is amongst the most popular AI books of all time, and nearly 400 ,000 people subscribe to his One Useful Thing newsletter.

0:33Jon Krohn:Today's episode on Enterprise AI is short, yes, but be prepared. It's so information-dense, you very well may need to slow down your player's playback speed to enjoy it. All right, let's go.

0:48Jon Krohn:Ethan, Co-Intelligence, your best-selling book from the spring of last year, focuses on collaborating with AI. has the rapid evolution of a Gentic framework since the book's publication led to any major changes in your thinking? So for example, if machines are so autonomous, do they eventually get to a point where we're not really collaborating with them at all? Or is that autonomy a natural complement?

1:12Ethan Mollick:Well, so there's now a bunch of modes of coordinating with machines that are different. So with co-intelligence, when I wrote the book initially, AI made a lot of mistakes. It wasn't and autonomous anyway, and you had to kind of do all the work yourself. That still is actually a very valuable way of working with machines. But now even that's become a little complicated, because there's two ways of doing this. One is interactively, and talk to a chat bot, push back and forth. But with thinking models and agentic models taking more and more time to do work, the models increasingly, I assign a task to the AI, I use my expertise to decide what that task is, I evaluate the results, correct my approach, and if that still doesn't work, I do it on my own.

1:46Ethan Mollick:And that is a really good form of co-intelligence now. And then as you're pointing out, we are starting to see some real autonomy from these machines for some tasks. So I think people will increasingly be picking among those sets of modes or just doing things alone. So things I do alone, things I do co-intelligently, things I assign to AI and assign the work, and things that happen autonomously.

2:06Jon Krohn:That's a great list. Very concisely done. You clearly spend a lot of time thinking about this. You mentioned in your talk at ScaleUp AI today your recent Cybernetic Teammate paper involving a field experiment with 776 professionals at Procter & Gamble. And it demonstrates that AI often functions like a teammate as opposed to being a tool. In light of this, how should executives, how should our listeners rethink team structures and management?

2:35Ethan Mollick:We have no answers to that question right now. I think that this is, so part of what worries me a little bit about the state of AI is the capabilities of the systems are quite high and people are not really using them. And everyone wants someone else to answer the question for them. So if you look at why American firms in particular did so well throughout the 20th century, 40 % or 20 % of the, 20 to 40 % of the extra competitive value of American firms has come from better management. And there was a willingness to experiment with management techniques and approaches that generally led to the dominance of American firms in the 20th century.

3:09Ethan Mollick:I feel like that's something a lot of companies have given up on. They either go to outside consultants to do the work or trust SaaS vendors to tell them how to run their company. And I think that companies that start to experiment are going to be in the best possible shape. So we're already seeing some versions of breaking down traditional barriers, like let's pull senior IT people out of the IT department and have them sit next to a subject matter expert and vibe code applications together. It starts to really change how things operate.

3:33Jon Krohn:Yeah, this ties actually into something. it ties into some of your contrarian stances. You touched on it a little bit there in your last response. For example, you've argued against consultant-led, external consultant-led AI implementations, as well as other popular enterprise AI approaches like retrieval augmented generation, RAG. Given your contrarian stance, you might have unique ideas about where the best opportunities lie for AI in enterprises today.

4:01Ethan Mollick:So first, my qualifier. I actually think there are rooms for consultants and room for RAG. Consultants especially can help organizations make the change. They want to make processes, things like that, but I think that the change does have to come from within to some extent. And the reason why I've been sort of pushing against RAG solutions, not RAG as a concept, but because the first and easiest thing to build is a talk to our documents chatbot, right? And I think there's space for this, but the thing is, is that if you built a talk to our documents chatbot when I was warning you not to do that a year and a half ago, you now have a mediocre talk to our documents chatbot that cost you a lot of money to make and is now easily beaten by an off-the-shelf model, right?

4:38Ethan Mollick:Similarly, if you had a large consultant built application, things have shifted very quickly. So I think that the goal is not so much that there's one bad application, but it's that you need to be at least doing some really ambitious work. And it's important to do that work ambitiously internally and not just look at vendors. Vendors play a role, but you need to have some internal effort going on as well.

4:58Jon Krohn:It sounds like based on your answer to the most recent question, as well as the one before it, that a key idea here is that enterprises, organizations, need to be looking at the opportunities internally, figuring out how their IT specialists or AI specialists can be embedded with the people that will be impacted by the work. That sounds like a key part.

5:19Ethan Mollick:Sort of. I think you're putting too much emphasis on IT. So I say you need three things in your organization, leadership, lab, and crowd. So leadership are the C-level people in the organization who are actually deciding what incentives will encourage people to use AI, giving a vision of what AI is going to do, deciding what processes need to be altered or changed as a result. They hire consultants to help them, more power to them, but they need to be making those decisions. The crowd is everyone in your organization using these tools. You want access to advanced chatbots because it's easy for people to figure out use cases in their own area of expertise.

5:48Ethan Mollick:It's very expensive to the outside person figure that out for you. So you need them experimenting. And what do they do when they come with the result that works? That's when they go to the lab. So the lab is your group of people inside your organization who are not all from IT. In fact, most of them come from the fact that there'll be a couple percent of people in your company who are just really good at AI. Bring those into the lab. They're going to be doing three things at the lab. They'll be turning around those ideas right away. Someone comes with a prompt that works well. Let's test it, refine it, and ship it out to the world.

6:13Ethan Mollick:They will be thinking about benchmarking, and they'll be thinking about whether you buy or build advanced products.

6:20Jon Krohn:Yeah, this buy versus build scenario. you've talked about how if you have these big expensive external consultant engagements then you know you're very likely a year later to be to have spent that money poorly because you know someone else comes along a frontier lab comes up with a model that can do a lot of what that external consultancy had built anyway so how do you advise on how we should be making our buy versus

6:47Ethan Mollick:build considerations so I think it's not just buy and build there's also rent as an option, which is buy temporarily. I think you need to be thinking about how core something is to your business, how much extra expertise you don't have internally. So you need a really good coach or mentor. You don't have a mentoring program that's very good. That might be a good thing to use externally. Then there's core functions of your business, CRM, other sets of stuff where you might want to use AI capabilities from vendors to do that. But I would ask my vendors to be much more transparent than they've been in the past.

7:16Ethan Mollick:Because your vendors generally don't have their own AI models. They are absolutely using someone else's models. What models are they using? What prompts are they using? Prompts, if you don't have an ability to audit them, can be easily manipulated in a way that makes results seem good that aren't. So you need a lot more transparency in the buy decision than you used to have.

7:33Jon Krohn:Yeah, it's interesting because you're asking them to give away some of the secret sauce, but I guess there's a balance that they need to strike if they want to get the deal with you anyway.

7:42Ethan Mollick:I think we should be pushing much harder for that secret sauce because we know enough that small changes to prompting can vastly change sycophancy, how people in your organization feel things operate. If you're using older models and not updating them, why would you let the launder models for you when some of the models are flawed? I think one way or another, if your secret sauce is we're using a prompt and we're using Gemini, that is not secret sauce. If that's your competitive advantage, you are in big trouble. For sure.

8:09Jon Krohn:All right, so we've talked about organizations now. Let's zoom into the individual level. A couple of years ago, you identified secret cyborgs, individuals and organizations who leverage AI for time savings of 20 % to 70 % on many tasks while maintaining or increasing the quality of their work product. Do you think that this 20 % to 70 % has continued to accelerate in the past two years since you originally started writing about secret cyborgs?

8:35Ethan Mollick:And we have some evidence on this. Over 50 % of Americans have said they used AI at work, right? And probably more actually have. They self-report that on a fifth of tasks that they use AI for, that they are seeing a three times performance improvement. That's the self-report. Whether that's true or not is hard to know. What's slowing that down from organizations is we don't have the process needed to make that, like, what do you do? You're running agile development, someone gets all their code done right away. Like, what's the point of their stand-up? What's the point of, how do you work sprint planning around that?

9:04Ethan Mollick:How do you think, like, what do we do about that stuff? Or they're just not telling you they're using it because they're just misincentivized.

9:10Jon Krohn:Yeah, yeah, yeah. Yeah. So if these people, if these secret cyborgs are in your organization, what can we be doing to surface them and to be taking advantage of what they're doing? Maybe have what they're doing be taught, be less secret and be taught to other people, have a whole army of cyborgs.

9:27Ethan Mollick:Well, this is where the leadership and lab come in. So your secret cyborgs come out of your crowd, the people in your organization doing things. Your leadership needs to incentivize people to actually tell you this. If people think that they're going to be fired or punished or other people will be fired because they're showing productivity gains, they're just not going to show you. If they're working 90 % less, they're not going to want to give that up for free. So leadership needs to think about the incentive plan that puts this into place. And then you need the lab because you need somewhere for these people to go.

9:51Ethan Mollick:And then, you know, either there's a reward to actually work in the lab or else to say, hey, I've got this prompt that kind of works and saves me five hours a day. Could you make it good and get it out to everybody? So it's not just a one component piece. You need the other pieces.

10:03Jon Krohn:All right. Thank you. Final question for you. It's always tough to look into your crystal ball and see into the future. But given your position in the market and all the research that you've done, you might be able to see into a crystal ball better than most of us. So what do you advise? What can we be doing today to prepare ourselves for the AI capabilities that will be available in the coming years?

10:28Ethan Mollick:So I think my advice tends to be very similar, which is use these models a lot. Use frontier models to figure out what they do. I think you can bet pretty reasonably that they will get better and cheaper, right? And that means a bunch of thresholds get crossed, but you need to use advanced models. You need to use them for actual work stuff. You need to use them trying to get them to work rather than being skeptical about whether they work. And you'll learn the shape of the frontier as you go. So I don't think it's as much as like to keep up with the news, but it's if the models update on their own, pick one of the big players, use their stuff a lot, and you'll figure it out.

10:59Jon Krohn:Perfect. Professor Mollick, thank you so much for taking the time with us today.

11:02Ethan Mollick:Thanks for having me.

11:32Jon Krohn:areas of expertise, and the lab refines promising ideas and benchmark solutions. He talked about why over 50 % of Americans report using AI at work, claiming 3x performance improvement on about a fifth of their AI-assisted tasks. And he gave us his advice for preparing for AI's future, that's use frontier models extensively for real work, assume they'll get better and cheaper, and you'll learn the shape of the frontier as you go. I hope you enjoyed this awesome conversation to be sure not to miss any of our exciting upcoming episodes. Subscribe to this podcast if you haven't already. But most importantly, I hope you'll just keep on listening.

12:07Jon Krohn:Until next time, keep on rocking it out there. And I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.