AI Proficiency: From Users to Builders

0:02Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show.

0:41Welcome to another episode of the Practical AI podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a principal AI and autonomy research engineer. How are you doing, Chris? I'm doing good today. How's it going? It's going great. I'm continually impressed by just the amazing AI work that's happening around, not only on the coast, but in kind of the, I guess, the heartland, the middle of the country where a lot of actually the large enterprises of our country are located. And we have a guest related to that today. But I do want to remind folks, we're also involved in a sponsor of the Midwest AI Summit, which is coming up October 15th in Indianapolis.

1:35Really cool event, Chris. You were there, you saw what was going on. There's tables where you can sit down with actual AI practitioners And, you know, rather than just hear a bunch of talks, you can actually get feedback on what you're doing, suggestions, design, etc. And hear great talks. So recommend people check it out. Midwest AI Summit. You can use the code practical AI 20 for 20 % off. But we have an amazing AI practitioner from close by my area, more towards Cincinnati. Matty. Mike Lewis was on a previous episode with us. We got a ton of great feedback on that episode. He's Chief AI Architect at Tier One Performance.

2:18Welcome, Mike. Thank you, Daniel. It's great to be here again. Yeah. Like I say, I had multiple people comment on just the utility and insights that they got out of our previous discussion. And just for context, because I think it's so impressive what you're doing and what you're involved with. Could you just set a little bit of context for kind of the types of projects that you work on? Like give an example of kind of some of the types of companies that you work with in relation to AI initiatives? Yeah, sure. So Tier 1 Performance is an end-to-end organizational performance and transformation partner.

2:58What does that mean? And we kind of help some of the most of the largest organizations in the world rethink transformation. I would add in, though, that we have a nontraditional approach and a multi-decade track record of success there. So, you know, the types of clients we'd work with day to day would be like Google, Eli Lilly, Takeda, Air Force. I mean, you know, just most of the Fortune 100s. And we see everything related to transformation. So obviously, AI would just represent a sliver of the types of challenges we encounter as an organization day to day. But that sliver is my whole life, you know.

3:44So I don't really see all the other stuff. I kind of get a glimpse into it every now and then because more and more AI touches everything. but we are sort of a transformation partner. And so you can just imagine how the phone starts ringing when a massive disruptive technology is birthed into our lives, whether we want it or not, right? Yeah, and a part of that, I think people are struggling with on the transformation side is there's just so many sources of information. There's so much noise around what you should be paying attention to, what people are actually doing, what people are saying they're doing, what is hitting the news.

4:27There's all sorts of a range of things. And obviously, a lot of that can be noise. And one of the things I've appreciated in interacting with you over time is that ability to kind of pick out some of that signal from the noise and develop some kind of structural ways of thinking about AI and transformation in an actual real-world organizational environment. In your kind of day-to-day, obviously you're hearing a lot of things both from customers that you're working with, the news cycle, the things from Anthropic, OpenAI, Hugging Face, whoever that's coming out with things. What does your process look like or how do you feel like your rhythms day-to-day help you kind of isolate some of that signal maybe kind of distill down some of what you should be paying attention to?

5:21Any suggestions? I think it's something on all of our minds, certainly on my mind. Yeah. Well, maybe I'm very good at this because I am just like the weirdo oddball guy who like stumbled into this at the perfect moment. I I mean, Daniel, you know this about me and your audience might remember. I was a portrait painter for almost 20 years. I mean, from when I graduated college till 2016, I painted portraits. And I owned a fine arts company and apprenticed a bunch of artists. And I only got into this because I'd heard about the Dali model from OpenAI and applied to become one of the commercial arts beta testers.

6:07And I lucked into that. I got an email one night at dinner and I was like, oh, wow, I just got access to this thing no one had heard of at the time. And so and I knew a little bit of like Python and coding or whatever. So enough to like fumble around and get things installed. And I felt in love with language models. I mean, almost instantly. I just couldn't believe all the new things you can make computers do. And, you know, this is before we really thought about them as chat bots or whatever. In my mind, it was like something you plug into code to just manage edge case weirdness to, you know, pick a path and make your app keep working.

6:41And, you know, context models were minuscule back there. I'm running things locally on my gaming computer, you know. And then, you know, medical records companies start calling and, you know, and I build this book of business and tier one acquires my company. Here's the only reason that's worth unpacking is to say like, I don't have this like muddy, long history in the industry where I had opinions about everything or like a solid way of thinking about it. It was all just fun toys for me. And then I think the way I've made it through business my whole life, I've owned maybe a dozen businesses, all small, but is I've always just had a firm rule is like I do not concern myself with things outside of my sphere of influence.

7:25So, you know, there may be this big thing going on at the edge of the AI sphere. But the reality is, like, if I if I don't have influence or if it doesn't impact, like, the way I'm interacting with my clients, if it's not, then I just ignore it. These things are not going away. Like, I know for sure whatever this is, it will be here probably forever. And in my experience, I'm a much better worker. I mean, I can fool almost anyone into thinking I'm a smart guy when I'm using AI and I'm aligning context to like whatever my work project is. I mean, it just works so much better using these tools. And I just have that conviction.

8:03That's probably true for most people. So, Daniel, my trick is really only focusing on like what directly impacts like the things going on in my day to day. And at tier one, what that is, is a workforce enablement performance, you know, basically not worrying so much about tools or whatever, but just outcomes and performance. Yeah, that makes a lot of sense. And one of those things, you know, last time we talked, I looked up the date, it was like 2024, right? Sometime in 2024, obviously, a lot has changed. It's impacting all of us in different ways. At that time, we had a discussion about what's, I think, doable, scalable, etc.

8:52etc. You've, I know, spent a bit more time recently thinking about this concept of AI proficiency. And I know myself, you know, obviously, if you're doing AI things, you're searching them, you get targeted online. I'm all the time seeing these like ads for whatever it is, Harvard School, like what the engineering or what becoming AI proficient in your business or whatever the course or thing is. And so there's a lot of people trying to understand this because of, like you say, the wide reaching impact across the workforce. So next week, our company's doing an AI accelerator with one of our customers.

9:42I know that there's going to be a whole range just from discussions from of quote AI proficiency. Um, but general, like other than thinking, well, I know that there's that range, right. Um, it's another step to go and say, well, in light of that range of proficiencies, what do I need to change for different proficiencies? How How do I advance people through proficiencies? Do they even need to advance through different levels of AI proficiency? What is enough for some people? What is too much? Right. So I don't know if you want to help us launch into this topic. Maybe just starting with a little bit of definition of what we mean by proficiency would help us just so that we're all talking about the same thing.

10:33If you don't mind, Daniel, I'd like to even go a step further back. Yeah. And just give you a sense for how and why I even started thinking about this. So if inside of our, if at tier one, we have, you know, most of the largest, we and three or four other companies are helping the largest companies in the world navigate this disruption. And we also see the same story playing out inside of every company. Like, I mean, I could tell you 95 % of the companies, I could walk through the timeline and you'd say, yep, I saw that at fill in the blank, fill in the blank, fill in the blank. They've made all the same mistakes.

11:09That, it's kind of watching that happen, sort of helpless at the periphery. You know, as a consultant, you don't have a lot of control, but, you know, and you don't have perfect ideas either, you know. So everyone's just sort of watching this thing unfold and like, oh, I really thought that was going to work, but that didn't stick. And the big thing that I focused on early on was developing tools that help people learn and get good with AI. So I developed our, you know, solution archetypes framework where we kind of like tried to pin down. We talked about that on the last episode. And then I built our super user habits index, which is basically it's like, hey, these are the 24 habits we see super users displaying.

11:47And then I built a coaching tool that people could use. And in my mind, it was like, I'm going to put all this together and turn people into super users. And guess what? It kind of didn't really happen. For some people it did, but I couldn't help but feel like most of them, it was going to happen anyway. And at the same time I'm watching, um, I'm at the same time, I'm sort of reading all the Substack articles on it, watching things unfold at companies, looking at my work history, you know, thinking through the people, looking at the statistics we create or, you know, stuff in the, in the literature.

12:23And then, um, I think the breakthrough through a moment for me. I was invited to a bachelor party with some young guys who used to work with me. I, you know, never again, but I brought 200 pages of research citations and synthesis. So this is not 200 pages of research. It's 200 pages of like the name of a paper, a URL, and maybe a sentence about why it might be relevant to my work just to read during that week. During the bachelor party. I thought I was probably going to need to pull away a lot. These are all younger guys. And I did, I read it all. And there were just, there were, there were just patterns in there that answered a lot of questions for me.

13:04And, and the breakthrough moment with all of that just sort of happening in within like 30 to 60 days, that stew forming. I read a Substack article, it was Peter Yang. I'm kind of skimming my screen here for the details on it, but basically it was called Your New Job is to Onboard AI Agents, How AI Native Companies Actually Operate. And they just had this way of thinking through the L0 through L3. So I didn't invent that. I read it in that L0 through L3 classification. So L0, L1, L2, and L3. And the L0s are the people who aren't using it. And the L3s are the ones who are just making a difference across the entire enterprise.

13:49And we all know those. And then they labeled L1. And L2, though, like knocked me off my chair almost. When they talked about the concept of a non-technical builder. And they just gave a name to a thing that I just thought, wow, this is something we need to really focus on. Is like the idea of a non-technical builder. We're all focusing on converting L0s, people who are disengaged. And we'll unpack what that is. I think we need to, because there's so much important stuff there. And they're trying to get people from L0 to L1. And L1 is what everyone knows. It's the user. It's the one who's like, I couldn't live without this.

14:27And they kind of feel like it ends here. And then that leap, you know, the big jump from like, okay, someone who like uses AI to, wait a minute. No, this is a non-technical builder. This is a person building durable solutions that emulate the company's work in an uncanny way. that jump is actually the one that now I sort of run around evangelizing. Like this is actually where you need to focus your energy. But so, so Daniel, I just, I guess I unpacked all that to say, like for me, I had the, like the stew, I guess is the way to say it of like, you know, the, the guy who like didn't have a dog in the fight, built a company on accident.

15:05It was bought by tier one. And now I'm here, I'm inside of all these companies trying to help them make sense of this thing. And then, you know, all these external inputs sort of just clashed. And so now we, this is the framework we push and advertise and help ask people to think through. If you've been listening to the show over the past few months, you realize how transformative agentic AI is. Whether it's Cloud Code or Hermes Agent or your own custom software that you're deploying for operational efficiencies or as a product into the market, these systems are transformative and it's where the industry is going.

15:46But agents have a lot of risk associated with them. They have agency. They take action within your environment, within your infrastructure, within your systems. And that's why security teams and governance teams view it as risky to deploy these agents within high-impact environments in industries like manufacturing, logistics, financial services, in the public sector. And that's why I've personally been spending my time working on a product called Prediction Guard with a great team of AI engineers. Prediction Guard is an AI control plane that you host within your own environment, whether that's on-prem, hybrid, or in your cloud VPC, even air-gapped scenarios.

16:30You can set custom AI policies or those that are aligned with NIST and OWASP standards, and those are then enforced automatically across every handshake, across every agent, whether that's being powered by your own self-hosted models or models in your cloud environment like those from Azure or AWS Bedrock. And all of that telemetry goes back to your monitoring and observability system. I would love for you all to check out what we're doing. Schedule a demo and a call with our team to find out more at predictionguard.com slash practical AI. Like I say, schedule a call today. We're happy to show you how this works and why we think it is so beneficial to the market.

17:18people already have this deployed in production, powering transformative agentic systems across a number of industries. So check us out at predictionguard.com slash practical AI. That's predictionguard.com slash practical AI. So Mike, I wanted to ask you a question. As you are thinking about how you get the employees of one of your client companies, you know, moving from the L zeros up to the L threes, um, want to throw another variable in. And that is, is the attitudes that people have, which are, you know, very varied in terms of like, there's, there's always people who are excited. As you mentioned before, there are people that will naturally want to go to that L one, L two and up to L three, but there are also people that are going to push back hard on that.

18:10And so when you're, when you're kind of coming in and you're trying to start your process of helping them achieve that, how do you bring along the L0s that don't really want to move or even the ones that have done a little bit, but they're also pushing back on that? Maybe they fear for their jobs or something like that. So there's that kind of L0s and maybe some L1s. How do you get over that process as you're trying to bring everybody into this to kind of level up through those? Any thoughts? Can you kind of throw that in as you're talking about your process? Yeah. I love that you're asking me about L0s because the reality is we don't try to move an L0.

18:53Here's the thing. The article I mentioned, the Substack article, I disagreed with 98 % of what I read in that article. I just latched onto the concept of an L2, a non-technical builder. That name resonated with me. The L0s in that article, at least at Ramp, the way I interpreted what I read, they were saying that, hey, if you're an L0 at Ramp, of course, it's a software company. You know, if you're an L0 at Ramp, that's grounds for dismissal. You have until this date for our system to assess you as an L1. And man, that just broke my heart because it's kind of like, OK, wait a minute. Let's back up.

19:30Let's back up. Name another technology that we would say that's true. But can you imagine you just go company wide? Hey, by the way, everyone, we want you to geek out about smart sheets and be great at smart sheets by the end of this year or, you know, AWS or, you know, like just it's ridiculous. This is just, it's another technology like that. Yet, I think a lot of executives are kind of scrambling and panicking and they're saying, hey, guess what? We're going to be AI enabled. We're going to be activated. Everyone in our company needs to speak AI by X date or there's not a place for you here.

20:04and so first of all you remember i read 200 pages of science and a lot of it was like it was just about learning it wasn't even about ai and and and so um threat framing actually does slow down adoption so you know if you put if you back someone into a corner and say learn this or there's not room for you here they will not learn it as well as if you came to them and said let's figure this out together if it interests you and i will acknowledge that there are some positions where you absolutely need an AI enabled individual. It's true. Like, you know, and so there are cases where an L0 is not going to work in that role.

20:39But I think we're going to realize in 10 years, looking back at all this, that it's just not true in every role. And I think the one that hit me the hardest is my wife, probably the smartest person I know in the world. My wife, smartest person I know. And she has no use for AI in her job. She's a nurse. And she kind of just, I know what she does. She pokes people in the arm with needles and some other things, and there's medical records. Maybe someday it'll be useful to her. It's just really not right now. It would be in the way. And I started doing research on the concept of L0, people who are disengaged.

21:19And I realized there are multiple buckets. So when we talk about an L0, it's not a thing. So an L0 is not just someone who doesn't use AI. I found five major categories. So one is the performative user. They've been doing this. They've been nodding their head in all the trainings, the meetings. They've logged into the thing. They've submitted a chat and they've gone about their job. Like, you know, like they're not really getting and and hey, they're still getting their job done. There's the disinterested. I think the moment when this hit me the hardest was when I was looking at all the research and I I saw a chart that showed the adoption rates of personal computers in the office space back when PCs were first released and AI.

22:02And there's almost the exact same adoption curve over the same period of time. It looks like it's happening at the same rate. And when I saw that, I thought, how silly is it to think back to that? No one is in a panic because the people in their company haven't figured out personal computers now. It took care of itself because everyone wound up with one. And the same thing is happening with AI. Everyone went home and just created a chat GPT account or whatever. It's free. This is not true with your CRM. This is not true with smart. Like everyone didn't run home and create a Smartsheet account because, you know, like they just got to have one.

22:39And so, you know, but the disinterested are still detached there. Okay, here's another bucket. The too busy. I could give you 10 names at tier one. We almost don't want them running around trying to figure out AI because they are just so critical in their role. They're so good at what they do. It's just like, it's like, it's a don't break it or don't touch it. It ain't broke type role. And, you know, who cares if they're using AI or not? And I think that is also really hard for the people who are here in communication. Use AI or you're out of here. And then they can look around and see the people who like, very often it's some of the highest performers in the company who don't really care about it because they're in a rhythm.

23:16So those I kind of push past. But the next two, these are still buckets in L0. These are people who aren't using it. really matter. And there's something you can learn from these people. And I think when you say like, hey, you're going to be doing this, you're going to be using AI at this company, they will just zip their lips and they will nod their head and they'll say yes. But here's the thing. First of all, we have the job fearful. And Chris, I think you mentioned this, you might have mentioned this earlier, the job fearful. What do we do? What do we do about the job fearful? First of all, very often these people don't really understand how the tool work And they're under the impression that if they use AI to do their work, it's going to learn how to do what they do and replace them.

23:54And that's almost never true. I mean, just if you kind of know how these things work, I mean, you know, maybe self-driving cars. I could think of a few examples of like the AI is actually learning from watching us, but my vote doesn't really impact whether or not it's going to replace that profession someday, you know. And so I think with the job fearful, there's an opportunity to help them see like, hey, you might be in the middle of a self-fulfilling prophecy here. Like if you continue to push back against this, you might lose your job. But it won't be because the AI replaced you, because Joe, who's willing to use AI, will replace you, you know.

24:28So that's one way to address an L0. But another one, and this is actually six out of 10 people that we talk to, and I've seen this number and research hover around 60 % in more than one place, are the quality disappointed. This is where I think every executive's ears should turn on. They should lean forward in their seat and they should realize, do not dismiss these people. If they are resistant to AI, there could be some very valid reasons why. And I met a person at a company who, her complaint was, it's not good enough for what I have to produce for my work. And she kind of showed me and she's just like, humans do this better.

25:06And it was a marketing role. It was copy generation. It was some other stuff. and she was just right. And it may not stay true forever, but no one had ever actually just, no AI expert had ever just sat and listened to her. And I walked away from that conversation with just a brand new perspective on how much value there is to mine from the group of people who say, it's not ready to do this work yet, or either it's not, or they need better tools or better training, but you can't really do that until you listen first. And so, and I also don't think about it like a ladder. I don't think an L1 is better than an L0, and I don't think an L2 is better than an L1.

25:45Actually, I don't see it that way. I just see it as kind of a division within an organization for how different ways of thinking about the toolkit. And maybe one follow-up question on that. I'm wondering, you mentioned maybe part of the stumbling block here is the quality disillusion, or however you put it. You know, getting into these tools, I sometimes I also wonder about the way in which like I would put myself into the category of the AI people like, yeah, I'm building a product, right? How much of this is us providing AI in a way that is just another thing that they have to add, like a different tool that they have to add in their workflow versus something that actually works in their workflow.

26:36And what I mean by that is, you know, if I'm to give an example, you know, we're working with an organization now they have teams of translators, we have essentially a team of digital agents that helps translate with it with an agent harness, and they use existing tools, they go through the, you know, drafting and quality checking and post editing, they do these cycles. But we don't actually say, hey, go to this different interface that you've never used and do things the AI way. The sort of agents work and then stuff pops up in the translation management system that they already use. Right. And so what they see is, yes, there's change, but it's actually allowing them to do the thing that they want to do with the tools that they're using in a way that actually they want to do that more.

27:37Now, that may not always be possible. Like there may need to be a shift of like how people, you know, the interfaces that people use to do that. But I don't know. Do you have any experience with that? Like how much of this is we're asking people to use yet another tool and they've already got tools that they like, whether they're AI or not. Right. And how much of it is the actual AI output? I don't know if you have any sense of that. No, I hear, I think I hear the sort of the spirit of the question. And so this is where I land, you know, when it comes to, you know, how do, who do we train to do what and what do we give them and how do we, you know, and honestly, at this point, I just feel like that is part of the noise to me in all the conversations.