The Gap Between AI Spending and AI Value

0:00It is widely reported that a gap has emerged between enterprise spending on AI and the durable value captured from that spend. Individual employees have enthusiastically adopted coding assistants and chatbots, yet those gains do not seem to be transforming businesses at an organizational level. One of the most important questions in the tech industry today is understanding why AI is not yet delivering returns that match the investment, and what separates the small number of enterprises succeeding from the many that are not. Scale AI is known for supplying the human-labeled data behind many frontier models.

0:39It now also builds AI applications and agents for large enterprises. That combination of working alongside frontier labs and inside enterprise deployments gives the company a rare view of why enterprise AI may be stalling. Emily Hsu is the head of enterprise AI at Scale AI and previously spent over a decade at Google. In this episode, Emily joins Kevin Ball to discuss the three layers where enterprise AI breaks down, why frontier model benchmarks miss what enterprises actually need, the data foundation problem, how the most successful companies combine internal domain expertise with outside AI specialists, and more.

1:22Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

1:57Emily, welcome to the show. Thank you so much for inviting me, Kevin. It's my pleasure to be here. Yeah, I'm really excited to dig in with you on this. Let's start with a bit of your background. So can you kind of give us a little bit of your career to date and how you ended up at Scale? Yeah, I absolutely love to share that. A lot of people were saying like my career is kind of a reverse of many people. So currently, I'm the head of enterprise AI and Scale AI. It's kind of a company, a startup company, but it's not like as small as a startup anymore, that people kind of note from the perspective of supplying the data to many of the frontier labs and being an active participation to the Gen AI large-language model to where we are.

2:39So it's very interesting work from both understanding the customer requirements on the enterprise side, as well as driving the innovations to research for pushing enterprise AI to land with revenues. Then before that, I was actually working for a really large company, Google, one of the hyperscalers, and also the Frontier Labs, two-in-one. I worked there for 11 years, been very hands-on engineers. I'm also early member of Google Brain team. So I was a researcher, hands-on model training, publishing papers. And I was engineering lead for several of the cloud AI product, Vertex product, tuning, Gen AI tuning, evaluation agent engine with my team members who really start with the concept, the idea and see the market needs for it.

3:25and then put our hands down there, build up the product and bring it to the customer. So it's a really fruitful experience working with the team, put up a team to bring this to the market to see what the enterprise adoption for this platform products are. So that's my experience at Google for about 11 years. But actually, before I joined Google, I was a professor. I got my tenure from Vanderbilt University. So my research is kind of a combination of like applying a lot of the theoretical analysis from optimization and statistical learning in understanding sort of networking system, a network and large-scale distribute system, and with the application to several domains, separate physical system, healthcare system.

4:08After I got tenure from Vanderbilt, I spent my sabbatical years at Google and decided to stay with the industry. So that's kind of like me. Yeah. That's awesome. Well, I'd love to dig into a few different pieces of that, but let's start also with a quick overview of scale because you mentioned scale. When I was doing the research for this call, I initially was like, oh, I've heard of scale. They do data labeling, right? But y 'all do a lot more than data labeling now, don't you? What is the span of what scale is tackling? Absolutely. This is a great question and segue. So people really know about scale.

4:40And this is also how scale actually incorporated as a company is really to bring data to machine learning labs and the machine learning companies as a data provider, starting from providing data for image, like vision perception models. And later on, the market is actually expanding from the vision models and then getting to language models. So this is where we actually get into partnership with OpenAI in very early days, providing human annotations, human feedback data to make GPT where they are now. So that's a core business. But meanwhile, at scale, and also I think the broader market also realized that beyond the foundation model capabilities, the landing of the value of all these models really depends on building reliable applications and agents that powers the applications.

5:30So that's actually another big chunk of business for scale. And within that sort of the application business, we'll have two business units, depending on who are the customers we're speaking to. So I'm currently leading the enterprise AI side where I'm in the enterprise business unit, where our customers are private sectors, 500 sort of enterprises. And we'll also have another business unit where it's public sector facing US and the global governments. But the goal is kind of like really building applications and solutions on top of leveraging the intellectual capabilities for models to serve the business needs.

6:11Yeah. Well, and that makes a ton of sense. I think we're all trying to figure out where are the value layers in this new AI world. And it definitely looks like it keeps moving up the stack. Well, I'd love to really dive into the enterprise side. I think there's a lot of interesting things there. And, you know, I work at a relatively small startup. The adoption challenges in a startup world are very different than they are in the enterprise world. I think you've actually written a bunch about the different failure modes of enterprise AI and things like that. So let's maybe start with that. Like what is different about adopting AI in a large enterprise and what are the classic failure modes?

6:48Yeah, absolutely. So this is very insightful question. And there's actually many kind of directions people are thinking about it. Right. So maybe what we can do is I would think about like to start with is it's a problem of two sides. On one side is on the machine learning AI capability side. Like, you know, people usually say, think about from sort of AI perspective. On the other side is actually from the platform system perspective. So in order to have system solution application that truly serves the enterprise customer needs, both sides have to meet with each other and be successful. And then on top of the technology layer, that is also adoption layer, which is the management side.

7:34It also touches the people side of the enterprise usage. So it involves education, leadership, routing, and all the stuff. So I just want to sort of maybe anchor this discussion on this people versus technology. And within technology, there's the AI part of it. There's also the system part of it. So maybe we can start with on the AI side of it a little bit. So if you think about AI side, the market, the community have a lot of exposure to what foundation model Frontier Labs are doing. I come from Google. Actually, I'm part of the Gemini team. I've been going through Gemini 1.0 all the way to 2.5.

8:13So if I look from the insider lens when we develop Gemini, right? So if you look at it, the development team actually have very limited exposure to what enterprise needs are. And this is exactly what I feel, you know, even from Cloud Ad, enterprise part of business of Google. Fundamentally, every model iteration, which we call data flywheel, understanding where the model gaps are and how to contribute data, train it. the whole engineering practice is pretty much owned by the engineering team who actually do not have the business exposure. And that is reflected as, right? So people are very benchmark driven.

8:56They have very focused goals, say, hey, now we actually have different capability pillars, math, reasoning, code, conversational, multi-turn safety. And then if you look at that, The enterprise part is becoming a very narrow pillar in all the frontier labs leaderboard progressively into 2025. So which means the model knows a lot of the human intellectual capability and knowledge, many of them is reflected by our flagship benchmark humanity last exam. But in terms of concretely, what is the professional requirement? What is the operational workflow? What are the policies that is actually super critical in the enterprise domain?

9:43There's no evaluation criteria. There's no intuition knowledge within the people who are actually in the mobile. So then from AI perspective, the fundamental capability has a gap from what it means. Well, it sounds like not only a capability gap, but even a goal or target gap. Exactly. It's not clear where it needs to evolve. Absolutely. So we need to set up the North Star. And the enterprise, it's a single word, but it's not as simple as math and code. Math and code are the true kind of like, we have a lot of breakthrough in terms of intellectual capability advancement in the past. But if you think about math and code are the things that majority of the people who hands on the code truly understand.

10:32right there are also there are also domains in which it's if not trivial much easier to define what correct looks like so you can build an optimization function and have an iteration loop and things like that absolutely right this is where the reinforced learning with verifiable reward is actually bringing a lot of capability advance in the last i think last year let's make the time flies. Yeah. So this is - I know, right? A year these days is like five years. Exactly. I think it was starting from the early of 2025, where we're thinking about reasoning models. So now I'm thinking about this like one year and a half.

11:11Yeah. And you're absolutely right. Having a precise definition on what is correct, what is right, actually bring a lot of benefit in the post-training process with R. But then in the enterprise domain, and back to what are discussing about. If you think about it, it's a single word, but it's even hard to define what it is. There's so many verticals, right? And then let's say healthcare. Now within healthcare, to think about there's primary providers, medical centers, there's insurance companies, there's pharmaceutical life science companies. Even with health providers, there's big academic medical centers, there's community healthcare service.

11:51So they have very diverse needs regarding what they're going to use AI for and what they regard as acceptable AI service are. So that's kind of like the complexity of the enterprise world. And the complexity is not very well captured and presented to the lab's developers. That's one of the root cores for where the capability gaps are. Yeah, that makes a lot of sense. So tech layer, there's some gaps, absolutely. But let's not leave it there. Let's look at the other layers too. The other layers, yeah. So that's kind of like starting from the foundation model, but then we also talk about the system side, right?

12:34So not just the models. When you develop an application, it doesn't matter big or small solutions or reusable applications agents, you need to have it to be integrated with the enterprise context environment. You need to understand what data are, how to get yourself authenticated. When we say yourself is the agent themselves, to get authenticated to the right data. And also to understand for all the enterprise services, their APIs, their data schemas behind it, what do they mean, right? With different table names, with different value names, how do we actually understand the context? A lot of cases people talk about context engineering.

13:18How we actually learn those tribe knowledge within the enterprise and bring that knowledge to be part of the context into the agent. And that is actually a non-trivial problem, right? So because it's not like all the information is right there presented to you. the information. Absolutely. Like I'm dealing with exactly this in a startup environment where we have like, you know, a handful of systems and a few different databases. You look at an enterprise that's been around for decades and has hundreds of thousands of people, like this is a mammoth problem. Totally. And then you hit another point in terms of the pain point, what we see is when we think about enterprise and then we talk about like, this is a new era of AI waves, But then let's look at in the past two decades, when people are talking about digitalization, like everything from paper to digital form, digital asset, there's a lot of technical depth actually in that wave.

14:19Because during that wave, the entity who actually are the information receivers are human. Humans are extremely flexible, and they are able to tolerate, like, you know, give me a PDF, give me a handwritten note. It's not like perfectly digitized in a way that is to the high standard, but I can still get it. This is actually like the problem we get, for example, from the engagement with Mayo Clinic, right? We think right now all the electronic back-to-record system are digitized, right? it confirms a uniform standard called FHIR. But the reality is there are always patients who are not actually within the network.

15:02They go through another system. They come to you with very complicated medical history. They're not going to give you the standardized, normalized electronic medical records. They will give you hundreds of pages of their historical medical records printed in PDF. So how people can read it, but then in order to actually ingest the data from different modality with the technical depth from the last two decades, fragmented, received, processed by human beings in the past, now you have to plug in an agent that is powered by foundation of large launched models, then the gap is actually emerged from there.

15:44Yeah. Yeah, absolutely. Like every shortcoming that we all have that a human has just said, oh, you know, that's not perfect, but I'll paper over it. Now we have to go back and revisit and say, well, can an agent paper over it? Maybe not. What do we do with that? Absolutely. That's kind of like one of the things we experience, how we actually integrate an application, and agent into enterprise environment really address all these gaps from fragmented information, tribe knowledge. So all the things become the barrier actually prevent accelerated deployment of AI solutions in enterprise. You're building agents that can write code, summarize documents, and automate workflows, but they're missing one thing, awareness of the world around them.

16:30X-Weather combines enterprise-grade weather intelligence with agent-ready APIs, natural language capabilities, and an MCP server built for tools like CLAWD, Codex, Copilot, and modern IDEs, so your agents can adapt workflows, automate responses, and make better decisions based on real-world conditions. Backed by Visilla, whose instruments fly on NASA missions to Mars, X-Weather delivers trusted data and unique insights that go beyond conditions to actual impact, from real-time lightning strikes to road surface forecasts. Start with 15 ,000 free API calls every month and pay only for what you use as you grow.

17:03Your full weather stack for developers by developers. Start building for free today at xweather.com. Your customer lives in the real world and it's messy. Intermittent connections, mid-onboarding drop-offs, edge cases on devices you've never tested. Mobile apps reflect reality in a way no other surface does. Yet, from an engineering perspective, they're the hardest to understand. It's common for mobile engineers to see green backend dashboards, normal error rates, no crashes, but inevitably somewhere there's a frustrated user watching your app spin. After a few seconds, they'll lose patience, close the app, and turn their attention somewhere else.

17:41They may never come back. The worst part? Most observability tools never see any of it. BitRift, on the other hand, captures 100 % of mobile data, unsampled and in real time, so it's immediately queryable by engineers and AI agents. It's mobile observability built for the real world. Try BitDrift today. bitdrift.io slash signup. This episode of Software Engineering Daily is brought to you by Warp Build. AI is writing more code than ever, which means GitHub Actions is running more than ever. Your GitHub Actions bill is now a function of how much AI code you generate. And every engineer knows the feeling.

18:15You push a commit, and then you wait. Warp Build makes GitHub Actions twice as fast at half the cost with a one-line change to your workflow. Linux, macOS, and Windows Runners in Warp Builds Cloud or your own, enterprise-ready, SOC 2 Type 2 attested, and trusted by teams like Sky from Comcast, Bitcoin, and Braintrust AI. Get started with$50 in free credits at warpbuild.com. Well, and this kind of then ties into this last piece, which is the human layer. Like what needs to adapt in terms of how we're engaging in work? Absolutely. So this is like at the lower layer of the technical issues, but the solution to the technical problems really lies in the human's hands.

18:59And this is where, from organization perspective, the leadership commitment, the leadership, because all these motions, all these AI motions, shows us it's not going to be successful if there's not a clear incentive on what is the benefit, what is the return, is the ROI. If the leadership does not recognize the value of the ROI, and then when we actually get into all these barriers, then people could easily sort of, will not have a clear actually success criteria. People could easily pause at impressive demo and not pushing forward because there's no incentive to show the fundamental return and the impact, economical impact to the solution, right?

19:46So when there is a, it's just drop it in the middle of nowhere and then not get into productionization. So from this perspective, the most important thing is the alignment at a high level on a leadership level. what is the we're aligned on the north dot successful metric what is the expected output of it and then driven by that we would have a clear technical solutions technical roadmap to say now in order to achieve this high level north dot success metric what other things we need to do what is the path you know what's the low-hunging fruit we need to do in the first step to get into the system foundation to be developed?

20:33And what's the next step to get to the user engagement? How we actually collect the feedback continuously from the usage to improve the system from the agents. So those are the things that actually defining the driving force to be successful. Yeah. Having leadership agreeing on where to go definitely seems important. Another thing that I'm pondering about, like one of the things that I've seen and I've heard a lot of people struggling with is, at least in the coding space, which is the one that I and I think much of our audience are most familiar with, these tools are really focused on individual productivity.

21:08Yes. But if you just drop a bunch of individual productivity improvements into an organization without rethinking about your organization, you often end up just bottlenecked on the next layer that didn't speed up the same amount. And you don't see nearly the level of organizational improvement that you expect based on the individual. How is that showing up in these large enterprises in terms of change management or reorganization or rethinking of how we organize ourselves? Yeah, this is such a great point. It just kind of really resonates what we see in the field as well. If you go to a lot of the enterprises and you ask them the AI adoptions, the answer could really go two directions depending on how people think about AI adoptions.

21:51If we say the people who are using cloud code, you know, GPT, and you're using all these individual AI tools, your adoption is amazingly high. Everybody is using it, right? But they're not using it in the context of like exactly what you captured in contributing in a coordinated enterprise workflow. They just use it to improve individual productivities. And then to do that, there's actually two major problems, kind of very related. The first one is there could be a lot of IP leakage. Because when you interact with an individual tool, all the queries you have, all the questions you have, there are a lot of enterprise confidential information.

22:36And even if it's not proprietary data, there's also the way you ask questions and then ask follow-up questions, review a lot of the judgment and the implicit knowledge within people's mine. And those information is actually leaked into the frontier labs. So that's one side of the problem. But on the other side of it is actually these are very valuable enterprise digital assets, the IPs. And when people are actually getting used to just interacting with the individual productivity tools, all this digital asset, all this implicit knowledge are actually not captured and shared across the organization.

23:24This is exactly the point you're talking about. In many cases, when you work on a project and then when you have experience and the context, the skills, all this expert knowledge from an enterprise perspective, this information is to be consolidated across multiple team members and then into a coherent knowledge base so that the next generation of the people who are working on the project will be inheriting the knowledge, the experience being accumulated from the previous conversation. So those are the things that are actually missing by building this. So that kind of brings us to the flip side.

24:06We've talked about where this breaks down, where the gaps are in different ways. What does success look like? I know you all recently published this report and you said, I think it was 6 % of large enterprises are really right now succeeding at integrating AI and seeing the benefits. So what are the patterns? What are the ways that they get around these gaps? How does that end up looking? Yeah, so when we have the 6 % sort of the reports, the first thing we think about is how we define success, right? So in this domain, defining success is actually not easy as well because as we said, if you ask them about AI adoptions, they could say some shallow adoptions.

24:45and that's not success, right? So the first thing what we did is we really think about what is the success criterias are. And then for that, we basically have a pretty rigorous way to say that. It's kind of like how people among all the projects people have been engaged in, how many of them actually go from pilot into production and then meaningfully integrated with the workflow. and meaningfully integrated with the current enterprise workflow that is measured by very concrete usage metrics. So those are kind of like the things we're looking at how we think about the success in a very concise way.

25:27We actually look at the characteristics to see what are the characteristics the 6 % winners actually share. We are not able to attribute to say, hey, these are the factors that are causal factors. If you did this, you'll be successful. We're kind of like doing machine learning. We try to capture what are the shared attributes about all this 6%. Have you ever been able to make a true causal argument in a human system? Like that's very, very hard. But yeah, I think that is useful, right? The pattern matching is what we're all looking for. It's like, okay, what are the patterns that I can try this?

26:02And maybe that will help move my organization closer. Yes, that's exactly the sentiment that I can, you know, what we're doing. So there are a lot of findings in the report, but the three things kind of really strongly reflected from all the statistics and the study we have. The first one actually is not surprising, is the data foundation. And this is actually we've talked about how data is fragmented, multimodal data, connecting data actually to understand the data access policy. What are the governance or how we actually collect the feedbacks from the field? All this data infrastructure foundation.

26:39is actually a prior, kind of like very strongly related with the success of AI, search deployment success. No surprise, right? It's not surprising, but I feel like this is a thing that comes up and there's memes that have gone around, right? Of like, oh, we have this huge data mess. We could solve that. Or we could just sprinkle AI on top and hopefully it will work. And it all comes back to, no, you actually have to solve your data mess. It's a chicken egg problem, right? If you don't have the data, the AI will not do anything. But then when the AI is actually getting to the enterprise in the right way, then AI could be helping you with, for example, one of the key technologies we're actually providing to the market is, say, entity resolution.

27:20So you have different tables. The keys, the way you name corporate entities are different in different tables. In the past, this is a very traditional machine learning problem that has been there for a long time. But now with the large language model, with a lot of priorly trained work knowledge, they can do this job really better. So now if you bring AI to solve the problem, you have a much better normalized, standardized linked data asset. Then the next layer of AI agent is able to build on top of it and then discover and unleash more capabilities of solutions on top of it. So it's an iterative process, but you need to have the foundation together first.