Why enterprise AI lives or dies on applied research | Contextual AI’s Elizabeth Lingg

0:06Welcome to Dev Interrupted. I'm your host, Andrew Ziegler. And I'm your host, Ben Lloyd Pearson. We're here on site at the Engineering Leadership Conference here in San Francisco, meeting with over 120 engineering leaders, including our friends at Expedia, showcasing their squads framework that they built on the Linear B platform. We'll have that episode coming out here in a few weeks for all of our listeners. It's a really great one, I think. You know, we've been sitting down with a lot of engineering leaders here in the dome at this event. A lot of episodes getting recorded that you all are going to hear in the upcoming weeks.

0:39So, yeah, stay tuned. Lots of good content upcoming. Been a really great event, Andrew. And, you know, I really feel like the whole tone of this event was set like the very first morning, like the first conversation I had. I had these two engineering managers walked up, had barely finished my first cup of coffee and had a really wonderful conversation with them. Naturally, in 2025, anytime you talk to somebody in tech, it always ends up coming down to AI. Like eventually you're going to start talking about AI. And, you know, they told me they've done what everyone's doing these days. Like they've adopted all the tools.

1:15They've got Copilot. They've got Cursor, Cloud Code, Windsurf. Like they had a laundry list. They just kind of listed off. And I was like, well, you guys are doing it all. And they even told me they had like bought some like dashboard from one of the vendors here to like measure it all. And, you know, it's paying a lot of money for it. Got all these pretty visualizations. but they were like almost distraught over it. And, you know, they're like, there's this AI everywhere. Like we have it all over our booth even, you know, but it's all over all the booths. The event itself makes AI like really a central component of the entire experience.

1:51And, you know, they were kind of distraught about how like they had all these tools, they had all these metrics, these dashboards, but they still felt stuck. Like they weren't actually sure if they were making the right decisions and if productivity was getting better and if their developers were actually being more effective, being more efficient, producing higher quality. In fact, they were actually starting to experience some backlash from some of their developers. And particularly over one of the metrics that their executives had started to really get stuck on was PR throughput. They saw that dashboard and they were like, ooh, can we make that number go up?

2:29And naturally the developers are like, why do we need more PRs? Like, is that really the thing that we should incentivize? And even the engineering managers are, you know, they're a little skeptical of that metric as well. And, you know, personally, me, like I immediately just sort of cringed a little bit because I once worked on an engineering team that we had two North Star metrics. It was the number of commits that we created and the number of lines of code that we changed. So, you know, naturally, we just wrote some scripts to make those two metrics go up. And sure enough, these two engineering managers, they even described a recent conversation they had with their engineering team where it was getting a little fraught.

3:09And the engineers were wondering why they were so focused on these metrics. And the engineering manager just had to say, look, if our executives are so focused on this one metric, PR throughput, and you have these AI tools, you guys know how to make that metric. You can use AI to make more PRs. Like, look, you can just do it. I know it's not the thing that's going to be best for all of us, but you can just do it. There's nothing stopping you. And I felt so bad for them, honestly. Like, that's not what you want to do to your team. Like, that's not the kind of relationship you want to have. That's not healthy.

3:44That's not going to be productive. But it's a common refrain we hear. A lot of these organizations, they really never make it past that measurement phase of the developer productivity journey. and they like buy this dashboard, they get a framework and the execs might think that's enough. But the reality is that you've really just barely even made the first step at that point. And if you don't take action, you know, if you don't realize, if you don't take any steps to improve upon the things that you're measuring, then you're never going to realize the full potential of AI. And I think that's a theme that we're seeing like time and time again here, you know, not only at this event, but it is coming up a lot here.

4:25and elsewhere. And, you know, we're obviously here with Linear B, you know, we have the Dev Interrupted Dumb. Linear B is also here with their booth showing off a lot of the cool new features. But what I really love is the, you know, particularly with Linear B is they're not here to just like show you how to measure AI and be done with it. Like we're really focused on how to get better with AI. So, you know, as a part of this, Linear B has announced a bunch of new features around it. So like we've got the AI code reviews that are doing really awesome, especially in head-to-head comparisons against all the tools that are out there like Copilot.

5:00Because you think about code reviews are one of the most common bottlenecks in the SDLC. So really giving practical improvements to dev teams. We also announced the new MCP server that just generates helpful artifacts to help you make decisions. It's a new way to consume data within teams. And it's almost like an evolution going beyond metrics and dashboards. Now you have like this companion that helps you make sense of your reality. But then things like developer surveys to help uncover pain points in the AI journey. And then, you know, of course, AI dashboards. Like after you've made improvement, you know, you do need something that proves that you've made those improvements.

5:41So, you know, rather than being like metrics being the first step, I think that's one of the things that, you know, us here at Dev Interrupted, but also Linear B, I think really makes a point is that metrics aren't the answer. They're the proof after you found the answer. It's been a really fun event so far. And we've had a lot of conversations like that. But I think if I had to really like summarize where I feel like the zeitgeist is today, and that is, you know, there's so much hype in the AI space. I think we are beginning to transition out of things being purely forward looking like everyone's going beyond just like, what is what will the world look like two years from now?

6:19There's still lots of that, of course. But there's a lot more now of here's what we have done and are doing and here are the practical changes that are happening now. And here's how that's going to change the world in two years or three years or five years, you know. So it's really starting like AI is really starting to show up in the real world, not just be some like futuristic thing that we're all waiting to show up. You know, it's a really powerful anecdote you shared from those engineering managers who came up and talked with you because across the conference, I think a prevailing theme has been that AI is a force multiplier for the things that are happening within your organization.

6:58So if you value things like DevOps and really strong process and quality, then AI can help magnify that. We just heard an amazing talk from Natalie Gleam, the chief engineering officer at Duolingo, who talked about how they harness AI as a force multiplier in order to create more content quickly without reducing quality. And that's the real key, is that balancing that quantity with the quality is how you achieve really great success within your team. And you can actually use it as like a force of change within your engineering org. And that's like a prevailing thing I'm seeing and hearing from a lot of the talks here.

7:36I think every single backdrop has the letters AI on it somewhere. Everyone here is talking about it and what it means to build incrementally. I think that's a really good call out that you made that there's less of the, oh, the nebulous look forward. What does it look like in X years? And it's more about what does it look like right now within our organizations? We've all been tangling with it for a year. What's happened after all of that? And are we upside down? Are we actually better off? And I think those are real conversations happening within all of these engineering orgs that are here. So it's amazing to all come together and share those insights with each other.

8:11Yeah. Yeah. So, of course, not everything is happening here in San Francisco at ELCA. There is the rest of the world happening. And there are other stories out there that, of course, we want to cover while we're here, too. So let's get into some of the news stories, Angie. So what's our first story that we had today? Yeah, so along the same theme, we read an article from Ankit Jain. This is at Aviator about throwing AI at developers and about how it doesn't really fix their problems. And this is a prevailing idea of something you just talked about, something that is really encountering a lot of engineers within their organization right now, that bottlenecks and problems within your org, they don't go away when you add AI.

8:47In fact, AI will magnify those problems. And so this article breaks down kind of a flipping the script on how you think about some of the ways that you would utilize AI. So like one that stood out to me is that complexity can never be removed. It can only be abstracted. So when you use AI to make a process more simple, it doesn't mean that all the complexity is no longer happening. It's just happening in a place where maybe you're not seeing it as much or you don't have as much touch on it. And that's what saves you time. but it's not good to save you time if you lose the quality and the exact like what you're trying to achieve with that actual process and so uh it really dives into like why it's important to understand under the hood what's happening within your ai workflows and goes back to things we talked about on dev interrupted like the power of evals making sure things stay on the rails i think it was really interesting to see how it kind of like laid these top level points for why it's a force multiplier.

9:48Yeah, there was a line in this that really stuck with me that has actually come up with a few times, actually just in these couple of days we've been here. And that was, don't just pick a tool, pick a problem. You should always approach AI from solving something that your organization is facing. If you buy a tool and then search for a problem, you're probably going to misapply AI. or even worse, you may find yourself going in the wrong direction using AI. And in fact, I spoke with John Amaral while we were here, and that episode will be coming out in coming weeks. And he brought up a very, very similar point.

10:26And his company is doing some really, really cool things with AI and actually productizing it. And he made that exact same point that when they were using AI to build a product, he was very focused on finding the problem and then figuring out how AI can be applied to that problem. So it's definitely worth checking this article out. What's next? I think we have an article from OpenAI about hallucinations. Oh, yeah. So this has been a really interesting one to follow. OpenAI released a blog post building on a research paper that they did about why language models hallucinate in the first place. And if you've been working with large language models like many of us have, you know that the hallucinations are actually more like a feature, right?

11:07When we call things hallucinations, the idea is that it's a plausible guess within the realm of what could be true, but is actually not grounded in reality. And what we deem as a hallucination from the machine is actually a common occurrence that we do all the time as language interfacing things, right? And so the way that we build these systems and then evaluate them actually has a lot of similarities with how we build our own knowledge and how we test that knowledge. So this paper from OpenAI dives into the provenance of hallucinations and why they occur. And it really dives into that there are actually more things that are reinforced through post-training as part of the process of creating a model and bringing it to market.

11:50And it's a thing that happens because of how the models are incentivized. And there was a really great post that we saw from recent guest, Dr. Tatiana Mamout, who came on DevInsrupted earlier this year to talk about the sycophancy within AI where, you know, we all know this experience where you're told you're absolutely right when you're absolutely wrong. And this behavior, it stems from, of course, the way that the models are post-trained and evaluated after they've been fine-tuned, right? And so she calls out rightly that it's just like on the SATs, how students are penalized for leaving a question blank.

12:27These evals put these AIs on rails to where they're always pressured to make some sort of answer, even if it's not true. And this is what ultimately ends up kind of bubbling up these hallucinations in these experiences that we have, because we tell the LLM to be right, to be certain, to be confident. And that's the same thing that it's tested in, in that same language in a post-eval, right? And then it's put to task. It has to make a guess. If it leaves a blanket, it knows it got it wrong. If it makes a guess, at least it's maybe one out of a four chance. And so this is just like how humans work.

13:00And so ultimately, the incentives for what will make AI more truthful are the same incentives that make humans better capable of synthesizing thoughts. And this means that don't just rely on binary results. Yes, no. Is it true? Is it false? Have them write short responses, essays, avoid the weasel words, and actually figure out what it is that you're trying to do and say. And that's the mark of a highly intelligent person who can convey their thoughts. And that's the road we have to go down with LLMs to get them there too. Yeah. One of the examples they included was if you ask a model to tell you or to say when someone's birthday is and it doesn't know, and it might take a guess, and maybe it guesses the month right, but it gets the day wrong, and that gives it a partially correct answer.

13:45It would rather do that than say, I don't know, because that is 100 % wrong. So that incentive model encourages it to get that partially correct answer. And it's going to say that partially correct answer just as if it was a 100 % correct answer. And as you mentioned, part of the issue with hallucinations is that variability is actually what makes probabilistic tools like LLMs valuable because it gives you some level of adaptability. I think a lot of people who are working on improving these models do believe that hallucinations, if they're not completely fixed over time, they will get much better.

14:24You know, we'll get better at solving it, at reducing the impact of them. Something else to consider is, you know, if you're operating in a space where this does cause problems for you, you can also find ways to implement deterministic checks to sort of mitigate the risks for that. Yeah. So if you want to understand a little bit about why hallucinations exist and why it's a problem and what the companies that are building this stuff are doing, this is definitely an article that you should check out. Absolutely. All right. So our last one is filed firmly into things that I don't think any of us knew that we needed, but here it is today.

14:58And now we all get to live with it. So what's our last story for the day, Andrew? This is my favorite one. So we've covered Joffrey Huntley and all of his work, Vibe Coding and how he uses things like Cursor and Cloud Code and then AMP to build things that he never possibly could have built before. And the way that he thinks about using these tools is so optimistic and just so cavalier that I'm just always constantly plugged in to what he's working on. I think if there was a Vibe Coding Olympics, I think Huntley would be like the best Vibe coder, like maybe in the world right now. Yeah, he has been at the top of some charts, maybe to his chagrin.

15:40But I will say that the work that he does is really interesting. And we're going to cover one of them right now. So he's been working all this year. Well, Claude has been working mostly to create a new programming language, a C compatible standard lib using only Gen Z slang. So what this means is that he used C to write basically a compiler in a new language. And then once he wrote enough of that language in that new language, he then used this cursed language is what he calls it to write itself. And in this process, what he did is he unveiled that you can really kind of do anything now within software engineering.

16:21And he really kind of breaks down this falsehood that a software engineer skills will atrophy in this world of AI because he's somebody who never had the curiosity before to even want to build a compiler or understand how they worked. But because he was enabled by these tools, he was able to work with Claude on a loop, running a very simple prompt to incrementally create a Golang-like programming language using only Gen Z slang. And the lexical structure of this is amazing. He released this website that we'll include in our newsletter, so you can go and check out how you could use this language to write your own functions.

16:55And what ends up happening is every single declarative word that you would imagine writing a program with has been literally replaced with Gen Z slang. So I'm going to call out some of them that are really funny to me. For example, you declare a function with slay. So you would say like slay add as like a way of declaring a function. Another one that's really funny is when you return, you don't say return, you say damn. That's pretty good. And then, of course, you have, of course, working with things like true and false. True is based. And cringed is false, of course. And my favorite, my absolute favorite is doing a comment block.

17:33You know, every language has it different. Maybe it's a double backslash. Maybe it's a backslash. And then an asperic. No. In cursed language, a line comment is frfr. So for those listening that don't know what that means, it means it's literally saying for real, for real. and if you want to do a block comment well you open with no cap and you end with on god it's hilarious and it's something that i didn't think the world needed until i saw this website and honestly i'm gonna probably sit down in the saddle and try to write something with this language he calls it cursed it is literally cursed it is the cursed gen z language and he also made the joke that he cursed at uh claude a whole bunch while i can i can imagine what do you think about this one well of course we had to consult with our resident gen z slang consultant yes producer adam he's the only way we could make sense of this although it is it once you know the slang it is a surprisingly readable language which is really phenomenal i'm gonna list off some of my favorites so to import a package yeet that's brilliant like i feel like that should be what javascript changes their import i feel like yeet should be like export it'd be like i'm done with to eat but i like it a constant is facts that's a great one that is a good one uh to break ghosted an int is a normie man there's there's just so many like you really have to go check out the cursed web website to go to go see all the the incredible ones and if you haven't seen huntley doing this vibe cody he actually does it while he sleeps which is the best part yeah it live streams on youtube he's got all this agents just like hacking away as he sleeps um he's australian so it's like during our daytime he's he's at home sleeping and like i'll just have it like casually playing while i'm working some days and like i'm like man this guy's working 24 7 you know like it's kind of incredible it's amazing and then he and then you know he'll wake up and he'll see the work that it did and he'll review it and it's amazing actually if you go to the repository where he's been building this language the amount of incremental work and all of the artifacts of production that have been created by him in the lom through this whole process i have extracted so many techniques that i use all the time from how huntly does this and the last thing i'll leave you all with is on the website we'll include there's a whole bunch of leak code examples so you can actually go read full leak code functions of common things people are asked to do in programming languages like determining the maximum depth of a binary tree using only the cursed language so definitely be sure to check this out and we would love to see if you write anything in this yeah yeah and i and i want to just point out that i think what this really illustrates is the brilliance of when you construct this in an ai agentic system with all the context it needs with the structure with the tooling that it needs how simple the the capability actually becomes because this is not like a thing that's going to get deployed into some high production system that is like well okay maybe maybe we'll see who knows so he sets this whole up and then gives it like the most simple prompt, which is the most incredible part of this.

20:36You know, it's for three months, he says his agents had the simple goal, produce me a Gen Z compiler, you can implement anything you like. That's it. That's all he told it to do. And here we are three months later. Like that's really a pretty extraordinary thing. And I think this will be a case study in how vibe coding this type of work happens in the future. So it's gonna be a case study. It's gonna be a full conversation. We're definitely have to get him in the seat here. to talk about it. Yeah, yeah, yeah. Stay tuned. I have high hopes that he'll be a guest on our show soon enough and we'll definitely get to pick his brain on how all this works.

21:09So amazing. Well, you know, those are our stories for the week here at ELC. There's just so many insights that we're extracting. We're going to keep talking about them week after week as we drop the amazing episodes that we've been recording here on site. So stick around. Are you investing in AI but struggling to see the real impact on your engineering team's productivity? Well, you're not alone. In a free 35-minute workshop that I'll be hosting with Linear B, we're going to show you how to translate AI metrics into business ROI, just like Expedia and Adobe have. And you'll learn a simple framework for understanding where AI is helping, where it's hurting, and where to focus your next investment.

21:47And besides, you're going to get a nice takeaway report on AI productivity as well. So don't miss out. The workshop's coming up on September 17th or 18th. Grab your slot, and we'll see you there. Today, we're diving into a really cool topic. We're exploring the world of specialized intelligence and the challenges of bridging research and product development. And we're doing this with Elizabeth Ling. She's the Director of Applied Research at Contextual AI. That's the team behind RAG. And ultimately, Elizabeth's impressive background spans major players like Microsoft and Apple. She brings a unique scientific perspective to making AI work for real-world applications.

22:29Elizabeth, welcome to our show. Thank you very much. I'm so excited to chat. We're excited to dive into your research background. And I wanted to start just by talking a little bit about your career because it's taken you through some impressive places that I just named. And it all started with you for a love of learning. What does it look like as an applied researcher on a software team? I would say that I've always been passionate about math AI. Even as a kid, I read 2001, A Space Odyssey, and I was so fascinated by the book and whether, you know, Hal had a software hardware malfunction. Did he achieve singularity when he ejected the person from the spacecraft?

23:11What happens with intelligence, with AI? So I've always loved AI and the research side for the longest time, math, statistics, data, data science. It's always been a passion. But then how do you actually bring that into a product? How do you build something, something that works for, you know, real users, real customer data? So that basically is where my career has taken me to actually apply the research, apply the ML model and build real and exciting products. So your journey starts by becoming a researcher, someone who's obsessed and great at science and math. And that lets you go deep on like an academic type of route of learning how to conduct research and collect great data.

23:55And then from there, how did you look to pivot those research skills to become an engineering leader? Sure. So I think just the fact that I joined industry after grad school at Stanford, I ended up, you know, working on different aspects of the code. So not just, say, clustering and similarity algorithms, but also information retrieval, building my own Lucene frameworks, and calling different APIs, external APIs, and so forth, which brought me into a lot of this software side as well. Of course, for anyone with a computer science degree, you do learn the fundamentals in different areas. But even though my job title was on the ML side, I'm still touching all these different aspects of the system and I'm working with the DevOps team and I'm working to train a model on a server on a specific compute.

24:50So I think what happened is once I joined the real world, I have to build these products. I have to make sure the customers are happy. Then I start worrying about things like latency, right? So even 16 years ago after grad school, latency was a thing. So I was working on governance, risk and compliance and fraud detection and using AI for that. But even for that, you know, if it's too slow, the customer is going to be unhappy. So that's that's what brought me into the software side of things. And then honestly, it's been the best fit for me. I realized that even though my greatest passion is on the research, AI and math side, it's really good to have those solid computer science fundamentals as well and at least have an understanding of code.

25:32Of course, you can vibe code these days, but you have to have the understanding of the system. Having that full understanding, I think, is really valuable. And what I've noticed is the industry has shifted. Like when I first joined, you either pitch, you're an AI researcher or you're a software engineer. You're a data scientist, you're a software engineer. But these days, it's very common to have the full stack, MLE, applied scientist, applied researcher, which works out great for me. I think it is important to have that full perspective. Yeah, that's a great call out that you've made there about the diversification of the skills and the baseline that folks are expected to operate at.

26:10We're hearing that from everybody. We talked with Lee Robinson, VP product at FurSell, echoed the same thing happening in web development where you have like the full collapse of front end and back end into this full stack engineer that's a baseline now. That's table stakes as someone working in that environment. And you see the same thing happening. We talked with a guest from Google DeepMind who talked about how most engineers and how engineering work. It's a collapse of back-end, front-end. It's not that specialists go away. It's just that the expectation level that everyone's supposed to be operating at shifts.

26:41And there was something else that you said in there, too, that I really like. I want to double-click on. And you said, when I entered the real world, when you went from doing the theoretical work to working in the practical application world, what was a big challenge you faced when you made that transition? I think anyone who goes from the academic world to a more of a production environment to learn about all the basically all the failure cases that exist and, you know, inner team dependencies. And to your point, I mean, yes, specialists still exist. They're great. I love specialists. It hasn't been obviously my trend in my career per se, but it is the baseline you do.

27:23Even if you're a specialist, you do have to have that baseline understanding. And so I think what was shocking to me is just at first, once I was getting to know how to build software and cloud software and applications and so forth, and eventually SaaS is like, okay, there's still much that goes into it. You know, I always thought in academia, the research is very hard. It is a very challenging aspect to it. But it's surprising to see how much you have to adapt that research to really put into production, work with multiple teams and multiple disciplines. Yeah. And especially when you work at like a large company like Apple and Microsoft that has a bunch of teams, has a bunch of disciplines, it's a huge spread out organization.

28:08What was it like shifting from working in that kind of environment to being an applied researcher and working in a more startup like environment? So my trend over my career has actually been either startups or sort of the startups within the large companies. So I have been a part of several early stage startups or startups that are quickly acquired by a big guy kind of thing. And I think what I love about it is for both of those situations is the ability to really contribute to the research community to be scrappy and move quickly. For example, at Microsoft, I was most recently working on the co-pilot team and leading the language understanding for the office co-pilot.