The hidden costs of pre-computing data | Chalk's Elliot Marx

0:05Welcome to Dev Interrupted. I'm your host, Andrew Ziegler. And I'm your host, Ben Lloyd Pearson. This week, I'm sitting down with Elliot Marks of Chalk, the data platform for AI and machine learning. We invited Elliot on-site to ELC Annual this year to discuss how AI relies on real-time pipelines rather than traditional storage and the costly dangers of pre-compute in a world that's increasingly consumption-driven. So if you're transforming data into business results, and let's face it, that's what we're all doing, then don't miss this one. But first, let's discuss today's news stories. OpenAI is code red, Google quietly launching its workplace studio, a pragmatic guide for LLM evals for devs, and ignoring the spotlight as a staff engineer.

0:52Ben, where do you want to start? Yeah, let's just start right at the top because I've been involved in this conversation on LinkedIn and it's a pretty fascinating story. But Microsoft's advantage in artificial intelligence is evaporating as Google Gemini surges ahead. So what's going on here, Andrew? Yes, so it's no news to anybody that Google Gemini 3, it's surpassed OpenAI's chat GPT-5 and multiple performance tests and benchmarks. And the Nano Banana Pro image generation tools are also outperforming competitors like Dolly. You know, OpenAI CEO Sam Altman, he declared a code red in the last week.

1:27He canceled planned marketing and monetization timelines and warned staff that Google Gemini poses a serious existential threat, with expected growth potentially slowing to single digits for OpenAI through 2026. And you know it's bad when the company cancels the ad product. So we've covered this recently, but even just last week, we talked about how OpenAI faces financial strain. We looked at a number of graphs showing their capital expenditures, requiring hundreds of billions to fulfill a$1.4 trillion in compute commitments over the next decade. And meanwhile, their primary partner, Microsoft, and their integration of the AI features, you know, they've received mixed results and it trails Google's seamless ecosystem advantage.

2:12This is a sea change event in the foundation model race. Ben, what do you think is going on here? Yeah, well, first of all, Andrew, you and I both know that Sam Altman is a regular listener of this podcast. So we know he's been hearing my advice on, you know, all these frontier AI model companies. Yeah. So Sam, listen up for this week as well. Yeah. So I'm going to start sounding like a broken record on this, but there really are just no moats within the frontier model space yet. But this story is really significant, I think, because it may actually be the first example of a company actually starting to establish a moat within AI model space.

2:52And that's Google because, you know, they have a lot of incumbency and they have a lot of data. So, yeah, I think that's very significant in this situation. And it's interesting to the note that Google's moat in this world, it doesn't come from the frontier model alone, but from its full ownership of the stack that makes that model possible. They created the chips, you know, the tensor processor units, the TPUs that trained the model. They aren't beholden to NVIDIA or CUDA. They serve their own inference on their distributed cloud platform where they serve all the rest of their services as well.

3:24And this creates a nearly untouchable position that simply other players in the market can't emulate. Yeah, I mean, it's really easy to forget about Google in this race because, you know, I think a lot of us remember the extraordinary flop that they had when they launched Bard a few years back. So, but, you know, I started using Gemini this week for the first time, really. And I think it still has a lot of room to improve, like particularly around like user experience. But when you think about it, they've got just a treasure trove of images and videos that they can use for training and the hardware to scale it.

3:57So, you know, and that's why I think Nano Banana Pro in particular is receiving a lot of attention because it's just an extraordinary image generator that is like, in my opinion, miles ahead of everyone else. But I think there's like, there's like two big things that our audience should take away from this. The first is data quality. So Google is really proving the importance of super high quality data sets. If you have really great data hygiene, really great accessible data within your organization, you can give AI the context it needs to do really awesome stuff. So because Google has all these images and videos, it's probably going to be very hard for other companies to sort of replicate their success on that.

4:38And then the second is agility. So if you're building like workflows, like either or products like internal or external, and you're incorporating AI into those things, now is not the time to commit to a single provider like these, these, the model leaders are shifting so dramatically so quickly, that you need to have the agility to, you know, sort of switch between them at a whim, depending on which one fits the individual use cases that you're trying to solve. so you know my my approach to to a lot of what we're doing with ai is really to be model agnostic like i want to be able to switch on a whim you know and i decided this week to start bringing gemini in for specific workflows within our team for some things that i found that it is better than other models or even some use cases that like i had written off as not yet like the technology not quite being there yet but now with gemini 3 i can actually solve some problems that i wasn't able to before.

5:32So, you know, the market is going to standardize over the next year or two, but, you know, now's the time to be more reactionary, to be able to adapt more quickly to all these changes. And speaking of adapting more quickly to the changes, you know, Google's making this easier for folks with other tools as well, like their new workspace studio that they just launched alongside Gemini 3. And this allows users to create, manage, and share no code AI agents that can automate those complex workflows and takes advantage of the workspace apps within Google that many enterprises and companies and businesses use to do their everyday work.

6:08It's where, you know, in many cases, the data lives that you're using with your other LLMs, things like Google Drive. And this allows you to create plain language prompt driven agents using Gemini AI that can integrate across this entire suite of tools along with external APIs and can be shared and scaled across the organization. This is pretty exciting for folks that are on Google Workspace. So definitely keep a lookout if this is yourself to see if you can take advantage of these features. I know I will once they become available to me. But I think it's interesting to see the evolution of these no-code tools and putting more agent building experiences into people's hands.

6:46I'm excited to see what the more non-technical folks who approach and build these tools ultimately make with it. I think we can learn a lot about how these tools could be utilized from them. Yeah, you know, so much attention has been put on to coding and how AI can be used to accelerate that. And there's so many aspects of their job that are not coding related, right? So we all have to recognize that knowledge work is changing. And that this might sound strange, but I've always had this like sort of weird desire to like talk to my spreadsheets, like to be able to tell my spreadsheet what kind of changes I want to make to it.

7:21But if you're doing a repetitive action with a data set over and over again, it's really nice to be able to tell. I just wanted to be able to tell my computer to do that. And we're kind of getting there now, which is really, really cool. I had an example just this week where I kind of asked an AI thing to do a bunch of spreadsheet updates for me. And it mostly worked and at least did a lot of the toil for me. There's going to be a lot of growing pains around these type of capabilities, too. These capabilities are going to become commonplace in all the tools that we use. So whether you're on Google or not, I think these types of AI agents and support services are coming to just about every platform.

8:00But, you know, I'm not really jumping like biting at the bit to get them today. You know, it takes a while to refine features like this. So I kind of always have low expectations going into it. And then eventually I'm surprised by them. So, but yeah, I can imagine a lot of situations where I would want something like this, like, you know, managing my email inbox, like who actually likes to do that, you know? So it'd be great to have something helping me with that. Yes, it's just good for also when you collect your best practices for how you do your specific flavor of knowledge work. A lot of times that can be captured into a workflow that saves you a lot of time without losing that spark or that ingenuity that you do bring to each task.

8:39So definitely something for folks to leverage. We're going to continue to follow that story. But I want to jump into our next one. This is an article from a past guest author and a friend of the newsletter, James Boyer, talking about how teams are measuring the success of their engineering organizations using metrics. We're talking about velocity, commit frequency, as well as sprint completion percentage, lines of code, and ultimately how this is disconnected from business outcomes. They don't connect with what executives are looking for for the impact of the engineering org. And so in this article, James emphasizes that leaders should use metrics that identify friction, bottlenecks, and feedback loop problems.

9:20Making it the same lingua franca that your executives use allows you to put your engineering problems at the forefront when they do cause systematic issues. And in doing so, we're not prescribing a fixed set of metrics, but rather defining what needs to be done in business terms. And creating this set of indicators is like a whole different way of utilizing the information within your engineering org to get more done across your teams and across your leadership. It also allows you to inform them in their own terms about the success of your teams. And this is what we're all about here at DevInterrupted and at Linear B.

9:57We talk all the time about how you use these metrics to tell the story about how your engineering team is impacting the results of the organization. This is a must read on how to apply these things. I strongly recommend that you check out his article. Love the stuff that James writes. I'm really happy to cover it here. You know, Ben, what do you think about this article? Yeah, at the end of the day, I just really like how James takes super direct approaches to the problems that he addresses and is always there with some nice reality checks as well. Yeah. But, you know, and at the end of the day, you know, executive leadership expects a return on investment on their engineering organization.

10:35And you should be ready to prove it as an engineering leader. I really like the, in this article, he calls out a lot of weaknesses of like some of the common pitfalls that people encounter when it comes to engineering metrics. You know, things like velocity doesn't tell you everything. You know, you can't operate and measure a software engineering organization like you would a factory. Moving faster doesn't matter if you ship the wrong things. and I really like his advice on like rather than picking like a metrics framework to solve your problems you should identify individual problems that your team has that can be measured and then pick metrics based on that that will help you improve whatever that problem is but yeah thanks for helping us navigate the efficiency era James yes and I want to jump into this next one as well because it's also helping us navigate new territories that we're all moving into.

11:28This is a guide from Hamel Hussain in collaboration with Girdly Oris of Pragmatic Engineering. And this is the Pragmatic Guide to LLM Evals for Devs. And this is a fantastic article for anyone who's building agentic systems. This is not our first time covering Hamel's work, and it certainly won't be our last. I would consider Hamel required reading for folks that are building these tools. He's one of the most knowledgeable folks that I turn to time and time again when I'm building and evaluating AI deployments for our own team. And he also previously was the co-host of America's Next Top Modeler.

12:03It was a hackathon where we built an agent with an eval framework to crawl over a massive store of data in order to answer some really tough questions. This guy is brilliant. I love everything he writes. And this article is another installment in how to build battle-hardened AI systems that are inherently random. How do you ultimately evaluate the results from them and make sure that they stay on track? And he's laid this out in a lot of really powerful metaphors, in particular the different gulfs that separate the engineer from the data, from the LLM, and all of the intentions and suspicions and lack of clarity between them.

12:45It's a very dangerous kind of sea to navigate between these three islands, but Hummel gives you a great map for doing so. He moves past vibe-based development to structured error analysis, how to use stack tracing to create really powerful and flexible internal tools to look at your own consumption of AI products. And it distinguishes between just creating a system that you think works and that, you know, gives you kind of where you need to be. And actually creating a system that is self-healing, can evaluate itself with an LLM as a judge, and ultimately reach out to expand into new problem solves that you didn't even think it could do.

13:23This is how folks should be building with agentic systems, and it's a really great guide. So I strongly recommend that you check out this one as well as everything else that Humble's provided. Ben, what do you think of this article? Yeah, we've been adopting some new AI tools and platforms over the last few weeks. And, you know, I think very quickly, one of my, I think for both of us are like one of our favorite and most important features is things like tracing capabilities. Absolutely. Yeah. Like I want to be able to evaluate every single GPT within our workflow. Yeah. And particularly this article, I really loved the story they had of their team vibe coding a purpose built app to solve the problem of needing to quickly annotate hundreds of GPT responses.

14:06Like this is something that we've started to kind of do here at Linear B where we just we have a problem. We just build an app that helps us solve that problem. And it's a really powerful capability. But I also just really like how they divided eval situations based on whether the challenge was deterministic or probabilistic. You know, there's a rush to try to solve everything with AI and with GPT tools. But code often works just as well. You can write deterministic code that can evaluate GPTs in some situations. And then when you're using GPTs as a subjective judge, like if you're not in a non-deterministic situation, one of the bits of advice is that it's better to do things like have a pass-fail judgment rather than a scoring system.

14:51So you have a GPT or a human that is passing or failing the results or the output of a GPT and using that to make the models better. So yeah, there's a ton of great advice in this article. I think we're going to be studying it ourselves just for what we're doing, but anyone working with AI needs to read this. I've borrowed so many things from Hommel's writing, including this Vibe-coded trace stack reviewing system. Just like what you said, you need to be able to see every query that your team or your users are giving to your tool. But then what the LLM did at every step, it's reasoning, the documents it pulled, how it synthesized it together.

15:30If you don't understand all of that, then it's not just a black box. It's a black box in a black box in a black box in a black box. And so it's critical that you build this structure. And he calls out how he just like vibe coded this tool for doing it for their own bespoke internal traces. You know, they did it in like an hour. This is how people need to be using the tools. This is the new standard way of building with these systems. So be sure to check it out. I want to jump from that into our last article of the day, which is from Lalit Maganti. He's a senior staff engineer at Google, and he's talking about why he ignores the spotlight as a staff engineer and actually what that means for his long-term commitments to the projects he works on.

16:11He calls out how long-term stewardship on infrastructure and developer tools allow teams to get compounding returns and deep technical impact on their team's organization. Contrasting this with high visibility, fast-paced environments, jumping into brand new products, which often happens right now in the AI world when people are jumping into shiny new teams and trying to build out something from zero to one. It's really easy to lean into that as the way of career advancement. You want to be in the executive spotlight. You want to be building the new, exciting, shiny things. But he highlights the alternative metric for career advancement, which is based upon his own experience working on Big Trace, which is providing a deep level of functional utility and critical cross-team collaboration on critical systems that power everything else.

17:00And actually, by building this deep expertise and ownership of a core part of an infrastructure or product, that earns you the ability to say no, because then you can protect the quality and the standards of the organization and how it's gotten there so far. And this article lays out his own journey in Google and how he's built that for himself. relates a little bit with Sean Godecki as well, which we've covered here on the pod. So be sure to check out this article. It's a really great reflection from someone at Google about how he's navigated that journey for himself. Yeah, I have a lot of respect for people who work in internal platforms, build developer infrastructure, like those types of roles, often are unsung heroes and always have to kind of fight for what they want.

17:43So it's, yeah, I love to just highlight personal stories like this, like sort of help others understand the journey that it takes to get to these types of roles and, and, and yeah, what that might look like. Yeah. Stick around because after the break, it's my conversation with Elliot. Want to know how AI is really impacting engineering productivity? Join us for the AI productivity roundtable, where Linear B is unveiling brand new insights from the 2026 software engineering benchmarks report powered by 8.1 million pull requests, 4 ,800 teams and data from 42 countries. You'll get early access to the full report, plus a 35-minute no-fluff discussion with industry leaders like Rob Zuber, CTO of CircleCI, Smruti Patel, VP of Engineering at Apollo GraphQL, and Yashai Viri, CTO of Linear B.

18:32We're breaking down the state of AI in engineering leadership, this year's SDLC metrics, benchmarks, and three new AI-specific benchmarks. Learn about why AIPRs wait more than five times longer for review and why acceptance rates are shifting across tools. Reserve your spot now and get the 2026 report delivered straight to your inbox. Today, we're sitting down with Elliot Marks, the co-founder of Chalk. Elliot began his career at a firm where he built early risk and credit data infrastructure, and he also co-founded Haven Money, which was acquired by Credit Karma to power its banking products.

19:09And today, we're talking about his company Chalk and how it's tackling the AI and machine learning compute layer. Elliot, welcome to Dev Interrupted. Thank you so much for having me. It's great to be here. We're really excited to sit down with you. And I just want to go ahead and jump into it. You know, we're here on site at the Engineering Leadership Conference, and everyone here has been talking about AI and how it's transforming their engineering organizations. And you working on this problem at Chalk, can you tell us how y 'all are entering the the field of engineering and tackling how folks are building and using AI to ship software?

19:43Yeah, absolutely. So Chalk is building real-time data pipelines. And this is really important as you have all these new models doing inference right at the edge. So I think the traditional paradigm has been to do a lot of pre-compute, something like a Databricks or a Snowflake. And for huge data sets that you want to pre-compute a lot of data, they're wonderful products. But if you want to fetch data really at the moment that you're doing some inference, especially if some of that data is expensive, especially if you need to do a lot of compute at that time, that's kind of where we come into play.

20:12Your background touches on financial stuff. So it's regulated, very hard to innovate within that space because you think about privacy and security and a lot of compliance, right? And so is that something that y 'all are tackling as part of Chalk? How do you plug into infrastructures and enterprises that are working with sensitive data. Absolutely. So I got my start in FinTech, as you know. And at a firm, we were doing transactional-based underwriting. So at the time someone put something in their cart, we wanted to underwrite them for credit. It wasn't like an overnight process where you have a revolving line of credit.

20:45It was a decision based on the item that you have in your cart. And that really informed a lot of what we do today, which is how can you actually pull the data at that moment as opposed to pre-computing something because the data just wasn't available. Yeah. I think that, you know, these days we work with lots and lots of fintech companies. So we power about a third of the world's debit card transactions, for example. We work with SoCure, Persona, and Alloy, kind of like the top three anti-fraud platforms. And we process, I don't know, we've probably processed every single social security number in the world.

21:15That's crazy. So we really work with very, very sensitive data. And I think a lot of those, for a lot of those companies, we're kind of an opportunity to provide more visibility into the decision-making process and introduce governance that they want to see. Yeah. And so how do you get your engineers aligned around this privacy and security? Kind of, you know, it's definitely a bit of like a quicksand to move through. It can be really kind of like there's a lot of pitfalls involved. So how do you equip and build an engineering organization to innovate within those kinds of constraints? Yeah, absolutely.

21:47We're building a query planner, which is kind of a fun. It's like a database. Yeah. It looks like the internals of a Postgres or of a Snowflake. And a lot of what we're doing is tracking how the data is flowing through these query plans from a technical perspective. And so you get the engineers motivated to do this because it's a fun problem to solve, to figure out how to build a new database engine and how to kind of track the data as it goes through. From a deployment perspective, we deploy most of our software into the clouds of our customers. And this is great for them because they don't have to worry about their data going to a new place or someone new being able to see it.

22:20We're going into FedRAMP deployments, we're in fully AirRAMP deployments. And it's easy to kind of make that work and fit within any compliance regime when it's in a cloud that you're already operating it. So what does it mean to be building a compute layer for other teams and softwares to be building into? Because I think for a lot of folks, you know, the need for AI compute and the costs involved with it is a new conundrum that a lot of CTOs are grappling with in conversations with their CFOs, with their board, justifying the expenses of how much they're putting into generative AI, how much they're putting into their AI workflows, and then tracking the outcomes and, you know, the impact of that.

22:59I'm curious, like, what does it mean to be building in that space? And what are the kind of questions that engineering leaders ask when they come to you for the first time? Yeah, absolutely. I mean, this stuff can be wildly expensive. And computing all this data can be wildly expensive. And I think that's something we haven't really like, thought about. Yeah, we're just grappling with it a bit, right? Yeah, we actually just put out a case study with one of our partners, whatnot, they're like the largest live streaming marketplace in the United States. And they were previously kind of computing all these predictions for like users and what they might want to do that day.

23:34Yeah. And then they were throwing away about like 90 % of it. So one of the reasons for moving stuff to real time is kind of counterintuitively to save money because you only compute on the data that you need to compute on as opposed to kind of like computing on everything. Or double computing in some cases. Yeah, absolutely. I mean, the other time you see this pop up is when you have to get data that has a cost associated with it. So sometimes it's like a compute cost that you're paying because you're running an expensive model. But sometimes it's like literally dollars and cents that you're paying to a vendor because you're asking them to like evaluate something on the phone.

24:09As an example of that, like we work with SoCure, for example, like the largest, to my knowledge, anti-fraud provider in the United States. And they run a hugely complex set of data that they process in order to make a decision. And banks and credit unions and fintechs are telling them a packet of information. Like here's an email, here's a social security number, here's an address, like, is it real? Should we let it through? And they couldn't possibly pre-compute on like all possible combinations of emails and phone numbers and social security numbers because it would be like, you know, more combinations than...

24:44be a lifetime of the universe to compute, right? It's just like totally crazy to compute over that. So they have to compute their data on the fly. And I think that's, you know, also kind of cost based reason. So I think you'll see a lot of compute in this new paradigm shifting to the time that you're making the inference call, as opposed to trying to like pre compute everything in part to save money. Yeah. And, you know, I'm curious to as part of that, you know, there's the when you do it, there's also the how fast you get the response. And how do you think about latency at chalk and what are the kind of things that you find most important about it?

25:19Yeah, it's totally critical. You know, there are certain areas where like the latest Gen AI stuff, it just cannot go. Like a lot of the Rexys systems, they want to see like P99s at like 20 milliseconds or something like that. Yeah. And if you think about like running, you know, anything through a chat completion model, there's just absolutely no way. Right. So what you're seeing is still like, you know, XGBoost, LightGBM, like taking the day and needing to compute data really, really fast. Part of where we come into play and like my philosophy around like, I don't know how data science and machine learning engineering should work is that like SQL and Python are the languages of data science.