Treat your data pipelines like code, because your agents already do | MotherDuck's Jordan Tigani

0:02Welcome back to Dev Interrupted, brought to you by Linear B. My guest today is Jordan Tigani, co-founder and CEO of MotherDuck and founding engineer on Google BigQuery. He used to write more SQL than anyone at his company, and now, like so many of us, he can't remember the last time he wrote a query. We covered Steve Yege's Gastown here on the show, and Jordan wrote the data version. He calls it Watertown, where a data engineer manages a fleet of agents. Those agents write the pipelines and the visualizations from a prompt, and Jordan's point is that the cost lands downstream, in review and in catching what breaks.

0:41His answers that treat all of it like code, which adds to the review bottleneck, a key problem we're set to solve here at Linear B, with tooling like AI code review and merge automations. Learn how the mother ducking genius behind Mother Duck thinks about the data equivalents for his own gates. Now, here's my conversation with Jordan Tagani. But I'm really excited to have this chat with you, Jordan. You know, you're the CEO and co-founder of a pretty cool company right now, in my opinion. There's a lot of really innovative things coming out of Mother Duck. And your background as a founding engineer at Google BigQuery is, you know, just like a driving force, I think, in all of this change in narrative.

1:21And it makes so many folks look to you to understand like how data science is transforming. So we're really excited to have this conversation today. Talk about some of like the ancient history, even back in like the early 2020s. I know that's like a long, long, long time ago. But even right now, how Mother Duck, I think, sits at the center of an argument that the practitioners of the data stack are AI agents now. and data scientists and the way that we work with data and the tools that we use and communicate through with our agents on, they need to transform to meet that new challenge. So we're super excited to dive into all of this today.

1:56Jordan, welcome to Dev Interrupted. Yeah, thanks. Thanks for being here. These are exciting times and it's good to be in the middle of things. It really is. And you couldn't really be more in the middle of things. I want to back up for a second and talk about you know, Mother Duck and how we got to be here because your background was in big data. You built BigQuery and one of the biggest distributed warehouses like that does data, right? And so we're talking about a fundamentally different approach from how Mother Duck works now, which I want to dive into with you, no pun intended, but also understand like how that change came to be in your mind.

2:34Like what did you understand about what big data was and what it couldn't be and what it needed to change into that, you know, you were maybe one step ahead of on the insights. So, you know, we used, when I worked on BigQuery, we often used BigQuery to understand what was happening in BigQuery. And I remember doing some like analysis, you know, trying to figure out, okay, well, what, if somebody was using Snowflake, what size Snowflake instance would they, would they need? And so that we could kind of do some comparisons because the way BigQuery worked under the hood was very different than Snowflake.

3:06And so I was looking at kind of the size of data that people had and the size of queries that people were running. And the thing I found was it was dramatically less than I had expected. And something like 90 % of queries were under 100 megabytes. And megabytes is something that at Google people sneeze at. Actually, they don't even bother to sneeze at it. They sneeze at terabytes. But like, you know, and just something you wouldn't think you'd need, you'd need sort of a massively parallel engine to do. And like kind of as I, you know, I was looking like, wow, like most of our customers are actually using tiny data.

3:45And then even the even the giant customers like, you know, we had like HSBC and Home Depot and Walmart and like Equifax, like some of the biggest companies in the world, like the stuff that they were doing wasn't that big. And I think internally, there's a lot of energy around, okay, well, how do we make bigger stuff work and make it faster to do tens of terabytes and hundreds of terabyte queries? But the things that people were complaining about were the things that their 100 megabyte query wasn't any faster than their 100 gigabyte query. And so actually, the things that people cared about was like, hey, I've got a human waiting for this result.

4:26Like if you add a second to every query, you know, because of all this distributed mechanism, then that's actually going to make the experience of your users worse than being able to handle some giant, giant thing faster. Because even those giant, giant things are expensive. And so people tend to not do them very often. So kind of like I filed that, you know, away and it took me a couple of years. you know, actually I worked at Single Store for a couple of years working on their SaaS offering. And people kept asking for smaller instances. And we, you know, we had one that was half the size of Snowflake and the Snowflake smallest instance.

5:07And people were like, what if we did smaller one? And so it was like, it seemed like there really was a need for, you know, scale to zero serverless kind of smaller instances as well as large. And then the other thing that, you know, you should realize is that, you know, these, a lot of these technologies that were, we kind of built the data infrastructure around were designed when machines were small. Like, you know, if you had a hundred gigabyte query, well, a hundred gigabyte wasn't going to fit in memory. It wasn't going to like, like you needed, you know, if you wanted a hundred cores, like, you know, you, you had to, you had to combine a whole bunch of machines to get a hundred cores.

5:49And, and nowadays is like the just basic machines that are on ec2 like the physical machines are you know um hundreds of cores terabytes of ram and so like you don't actually need these distributed distributed systems and if you don't use the distributed systems you can just do things you know much more easily you can be faster lower latency like a whole bunch of a whole bunch of important important things. And so, yeah, it was like, hey, if we're going to do something, if you were going to build these things now, you would do them differently. And to me, that was an opportunity. There's a really interesting, I think, parallel in there to unpack of the idea that you even said the hardware was smaller, it was older, and you couldn't even get some of those mid-sized queries, those things could live in the RAM of the machine.

6:42It was always thought that we were moving towards having these bigger monolithic kind of like services that could handle all of that for us because it was just so hard to really visualize and understand how quickly the tools and the technology we were using were going to evolve and change and become more capable underneath us. And so like we make bets on like what we want to build and what we need for the future. But then sometimes we have to make the bets to take things away and to simplify them. And there's a lot of things I think we're learning about that even right now with, you know, how everyone's workflows are transforming with agents.

7:15You know, there's a lot of temptation to add a lot of complexity, a lot of stuff on top because you think you can orchestrate it in this kind of way. But there's a much bigger opportunity sometimes in taking away stuff you took for granted or that isn't necessary anymore. And you can only discover by starting to hack at it, you know. And so there's like, I think, a lot of lessons there that have parallels about how we work with technology and how our expectations change. And it's really important to call out, too, that data, too, became more widespread and just more accessible to more people. Like it was it kind of moved out of the domain of just being solely something that the data scientists or the data researcher would do at scale on these huge amounts of data.

7:56It became something that was more approachable and embedded into more tools and delivered reports and all sorts of ways to where working with data became more democratized. And because of that, we needed smaller, more customizable bite size on your device ways of working with that data to meet the demands. Right. So like understanding how to throw away something that you took for granted before and like trust a new definition of good. Like part of doing that, too, is understanding what you think the next level of good will look like. So for you, was that a vision of the access of these kinds of tools just being more widespread?

8:37And what are the opportunities that you still see for the tool to continue to evolve for now? It's like more agentic consumers. You know, I think a lot of things in technology, you know, you can have two things that solve a similar problem. They have a different architecture. They're going to be able to grow in different ways. And I think that kind of some of these older systems, the ways that they can grow are reasonably constrained and they move very slowly. I mean, it's just, I remember some of the optimizations that we worked on in BigQuery would just take months and months for something relatively simple.

9:15And, you know, working on this single node system, you know, we can just, those things can be done in, you know, in a week or weekend. And you can just move much, much faster. And so because kind of the system has been shaped a little bit differently, the way it can expand and extend and react to new changes is different. And I think, you know, you mentioned sort of, you know, all the changes that are going on with AI. I think if you have these sort of giant warehouses or distributed warehouses, just from a shape perspective, they feel the right shape for AI agents. We might have a lot of agents that are sort of hammering on different things and like, and having a single node system where you basically, you know, one, one warehouse, one agent, you know, for as many agents as you, as you have, like those, that seems to sort of fit the problem better than, than some of these, these older, older systems.

10:20Yeah, absolutely. I feel like we're trying to make all of the bits that an agent touches as atomic as possible so that we can isolate them and put them in these like more sandbox things. The idea of having agents at scale and at real time, all touching a same kind of data space is I think scary for most people who would want to protect their data. So I do think that becomes like even like a strategic wedge and like how the tool is adopted and who its consumers are. And I want to talk a little bit about who those new users are, because, you know, it is the agents, but it is also still the engineer, the data scientist, the data engineer, you know, figuring out how to orchestrate them.

11:01And that's where Mother Duck, I think, is really leading in terms of teaching the future of how the data scientists and the data engineers are going to work with their tools. Like, we really loved your entire coverage on Watertown. We've talked a lot about Gastown here on the show, and we've covered it since the top of the year. All of the many things from Steve Yage and how those different narratives have evolved. We've actually adopted a lot of them here on our engineering team and on the show. And so like when they see the same parallel applied towards data engineers and like the ways that they work with their data and how it will transform, I think was really powerful and really smart, too, because it gave it gave us language to talk about the levels of AI fluency as you got better using those tools.

11:44I wanted to dive into your head again, no pun intended about that and learn a little bit about how long it took to develop that idea after thinking about, you know, what Gastown was doing for engineering and like what, how, how innately did all of that come to you? So I think that we're in, you know, we're in this like transitional moment right now. Like I think I had a college professor and, you know, who like he was the guy who invented punctuate or named punctuated equilibrium, you know, in evolution. And I feel like software technology, like you often have something similar with meeting, like you're sort of at this equilibrium state and then some change will happen.

12:27And then there's sort of rapid transition to a new equilibrium. And so I think we had the modern data stack was this sort of equilibrium state where you have like, okay, you have these, these set of tools, you would just transform, analyze, visualize. And those are each different, different companies. And everybody had their swim names, swim lanes, and everybody was happy. And then along comes like AI and agents. And all of a sudden, like that, you know, the world is going to change. And so trying to predict exactly the way that it's going to change is sort of, it's a recipe for, you know, spectacularly bad predictions, but it doesn't mean, you know, you shouldn't, you shouldn't try.

13:09So, you know, it's also when it's sort of super exciting because that's when the biggest, the biggest opportunities happen when you are in these sort of transitional times. So we started to notice that like, hey, text-to-SQL is getting, you know, it works. It's not necessarily the way we thought it was going to work, you know, but, you know, with an agent, you basically can get very high fidelity. So it's like, okay, so that's happening. And then you realize, okay, this Claude and that agents can really do very good visualizations. and those are only going to get better. So if you can run a query and then you can visualize it, that's a lot of what you have in a BI tool.

13:53So it's like, okay, well, BI, there's really an opportunity to just sort of change how you're doing BI. On the other side of things where you're bringing data in, you're transforming the data, that also seems highly vibe-codable. If you ask Claude, if you point Claude at Salesforce force and say like, Hey, you know, pull my data in. It's going to be able to figure out how to do that. And, and it's going to get better. It's going to, you know, like the, the, the, how long it takes it to do today. It's less than it'll take in the future. And so you kind of think about the data stack and what people are doing and what data engineers do and what data analysts do and sort of like, okay, how does that job change in, you know, when building pipelines can be done from a prompt building, you know, visualizations can be done from a prompt, you know, transformations can be done from a prompt.

14:49Well, what, what you do that, what are the things that are going to go wrong? And, you know, I think, you know, well, people are, you know, the agent's going to build a bad, bad schema, or they're not going to have the right context or they're not going to have the right, you know, then there becomes like this, these other jobs that, you know, humans are going to have to do to, but they're almost like, you know, I mean, just like, you know, they say software engineers become managers of agents. Like I think a data engineer is really going to be sort of a manager of a fleet of agents that are going to be doing, you know, doing data work.

15:25And really the things that they have to do are going to be the, you know, reacting to changes. And I think one of the ways that data engineering is different than software engineering is that it's so much easier for things to break through no fault of your own. It's like, because you have this data that's coming in, data may be changing. You know, there's schema changes or, you know, something may be stalled upstream, or there may be a bug upstream or the format or the distribution of the data changes. And that, you know, changes how you have to visualize it. There's just all these things that can, sort of go wrong, even if you've built the right pipeline, that I think humans are going to have to, you know, somebody's going to have to handle that.

16:09And then you think, well, okay, well, but can some of that be done automatically? It's like, hey, if the schema changed from a, you know, big int to a float, you know, can Claude, you know, go and chase that through, or if a field gets renamed, can Claude chase that through all of your visualizations? And chances are probably yes, Um, so then you, you think about, okay, what do those agents look like? And anyway, so that was sort of the idea behind, you know, I, I called it Watertown after, after, you know, Steve Yaghi's, uh, you know, gas, you know, gas towns a little bit, a little bit tongue in cheek.

16:43And so exactly what are those agents going to be and what are they going to do? And what are the jobs going to be? Who knows? I mean, just sort of like, I think when, you know, in gas town, you know, is, do you need the deacon and the, like all of these specific roles? Like, you know, maybe not, you know, maybe, you know, but I think you're going to need some way to sort of handle, okay, there's a bunch of tasks to do. There's a queue. There's somebody, you know, and then there's like a mechanism to surface things to humans as well. And then again, exciting times, stuff is changing, stuff is moving, stuff is moving quickly.

17:20But I think we're already starting to see some of these things, you know, some of these things happen and something's changed. Like I can't remember the last time I've written SQL and I used to write the most SQL of anybody at the company just because I love to poke at stuff and to sort of be like, okay, well, this is happening, but like what's actually happening? And, you know, but now it's just a, you know, Claude session and so I think that's going to be happening through the, you know, through the rest of the data stack. And it's exciting. and it's not just, I think it's not just like going to make people less, you know, useful.

18:00Like I think you're still going to need people, but it also brings, you know, we call them at Mother Duck, we call them NTDs for non-technical ducks, which is like the people who would otherwise feel like they can't do this, this stuff. They're like, I'm not smart enough. I'm not good enough. I don't have the knowledge or the background to be able to, to do these things, to be able to, answer these questions about what's going on in the business or their sales book or their, you know, their marketing campaigns. And now all of a sudden they can do those. And I think that's really powerful and liberating.

18:34Yeah, exactly. It goes again to like the whole democratizing of the tool. Now all they need to do is know what to ask, which is its own challenge, by the way, understanding what you're looking for and what the query needs to be, what you're really asking for. But that becomes the new barrier, right? Which is exciting because text to SQL, like you said, just really transformed. It's gotten amazing and it's gotten a lot better. Like I remember back in October of last year, I was at a hackathon. It was a data hackathon actually for data agents. Brian Bischoff of Theory Ventures hosted this really wacky hackathon called America's Next Top Modeler, where we had to basically decide or figure out how we wanted to sort over like 10 ,000 unsorted documents and park it files and all sorts of just like unstructured and unstructured data for a fictional company that had several mergers.

19:28And then like adding cruelty to it, he even had like a dusty binder with like old things in it that can contradicted what was in our digital files. And we'd have to consult it and like the real world. So it was like mind boggling. And as part of that, I was struggling with, I remember back then with, with Claude trying to get some basic interactions with SQL where I was comfortable with, where I was like able to actually understand what was going on in the data. And now I just feel like whenever I do work with, you know, modern models to work with data, it's really, really simple and conversational.

20:01Like a personal story for me is like I whenever I work out, I use an app called Heavy and it pushes a web hook for after my workout of all of the different things of what I did. And I actually have that hit a database where it might have an agent that then just looks over it and helps give me like a daily understanding of my workout. and how I'm trending. And it even gives me nudges of like, hey, you haven't like increased that weight in a while. I see you're like, you know, holding back or something. And I've found that really fun and interesting to experiment with the data. But that's just something that's become really accessible to me is like, I guess, a non-technical duck or a non-data engineered duck.

20:39You know, I'm like technical, but I'm not of the data science world. But before I would have never really even thought about trying to do that. But as agents get better at making those SQL queries, like all of a sudden now you just have a rise and these huge or new or ad hoc SQL queries, you get the needs for having these like pipelines that can run it and create it. You know, I understand that y 'all are tackling that problem too with flights. And that's what pipelines are where agents build and they schedule and deploy things because now like you, they can build up that big SQL query and push it.

21:13Right. So like you're understanding like the platform that they need to do their distributed work on. I have a question for you, though, in that world where you have agents deciding what SQL to run and then to put it up into a pipeline to run it. Where do the new gates and the checks fall? Because agents can make so many queries or even mutate the data. And so in engineering and in the Linear B world where we talk about the bottlenecks and the places where the humans put down the gates, that's like code review. That's like a PR, right? But like, there's no PR for like a data or there's not one that's like immediately easy to visualize.

21:50So like, what is that? What is that like on your side? I think at some, to some extent, these things have to be code. These have to be like, and treated like code. I mean, obviously they're code because they, you know, they run, they run stuff, but like, but they have to be treated like code and they have to be, you know, I think there's going to be, you know, review processes and, you know, check, you know, people can submit PRs against, against them. Like, you know, whether it's going to be, you know, AI agents or humans, you know, writing the PRs or reviewing the PRs, like that's certainly, you know, an open question and things are going to be, you know, and that part is going to be changing.

22:31You mentioned that we built this thing called flights at MotherDuck. And so what flights are is really they're just a Python script. And we'll be adding Node as well, but TypeScript that runs on a schedule. And we have some hints around here's how to do good things with MotherDuck. Here's how to connect to certain data sources. These sort of set of guides that make that easy. We set our templates that basically the AI will be able to pick the template or be able to write these scripts. But at the end of the day, it's Python scripts. And I mentioned when we saw that a lot of the data pipelines and data ingestions is sort of heavily, highly bi-codable.

23:18We said, well, how do we – our goal really is to make it easy for people to get data in. because people would start using MotherDuck and we say to them, okay, well, what you need to do is you need to go and sign up for Fivetran and set MotherDuck as your destination and pull in a bunch of data and then you can start using MotherDuck. That's a lot of steps. There's a lot of things you have to get right. And so one of the things this lets us do is we just let you sort of start from in medias res and everybody's watching the Odyssey. It starts in the middle of things. The middle of things is, hey, I have a thing I want to do with my data.

23:58And then, okay, well, then we're going to go back and we're going to figure out where that data is and we're going to figure out how to get it in. I just did that this morning, actually. I'm working on a new workload intensity metric with Claw trying to figure it out. It's quite a complicated query. And so in order to get, you know, rather than having the visualizations to compute that, I wanted to sort of pre-compute for our historical data. And so, you know, it basically kicked off one of these flights and creates one of these flights and did a backfill. And that also is going to be running every hour.

24:33So they'll be able to sort of keep that up to date. But I didn't start out to sort of build a data pipeline. The data pipeline was sort of pulled in by, OK, well, there's this thing that I need to do. this like there's maybe this data that's missing or this is like this computation i i uh i need to do that you know uses some external data or is uh is uh is highly intensive and i think that tends to be how people how people work is that you know the the the end goal isn't the data pipeline the data pipeline is sort of this necessary thing in order to achieve this other this other goal you know i think getting back to the you know getting back to the gates i do think that you know so do visual on the visualization sides we have this you know you know our visualization tool is called dives which is which is similar it's basically you know it's a TypeScript thing that can be created by by you know your favorite LLM you know I use Claude as sort of the generic LLM monitor so you use Claude create this TypeScript file which is a data visualization and Claude is really good at it as you know as a gemini you know and um etc and so you kind of have these like two things you have this this these dives which are type script visualizations you have these uh flights which are python uh you know python scripts you know but those are both code those are both you know things that like you know the way we use them internally is we have a github github repo and we um we set prs against the github repo and then we basically a commit hook that like that synchronizes that to MotherDuck.

26:10And so for us, GitHub is the source of truth. But then that makes sure that we can do reviews. Anybody can see what they are. We can clone them. We can tweak them. But then also those get pushed into the tool so that it's easy to use from the tool. And I feel like that's a good sort of stable way of sort of making that stuff work. It's all code and, you know, treated like code. And so you can use the same mechanisms that you would use for anything, which are, you know, also being geared towards, you know, agents and AI and those things are getting better. I think just the last thing I'll say on this topic, which I realized being a little long-winded, one of the things that we're trying to do is make it so that as these agents and these models get better, the features also get better.

27:11And that which is tricky because often people work on something and as the agent gets better or the LL gets better, it's like, oh, I don't even need to do that. Like that whole thing goes away. But I think if you can build things in the right shape, then as the models get better, these things just get better. So for example, the visualizations are dives. Like they just get better as Claude gets better at building visualizations and gets better at writing SQL. And the flights, the same thing is, you know, get better at like pulling in data from different sources, you know, building, writing Python.

27:44Like those just get better. And synchronizing it to GitHub, like, well, you know, I would get up the whole like developer tools are only going to get better at working with these agents. And it can be tempting to just like, oh, well, the model doesn't do this. And so I'm going to fill this gap. But, you know, chances are the model is going to, as it improves, that gap is going to go away pretty quickly. Your AI software factory is shipping more code, but is it actually delivering more value? Linear B's new guide, Your Software Factory Needs a Context Layer, shows you how to find out. It breaks down why AI adoption alone isn't enough, how to measure effective PR yields, and why cost per effective PR may be the metric that your executives want right now.