1021: How dbt Won Analytics Engineering, with dbt Lab’s CEO Tristan Handy

0:00Jon Krohn:Today's exceptional guest coined the term analytics engineering. He built the tool 100 ,000 data teams rely on and now says agents can compress a million dollar year long data migration into just six weeks. Welcome to episode number 1019 of the Super Data Science Podcast. I'm your host, Jon Krohn. Today's guest is Tristan Handy, CEO and founder of DBT Labs, the company behind, you guessed it, DBT, which is the beloved open source tool that brought software engineering rigor to data transformation. A decade after founding DBT Labs, Tristan is now merging the business with Fivetran and becoming president of the combined company.

0:40Jon Krohn:In this episode, he explains why he turned down acquisition offers for years, how the semantic layer keeps AI agents from confidently getting your metrics wrong, and how skill files let agents do work it used to take humans weeks of training to learn. Enjoy. This episode of Super Data Science is made possible by Anthropic, Notion, and Gouroubi. Tristan, welcome to the Super Data Science podcast. A treat to have you here. Where are you calling it from? Oh, yeah, thanks for having me. I am in the great state of Pennsylvania. I live in the main line outside of Philadelphia. That's right. And your company was originally called Fishtown Analytics.

1:19Jon Krohn:Is that Philadelphia is also known as Fishtown, I guess? Well, Well, Fishtown is a neighborhood in Philadelphia. When I started the company, I lived there. Yeah. Gotcha. Yeah. That makes a lot of sense. I probably should have looked into the facts behind that before asking you. It's been a long time since, but originally it was the biggest cod fishing area on the East Coast. I came to learn. There you go. And so that was a decade ago that you founded Fishtown Analytics. Now, DBT Labs, as most people would know it. You've raised hundreds of millions of dollars over the year, and your platform is trusted by over 100 ,000 teams today.

2:02Jon Krohn:That's a wild journey. In 2016, you articulated a vision that data teams deserve the same rigor and tooling as software teams. And you coined the term analytics engineering. What led you to that insight in 2016? And what has the journey been like to today? It's a good question that I don't often get asked. Why did I think that that was true? It was not like some bolt of lightning insight. The prior company that I worked for had a bunch of smart people that worked at it. I ran marketing. The company was called RJ Metrics. and we were a kind of prior generation BI. We could, in our numbers, see the market shifting in the like 2014, 2015 time horizon.

2:50A lot of the really early adopting like tech forward companies were leaving. So we did a study, a bunch of us on the exec team talked with, I think the grand total was like something like 100 companies. and we found out that they were moving to what eventually became known as the modern data stack. Back then, it was people were storing their data in Amazon Redshift and they were using BI tools that were purpose-built to run on top of Amazon Redshift. But the funny thing is they were using this new technology, but they didn't, like a lot of the problems that they were experiencing were the same problems that they'd experienced before.

3:32So they like new technology, same problems. And you're like, well, what's going on here? Like, what have we failed to learn something important? And, you know, I could I could talk about this for about forever. But the conclusion that I came to, along with some of the other folks on this team, was that we were asking data people to build production systems. The cloud meant that everything was always on and there was an expectation that you could hit the button run in the top right of every dashboard and that it would produce new data and that data was always correct. And so this is really a production software system, but we were not equipping them with the tools to do that.

4:08And so that's kind of where everything flowed from that has happened in my little neck of the woods over the past 10 years.

4:16Jon Krohn:And more recently in 2020, you said that DBT's purpose is to empower analysts as first class owners of another transformation process. So, you know, in 2016, you saw this transformation process, this opportunity to be providing tooling to enable analytics engineering to form it all. And then in more recent years, there's this desire to empower analysts to be transforming whole organizations. It seems like power has shifted as data capabilities, analytics capabilities, AI capabilities have become so important in organizations. There's been this power shift more and more towards analytics type people.

4:59Jon Krohn:And yeah, so how are you balancing empowering us technically without overwhelming us with software engineering complexity? The thing that I think is the most magical about people who call themselves data analysts is that they're not afraid of complexity and they tend to dive in to kind of multifaceted problems. Now, that sounds kind of like high level and vague. But the thing is that data analysts are a role that spans two big disciplines. One is the actual quantitative discipline. You have to actually know something about math and statistics. And I would even put the software engineering part in here, like the kind of more technical mathy brain stuff.

5:49And then on the other side, you have to know about the actual business domains that you're working in. Because a data analyst that doesn't actually know anything about the domain that they're working in is really just an order taker. They're not adding any value beyond their technical capabilities. And especially today, when anyone can have technical capabilities with your agent, those kind of can't stand on their own as a source of value. You know, almost everybody finds their way to the data analyst role in some kind of weirdy to use in Chronic Pathway because you generally become an expert in the one side or the other.

6:29And then you learn the other side on the job. The way that I've always thought about building DBT and is this like idea that we should empower these magical unicorn humans to be able to live up to their potential. potential, but not overwhelm them with technical details at the outset. Now, that's not to say that data analysts are not capable technically, but that's not the first order concern. That's not why somebody becomes a data analyst. Back in 2016, the kind of cool kid way to do data transformations was Spark. And Spark is exactly the wrong tool if you want data analysts to be able to do any of this work because right at the outset, it is just incredibly challenging to get it set up.

7:22The syntax is very complicated. Nobody with a data analyst background knows how to write Spark code at the outset. And so one of the reasons that I chose SQL was to make it accessible to data practitioners, to data analysts. And it kind of goes from there, but the idea inside of dbt is progressive complexity you start out and everything's super simple and as you need it it turns out that there's features that you can take advantage of that help you scale up from there

7:52Jon Krohn:let's double click on that experience of using dbt for our listeners who haven't had it before and it's kind of interesting in a podcast format where we have to be able to have an explanation that's easy for people to understand and potentially an audio only format even if people are watching this video of us talking it's not like just visual cues as to how dbt works so this will maybe test your storytelling abilities, but I'm sure it's a narrative that's well-worn for you over the past decade of running dbt labs. Tell us about the experience of using dbt and how that might be different. Like walk through a user story or two of how it's different from SQL or how you recommend analysts getting into dbt.

8:37Jon Krohn:Just, yeah, fill our brains with what the dbt experience is like. The core idea behind dbt is not particularly new. In fact, in the first five years of the journey, we probably came across literally dozens of, I think maybe 50 plus tools that had been built like this before. So the core idea is not unique. The idea is that databases became really powerful for analytics with the launch of Amazon Redshift, and then successively with things like Google's BigQuery, and Snowflake and Databricks. And increasingly, you could rely on databases that primarily spoke SQL to do work that had previously been done by like Hadoop or Spark.

9:25And so that meant that you could express it in SQL, but data transformation or taking the raw data that comes from your systems of record, whether those are kind of business systems like Salesforce or whether these are like logging systems like Snowplow Analytics or something like that, that raw data being transformed into something that you might actually write a query against as a business user is a many, many stage process. And what you don't want to do is you don't want to have these super, super complicated monolithic transformations, but you want to kind of stage things out. You want to take your raw data and at first make sure the column names are reasonable and that you have basic data guarantees like foreign key integrity and not null constraints and these kinds of stuff.

10:17And so you clean it up just a little bit. And then maybe you start to kind of consolidate a couple of tables together. You join customers and orders so that you can get first purchase date or this kind of thing. And as you go, you maybe get to a place where you have a single table that has all of your customer data in it. But really, that customer data has to be sourced from like 50 different tables because it turns out that customers touch every system that you have. And this process in DBT is accomplished by a series of what are typically like fairly modest SQL select statements. Like ideally, you don't have any select statements in there that are over, I don't know, 100, 120 lines long.

11:03And each of them does something really discrete. And then they form a what's called a directed acyclic graph, a DAG, that goes from left to right. And it does all these subsequent stages of processing. And the magic of dbt is that it makes all of this stuff really simple to implement. Building your first table takes basically nothing. You then type dbt run and dbt manifests all of this stuff in your data platform.

11:32Jon Krohn:I gotcha. So it allows people to be doing kind of in a simple, this is probably going to be an oversimplification and you can correct me on what I get wrong here, but it's allowing people to do similar kinds of things as they might want to do in SQL, but easier and at a larger scale faster. In SQL, if you are writing a select statement, that is something you might do in your BI tool. You want to build a chart. So you write a select statement that does some group by and you get some numbers back and you put them on a chart. And that doesn't persist anywhere. But what DBT does is it actually takes all of these different tables that you have defined and it persists them in your database.

12:17And there's a lot of work involved in that. How do you do that well, et cetera. But it persists them in your database. And that means that when you go to your BI tool or when you load up your analytics agent or wherever you want to consume this data, the data that these front ends have access to has all been really nicely modeled. And it's all there in the database waiting to just be selected from.

12:43Jon Krohn:Cool. And all of this stuff is available open source, right? Mm-hmm. Mm-hmm. So listeners can right now make their way. You can very quickly, we'll obviously have a link in the show notes as well, but it's as quick as typing dbt, three characters into Google and you will make your way. What does dbt stand for, Tristan, actually? Originally, it stood for data build tool. We didn't really know what to call this thing. And so the first person to ever write a commit to dbt needed to give the repo a name. and he was just like i don't know he typed dbt on the repo and at the outset we thought it was kind of a bad name because it i don't know it didn't have any character but uh over the years the dbt community has grown huge and the logo and the brand have a lot of recognition and so we realized that we had kind of missed the boat to ever change the name and this is in fact how people thought of it.

13:45Jon Krohn:Oh yeah. No, I mean, I feel really lucky to have you on the show. To me, the DBT brand is like this kind of rockstar name. And so to have the founder and CEO of DBT Labs on the show, it's a great honor. Speaking of the community that you built, DBT Labs is built around an open source foundation and a community driven identity. Even as the company has expanded commercially through acquisitions like SDF Labs, more recently, you had an all-stock merger agreement with Fivetran that will see you take on the role of president of the combined company, according to our research. So how do you balance this vision of open source and community stewardship with practical realities of managing now kind of multiple large successful commercial organizations?

14:36Well, hopefully they're not distinct commercial organizations, I think they fit together pretty nicely. But there can be, on a kind of tactical level, tension between open source and capitalism. Open source wants things to be free, and free as in beer and free as in speech. But then capitalism wants things to be monetized. And so I think that That is a common way that people who are not deeply involved in open source see the world and they're like, I don't really get it. How does that work? In fact, companies do things all the time that they don't directly monetize. I mean, just for example, the large tech companies in the U.S.

15:26have written academic papers for, I don't know, forever, for as long as I'm aware of. The Google's technology that eventually became Hadoop, they wrote about it in a paper. Google also did a paper, some folks inside of Google wrote a paper called Attention is All You Need, which created the entire transformer revolution that we are living in today that's led to large language models. So this stuff happens all the time and there's very discrete business reasons to want to do it. The commercial justification for having people write open source software is that you get to train an entire industry of practitioners how to do their jobs.

16:15And because, like, trust me, people did not do data work 10 years ago the way that they did today. And we're a big part of how that transition happened. But then they learn how to do it on your tools that you can then, like, sell commercial versions of. So there's a real commercial justification there. But the nice thing that's almost like a side benefit is that it's just like highly motivating as humans to feel like you're making a much bigger impact than your P &L might otherwise show.

16:47Jon Krohn:Machine learning predicts and Gen.ai creates, but neither is built for complex, constrained decisions. That's where mathematical optimization comes in, giving you explainable, trustworthy decisions you can act on with confidence. Girobe is the fastest, most reliable solver organizations rely on for their high-stakes decisions. Want to see it in action? Join the 2026 Girobe Decision Intelligence Summit, September 22nd and 23rd in Las Vegas, for training, expert insights, and Gen.AI-enabled accessibility. Discover why 70 % of the world's leading enterprises trust Gurobi and start achieving optical outcomes yourself.

17:28Jon Krohn:Head to superdatascience.com slash Gurobi for the conference details. That's superdatascience.com slash G-U-R-O-B-I. Cool. Yeah, it sounds like you're striking a great balance. You're certainly enjoying a lot of success. something that we noticed in our research, something that you said a while ago, and I can't, I don't have the exact quote in front of me right now, but we read about how over the years, over this decade of growing DBT Labs into this large organization, you've had various opportunities over the years to be acquired and to basically experience a big wealth event for yourself and probably lots of other people in the business.

18:14Jon Krohn:How do you end up kind of deciding, like, it kind of reminds me of watching HBO's Silicon Valley TV show where they, you know, they're, they're just always, you're always cheering for them to stay independent, to not be acquired, to make their next big leap, um, you know, on their own. And you continue to do that despite presumably the temptation of, you know, the, you know, the capital inflow that could happen through an acquisition. You started off this conversation asking where I was calling in from. And, um, the, honestly, I think that this, this has at least something to do with the answer to this question.

18:55Uh, culturally the East coast and in, you know, specifically Philadelphia does not have the kind of, let's just say like the, the, the culture inside of Philadelphia and the culture inside of San Francisco are like wildly different. Nobody cares how much money I have here beyond some like relatively modest number. The housing prices are, you know, you can like for the prices, you can get a reasonable home in San Francisco. You can like buy a castle in the Philadelphia suburbs. And so I like, I just don't need that much money in my life and generally optimizing for it. And I started a consultancy.

19:36Like all of this is a surprising upside to me. The thing that I've chosen to optimize for over the last decade is creating an impact in the world and primarily an impact on users of our product and data analysts most especially. And so when thinking about any type of corporate transactions, the thing that I've thought about first is how would this impact the users of DBT? And the merger that you mentioned earlier with Fivetran was the first time that something hit the bar for me where I was like, oh, actually, this would be good for users of DBT. generally i think that people don't set out with a specific intent to transform data what they set out with an intent to do is like hey i have some data it's sitting over there i need to like bring it over here and do some stuff with it and so fivetran is the pipes for that and we are the meaning creation part of that and uh you know thousands and thousands of companies use these products together.

20:50I can't remember. I feel like our best estimate was something like, it was like over 10 ,000 companies used these products together already. And so it just made a tremendous amount of sense. Whereas conversations we've had like this in the past might've made sense financially, but they didn't make sense for users.

21:09Jon Krohn:Well, we all thank you for continuing to be concerned first and foremost about this community. And I think it will serve you well in the long run. I think that that kind of drive to be as open source as possible, to be community minded, to continue to grow in a way that allows everyone, all of your users to benefit is going to in the long run be great for you. Probably financially as well, but you don't need to think about it as like your primary goal. You mentioned there in your most recent response how dbt adds meaning. So the fivetrans like pipes and dbt adds meaning. There's a term that came up a lot in our research for dbt labs, which is semantic layer.

21:58Jon Krohn:Do you want to explain how dbt acts as a semantic layer for your data? This is a topic that is particularly hot right now as analytics agents are very in view. The problem that the semantic layer solves is not a problem for small organizations. So if you imagine that you're a part of a whatever, a 20-person company, a 50-person company, you probably don't need a semantic layer. But now imagine that you are Siemens, you're a global company, you have 2000 data engineers that are serving 300 ,000 employees globally, it is not possible to just know the answer to random questions that you might need to know the answer to, without getting meetings together of people that you search for.

22:55in your Outlook phone book. You get everyone together in a room and you say, how should we be measuring this thing? And there's a lot of conversation and everybody's got to figure it out and which table should we be using and all this stuff. And literally that's how big companies have for the past, whatever, 30 years, tried to answer questions like this. That's the process. And the semantic layer is tooling that allows those types of decisions to be made and then stored so that successive people, when they ask those questions, can confidently measure things in the same way twice. They don't have to reconvene the whole group.

23:44It is a technically complex problem area, but the problem that it solves is really an organizational problem. It is how do you scale knowledge to increasingly large groups of people? And that is honestly a tale as old as civilization. I mean, we don't have to go too deep on this, but like as long as people have been organizing together to like figure out how to do stuff, there's been this question of, well, how do we make sure that we know things and that everybody in this organization knows a consistent set of things? And so the semantic layer is an attempt to do that. And it's particularly relevant today because absent these types of cues, how do you measure X thing, AI agents have to try to re-derive that for themselves at every turn.

24:33And oftentimes, they do one of two things, or almost always they do one of two things. One is that they, in re-deriving how do you measure something, they just take a tremendous number of tokens to do that. They just have to look into a lot of stuff and think a lot, and that becomes slow and expensive. And then the other outcome is that they just get it wrong, or they come up with an answer that may be kind of reasonable, but it's actually not the way that your organization measures these things. So the semantic layer extends very, very nicely into a world of agents.

25:08Jon Krohn:Yeah, really cool. It seems like one of the key use cases. And if people aren't aware, that word semantic basically just means understanding, just means like the underlying meaning of some data. And so by having this common playing ground, there's this common lingua franca, this common agreement on what meaning is across data sets, across agents. There is efficiencies, especially across the large organizations that you were describing. They're like 200 ,000 person companies with 2 ,000 data engineers, that kind of thing. Yes, totally. And there are these successive layers of meaning where you start off with raw data, you go to modeled data, then you go to semantic layers, which are typically, they help you understand how to join these tables together and how to measure certain metrics.

26:06But then you can even go one step further. And this is not an area of particular expertise for me, but it's a very interesting conversation happening in the industry is ontologies. So ontologies are another layer of meaning making on top of data that are not just how do you measure a thing, which is typically how we think about the semantic layer. But it's also how do you model business processes? How do you model causation? And I think that most of us do not operate in an environment where it's appropriate to say we need ontologies for this because the world changes quickly and sometimes it's hard to keep up.

26:50But in very specific high value domains, think like genetic research or like famously, this is employed. Ontologies are employed a lot in defense. So I think about these four layers as kind of like the knowledge or meaning hierarchy.

27:08Jon Krohn:Yeah, the ontology thing is a bear and you definitely want to have a system that can be flexible or do it automatically, I suppose. for five years up until a couple of years ago, I was a co-founder and chief data scientist at a business that was working in HR tech. And in HR tech, you're constantly, there's lots of companies out there that make their whole bread and butter on creating these ontologies and updating ontologies so that you have like, okay, what are the skills that make up a data scientist? And then it's this constant, okay, so you have the skill ontology that you're needing to update on a constant basis.

27:44Jon Krohn:is like, oh, now there's PyTorch instead of just TensorFlow, and we've got to add that in. And then whole new disciplines emerge. I might argue that data science is kind of fragmented into lots of these different kinds of specialized careers like AI engineer, LLM engineer, context engineer, all of these kinds of very specific roles. And it's somehow some ontologist's job to figure out how to understand a domain so well that they can manually create these relationships between entities in that ontology, you know, skills to jobs and jobs to different job categories and difficult in a constantly evolving world.