Your agents are starving! Airbyte’s Michel Tricot on the data ingestion crisis

0:02Welcome back to Dev Interrupted, brought to you by Linear B. My guest today is Michel Tricot, CEO and co-founder of Airbyte. And he puts his finger on the whole SaaS data sprawl problem. Can your systems tell that Alice in GitHub, Alice in Jira, and Alice in Salesforce are all the same Alice? Most agents can't answer that. And if your orgs can't, then that makes coordination tricky at best, especially across platforms where your context lives. And reconciling who and who is a major part of understanding how even your engineering org works. And that's why we've built that layer here at Linear B.

0:43So mapping action across tools and context is how you prove outcomes. And we're about to learn strategies about how to achieve that from Michel. So Michel, it's really great to have you here. Very good to be here, Andrew. Thank you for having me. Amazing. Well, I want to kick things off by talking about something that I'm maybe calling the ingestion crisis, because as a member of the tech journalism and media community, I'm either responsible for creating a new hype or saying that something is dead. Right. So I'm going to say this right now, the ingestion crisis for data. So there's a lot at stake when it comes to providing data to agents because, you know, you are what you eat or in your agent's case, they are what they read.

1:26And so there's a lot of complexity in what you provide to your agents that allows it to operate at scale and safely and reliably. And so much of that comes down to pulling data and things out of all sorts of silos and services and applications all across the world, things that are external to your company. For the most part, you have to reclaim your data from other people's hands. And then in doing so, it's messy and no one can agree on how and what or why and best practices. So this just becomes a huge sprawl of connectors and tubes and pipes, and it just becomes impossibly hard to understand and work with.

2:06And I think that this is a huge contributor to problems with AI that works in production and at scale for teams. They can't operate on that data reliably and cleanly. So I want to hear it from an expert and from your position and vantage from what you do at Airbyte. What do you think about this complexity sprawl? And how do we start to get a control over all of those data sources so our agents can act reliably? There is a crisis on data access. It's not a new problem. Data access has always been a big, big topic, like ever since we invented computers. Now, when it comes to agents and living in an agent-equal, the difference is that suddenly you can scale massively like automatizing tasks.

3:02And it means that, you know, like the standard, you ETL into your warehouse and then you do some analytics on it. it's still a valid model in an agentic world, but you have a lot more new use case, like very similar to some of the use case we had where you have an application that needs to talk to another application. That's great. But at that point, you should really start thinking about your agent as an entity that is going to become fully self-sufficient. And at that point, the real issue you have when it comes to data, which is something we actually have kind of fixed with people is that you need to understand and you need to keep the control over who is accessing the data, who can access the data, at what pace, at what kind of operations they can do.

3:53Because normally you expect some human judgment, but now suddenly you have to trust agent judgment. And also when should the agent bring that decision back to an actual human? And I think that's really what's happening. and there is this piece around you still need to have access to databases, but you need to also have access to live systems. Like it can be a SaaS service, it can be an internal API, it can be an internal service. And this is really where a lot has to be done when it comes to ingestion. Right. It's about like a real-time ingestion. It's about looking at historical information. So understanding that the agent can operate much faster and quicker and make decisions and potentially harmful operations at a scale that just exceeds what humans can.

4:42So the scoping of permissions becomes much more of a requirement. You said it is for humans. It totally is for them as well. But even more so, that becomes the clamp that lets you control. And the other piece is the more system you're connecting your agent to, the more intelligence you need to be able to develop in terms of how do you map different pieces of data across this system? So how do I know that Andrew on Zendesk is the same person as Andrew on Stripe is the same person as Andrew in my product database? And, you know, normally if it's a human doing it, well, you exercise judgment when you do that.

5:24But what you want is you want to have autonomous agents. So you need to figure out like what are the rules, What is the logic that governs all these different links that are established between these systems? Yeah, completely. That's a huge challenge that we've tackled at Linear B2, that idea of attributing that same user across the different seats of the different platforms they use. Because it's one thing to just ingest the platform and ETL it and figure out how do we normalize it into whatever place we want to put it. it's another thing to understand the activity and how it correlates to one person or one entity and more importantly lets you track the life cycle of things because more often than not applications that if someone's touching or has seats in multiple applications it's because those multiple applications represent stages of probably some pipeline they move through reliably think of like going from figma and then you're on the other hand cicd like there's a story there and so you want to be able as a as a product manager a product leader have accurate forecasting and understanding on your teams like their capacity on making and producing code but then also on like the like the jira in the figma side and just getting the actual um understanding of the process right so there's like a huge and and then that itself is a new layer of context now you have this context layer that sits on top of all your operators that knows all the tools they're using So it's a higher order of context.

6:51And it brings me to, I guess, one of the issues that I wanted to kind of dig into as well is that, like, it can become a context sprawl. Like what you alluded to, there's like a complexity that definitely increases. And as you provide more of it to the agent, you also have to be thinking about how do you provide it to that agent just in time and exactly what it needs and about being precise. So you can have like long running operators and such. So what do you think about that kind of balance? That's typically a place where this concept of entity resolution, so creating this linkage between entity, becomes extremely important.

7:34And actually the quality of it and the self-learning of it. Like ideally, it should not be a human that tells, that creates this link. it should be the agents being able to discover across different silos and figure out, okay, these types of entities are very likely connected. Of course, after that, you put that, you log that when you've done matches and things like that. But if you want to avoid any kind of bloating your context, it is really about how accurate the data is for the task at end. So it comes from just retrieving the records that you actually care about, but it's also about understanding what type of records you need to have access to in order to perform that task.

8:15And not, you know, we had something similar with MCPs when it just started. Like, the client would just load 100 % of all the tools available that would blow the context completely. And at some point we started to do, okay, no, let's be a little bit smarter and start searching for what tools are available. And that is exactly the same thing. Like, you go, like, the agent has to go through this discovery phase around like if I'm asking like give me the the the life cycle of that customer on that specific feature first you need to have the agents to know okay what is that feature because that's the first part you start with and then you start discovering okay this feature is associated with this gerr thing it's mentioned in this zendesk ticket and also it's filtered by this particular customer so this is really like the piece around the discovery of what data and how you are actually going to be working through all these different stores that make it so that your context remains extremely like lean and relevant and accurate.

9:19Yeah, no, this identity component actually becomes a really key part of controlling the schema of what things need, because it makes me think of even right now, you know, you make the joke about MCP blowing at everyone's context. MCP has had a run in terms of how the industry has used it and misused it and has been mischaracterized and it's reached now a different form because it just reached its 728 spec finalization or 1.0 so now it's totally stateless and you know they listened to changes that needed to happen to make it more economical at scale but also too there's the cool thing that's happening of serving skills over mcp mcp starting to become a conduit with like mcp plugins where you can define what you're kind of saying like a user or an entity or an identity and say these are its skills and then this is its MCP server or these are the specific tools that are available for it.

10:13And so you start to get these strategies that emerge where you can define the consumers of data, the actors on that data, and then you can better scope it to permissions and what things need. Because I think that's often, often with agents is the most difficult part of trying to wrangle them. I speak from like my own experiences as someone who builds like agents when I do build them. it's so tempting and sometimes it even is easiest to start this way to give them access to everything in a very sandboxed way and then ask it what you're trying to do and see what it does and what it grabs for and figure out that golden path and then cut everything else away and find what's left like oh they really only needed these few tool calls and this kind of pathing or this kind of thing and then there's really cool tooling now like uh like you can use agent gateway to create a very specific slice of like an mcp server that has just these things and you could do that from a schema so your workflow could become an mcp schema that's exactly defined for what the agent needs and then that feels like a great way to work but then it's also like to start you have to give it the world first that's not going to scale so like how do you think about of finding the right level permissions you need for agents yeah i mean at the end of the day building an agent is very similar to a it's a learning process so you need to let it roam free initially understand what's happening and then narrow down the thing the like the perimeter in which it's operating or if you want to narrow it down there need to be an escape hatch for the agents to figure out like this is a piece of this is a system that i don't have access to that i believe can be relevant for what I have to do.

12:01And that's a moment when, well, the agent can actually ask permission for it. You know, we've heard about OpenAI hacking, hugging face. That's basically what happened is just you narrow down what the agent can actually do. And you're blocking access to internet, you're blocking access to everything. And if you don't have the right boundaries in place, well, it's going to figure out way to get out and well mess mess with hugging face but that's that's that's very much the same the same thing ideally you should be having being in living in a world where you're not just blocking based on your first iteration but you can actually get feedback from the agents about what is missing and that's a place where actually metadata about systems that exist should remain available not the data itself so that the agent can actually figure out, oh, you're preventing me from accessing Salesforce, but you're asking me about sales results.

13:09I think I should have access to Salesforce. Please grant it. And then that becomes part of a new grant that you're providing, or maybe it's a temporary grant, but that's really probably the place that we need to provide. It's very similar to a human. I don't have access to a specific report, but I needed to do my work. Well, what am I going to do? I'm going to ask permission to the admin of the platform to give me access to the report. Yeah, no, there's a perfect parallel there. It does make perfect sense that, but in order for that to work, the agent has to be aware of what it can't touch, things outside of its boundary, which is a key, it's an important distinction.

13:49And it goes back to the idea of if you put it in the jail, If you take everything away, if you strip away the resources, then it doesn't know what it can or can't ask for. And it can deviate into more extremes on its path of trying to solve your problem. So if you want the happy golden path, you want to make it the easiest way for it to flow. So the easiest way, if anyone, a human encounters an obstacle is they figure out, you know, who do I need to ask for permission? And so creating an infrastructure where agents can raise questions and feedback and gaps, that's something that I have found works for me.

14:26Like when I have an agent that maybe uses a bunch of data sources to put together a report or an analysis, like think of like a customer health thing, right? At the end of that process, part of the report is the agent reflecting on things that could have contributed to the report that maybe exist or don't exist. It doesn't know. But like if we were to know that exists, now we know, oh, this is a gap the agent has. And yours seems like even one step further, like a registry. Like this is the world that's available to you or could be if you can make the case for it. Yeah. And to me, that's also, you know, you are talking about like the ingestion crisis.

15:05That is something that requires additional investment from companies is this documentation and registry of what is available, what it can be used for, what type of data is available inside, and make sure that this type of systems and this type of metadata are made available to agents. The last thing we want, I mean, the ultimate goal is to have autonomous agents, like entities that behave and are going to perform a task with as little limitation as possible. But we don't know everything that's when you start building your agents, you don't know what it's going to do. So you need to provide this path for like self-learning and self-improvement, upgrading your memory, upgrading your skills, et cetera, et cetera, so that the next iteration is even better.

15:55Yeah. You know, let's talk a little bit too about that ingestion crisis. Going back to it is part of the issue that we've explored is what you do and how you work with the data and how agents consume it and how you manage that complexity. But there's an even earlier problem or part of this problem that's even bigger, and that's trying to get the data into your system in the first place from all the different providers and places that they live. and the reality is is that everywhere that you know bakes and provides data to you kind of has a different policy in a different way that they want to serve it and there's different ways that you can get it both in real time and maybe on a lag or on a schedule most of the time you have to have your own kind of tooling to either receive or to grab it and this becomes the whole you know etl world where you're trying to pull that data and figure out how to consume it into all of the building blocks that you and I were just talking about that real engineers want to work with and enable their teams with.

16:52So what is the state of the vendor world right now and the way that they push data out? Because everyone's acknowledged that their new consumers of their data are agents. So there's a proliferation to push everything to the front. What is that like from an ingestion standpoint? What's that like from a handling the noise standpoint? I'm curious from your perspective. So I think it always goes back to discovery. It's like, how do you let and make sure that an agent can actually discover what is available and provide accurate data whenever. So at the end of the day, every data problem that an agent has is just a search problem.

17:35So how do you make sure that the data that is made of, you know, that's also like a rag is actually an example of that. Right is a search problem. It's just I have unstructured data. I want to be able to search. How do I index it? And it's very similar for any type of data. Now, the problem that exists, and we need to see how the industry evolves, is so many systems don't offer any kind of search primitive. If you want to connect to Gong, if you're using Gong for calls, Well, I can tell you that getting the list of transcripts for Andrew is going to force me to iterate through pages and pages and pages of data before I can find the transcript that is relevant.

18:25And so why I do that? Well, I'm missing my context. And so that's the moment when you actually need to do something that is smarter, that allows you to do that search. And the search is not just for unstructured data. It can also be that, well, maybe you're using a payment provider that doesn't allow you to search for transactions that have this specific amount. And that's a big problem because that's what the agent will need. And that's when we go back to ETL. ETL is not just an analytics thing. It's just loading data into a place where you can build big processes to process that data, whether it's search, whether it's analytics.

19:05and well that's that to me is something that is absolutely key. Now will providers of this data want to keep like wall garden of the data and making it accessible or do they want to charge for it or do they want to own their own retrieval agents that they sell that is what's going to happen like Salesforce, HubSpot Gong are they going to offer like free fledged search features or are they going to monetize that or are they just going to say actually you should run your agent on my platform and that that is really the like i'm monitoring it very close very closely because that's a big value prop of the context that we're building which is this ability to just ingest the necessary data the necessary metadata the necessary permission so that suddenly you have a place where your agents can just roam free, figure out the data, applying permissioning, et cetera, et cetera.

20:06Yeah, it becomes like a fief stone. Like everyone has their slice of your data and they realize that there's now several layers that they can operate on top of it. Like they can provide it to you and add more of that. They enrich it. They can pay for the egress or the ingestion or the movement of that data or services on top of it, like you're describing apps and agents, we're definitely seeing this where SaaS companies and portals were places where we would go typically to, you know, pivot and look over data or otherwise interact with it. Now we're becoming fully agentic assistants and they're trying to be the place.

20:43They want other data to go into it instead of it going somewhere else. And everyone's trying to have a stake in all the different parts. So it becomes like, does the, so then it becomes the question for leaders of, do I draw all the little highways between all of these services and let them all talk to each other? Or do I pull them all into my own layer and trust that I know best of how to do all of those things? Because the real challenge actually becomes, and this even ties back to the build versus buy narrative that's really strong right now in SaaS is like, do I trust them to be better domain experts about what to do with that?

21:19data because they do it for bigger customers than me and way more customers than me. I've seen every edge case because that could be a reality or do or or is like the opposite true where we have the domain expertise and we've always been beholden to it. And this is now our opportunity to take that in house because we can own the process once and for all. So like what does it look like from your perspective when you work with teams that are in this positions? Yeah, I think it's a matter of of of maturity. You know, there is a lot of marketing analytics tools that exist in the market. And in general, they're opinionated about what is marketing, what analytics look like, what you need to be tracking.

22:01The moment you start introducing small, small deviation in like how you're doing attribution, how you are tracking emails, how you're importing lists and things like that. that's when this type of like full fully fledged product that are turnkey falls through and that's why we're using warehouses this day is because at the end of the day you will need an escape hatch and your escape hatch is the raw data and the moment you start getting hooked on the raw data becomes very very hard to actually go for a specialized platform because you know that the The moment you plug it on your data, there will be some issues.

22:48There will be things that are not taken into account. Now, with agent-to-agent protocol, maybe that's a solution where providers just provide this type of interface that your agents can connect to. But they run the agents, so they run the retrieval of the data, the replication of the data, or whatever they're doing with the data. but you still have the ability to inject specific business knowledge as part of the agent roaming free on the data, but within the world garden. But I think today there is so much unknown and not all companies are working at the same pace that people just want to go down to fundamentals, tools, which is just give me the data and let me do my thing.

23:39Let me learn about it. And let me also train my organization to just build agents that are actually going to be able to leverage that data. Your AI software factory is shipping more code, but is it actually delivering more value? Linear B's new guide, Your Software Factory Needs a Context Layer, shows you how to find out. It breaks down why AI adoption alone isn't enough, how to measure effective PR yields and why cost-per-effective PR may be the metric that your executives want right now. You'll also get practical ways to connect source control, issues, CI, deployments, and AI signals into one feedback loop.

24:16Download the free guide from Linear B today. And now, you know, we've covered the different parts of where the data is and how it moves. Once it gets inside the company and teams are able to work with it, you know, what are some strategies that you see work for engineering leaders to make the data that they can pull in from other providers and sources and make that available in a way where all teams can experiment and make new workflows and share things. Do you see successful strategies on the ingestion and success story side of the data is in-house? I think there are already some very solid foundations that are still very much human-driven.

24:55You talk about semantic layer, you talk about golden table and things like that. So that is a good starting point. But ideally, there should be a way that a human doesn't have to do that. And there is a way for an agent to actually learn about the company business, the different rules based on, hey, someone, my head of finance, takes me and says, hey, Michel, I think we need to change this particular piece in the ARR. Say yes. So do I really want him to go to the data team and say, oh, please change the model so that the table is generated with a new thing? Or should just that become part of your context that gets applied whenever you're querying the data?

25:39And I mean, that's the world I want to live in, where this type of decisions are just being encoded directly at data compute time. But we're not there yet. Clearly not. There is a lot of infrastructure that needs to be built. And that's why people today are looking to get access to the raw data so that they can understand what they can do with it and how they can adapt their data processes with actually agents working on top of it. But yeah, right now, semantic layer is great because it contains all the definition of your company models and your company operating models. Skills are also a good solution where you can describe specific rules, specific tables you can encode that logic but skills remain a little bit unpredictable in a way so the question then becomes okay do you go from a skill to a query that you can then review or do you just apply the skill directly and let it roam free and create uh sql queries or api calls to a system yeah we're still living in between these two worlds but we're seeing that people are just trying to move in that direction, which is how do I do the minimum amount of work, manual work, to actually create that light?

26:57Yeah, there's a really interesting synergy that happens between data query and skills and iterating on them in particular, because it goes back to even what you were saying earlier about it knowing and not knowing what it knows and being able to access the world of info that it needs. The same for iterating with data and understanding the best way to work through a workflow, especially if it's a really data heavy one. And then like encoding that process in a skill is helpful. The unpredictability emerges from the traditional problems around like, how do we scale and share this reliably and prevent the one on Fred machines from getting out of sync for the one on Alice's machine.

27:33And like, so there's an MCP solves some of that, but not all of it still. So there's like a lot of, so a lot of like operationalizing problems. I do think that going back to what you said about, you know, how do you make changes? Oh, you don't go to the data team, you change the process itself. And I think that's still a new reality that that teams are grappling with. There was a really standout quote, I think it was from someone on the anthropic team that talked about how they make decisions. And it was about how they spend so much time talking and aligning before they take any action. Like alignment is the obstacle now.

28:10I think if you and your CFO or whoever agree on how the modeling needs to be or what needs to be expected, that is the job. That is the work that has to happen now. Because truly, if you have the data, if you have the agents, then that intent can quickly get inflected into the change without having to become a burden on someone else's plate. And what used to be like you'd pivot in your chair to a person can just be like a delegation that happens by matter of the process. Yeah. And to me, data drives a lot of decision or drives a lot of information, but it has always been bought on teams that generally are a cost center.

28:54So very hard to grow and that have their own roadmap. You know, a few years ago, for example, I was working in an ad tech company. And we didn't have a warehouse. It was when Redshift was starting. So a lot of the query and data analysis that we were doing was big Hadoop job running on terabytes and terabytes of data. And one question could take two or three hours to run. And so it means that when you're in that situation, to be clear, we're still in that situation. It's just that the bottleneck is not that we don't have a warehouse is that a human needs to actually run the query build a query right review it yeah review it and the problem is like whenever you have a question in general you are going to want to follow up you might want to might lead to like 10 other questions or 10 other side quests and the moment for example i introduced redshift at five ramp at the time And it completely unlocked the amount and the depth at which we were able to analyze that.

30:07And to me, I mean, at the time, I was, in a way, like the guardian of that warehouse, the quality, etc., etc. We made it better, but we still have bottlenecks. But the moment you can start providing that and getting an agent to do it based on, oh, you know what? just tell me like i don't know like how many publishers are giving me more than x thousand requests per month if you don't have a table well someone have to build it but now suddenly you can just write that query that request sorry not even a query and with all the learning about your business like the agents will be able to and that just unlocks the amount and the depth at which you can make the decision.

30:54Yeah. There's another interesting change that happens too in that world. Maybe this is one you can speak to from your perspective is like, ultimately what happens is traditionally data ingestion, data warehousing, the data that everything runs on is kind of more like, it seems like an IT kind of thing, a cost center. It's like maintenance and pipes and just like those types of things. But now there's actually such a huge opportunity for it to become a really a profit driver and a part of the profit center because it helps the teams optimize and improve the value add for customers or what the organization delivers.

31:32And data, to be clear, always did that. And every part of that layer always did. But the perception became this cost center because not only were they, you know, the executives are pretty far removed from the programming world. They're even further removed from the data world. So there's like an alienation of the mindset. But now because data and natural language and the connection of them has never been closer, they actually feel closer to their data than ever before. It's now cool to go up to your data guy and be like, hey, what's going on with your data? I think it's like every C-suite wants to be in that position because they know that data is king.

32:11So what do you think about that change in mindset? I think data is going to become more accessible as a productivity tool. So let me develop here. An exec, if they have, like, one of the reasons why there is this, it's a cost center, et cetera, et cetera, I don't always see the value, et cetera, et cetera, of why I have data is also because of the latency at which they are able, like, we are able to get value out of it. And the problem is, like, that latency also translates into why do I even need that? Because actually I'm not going to go as deep as I need. So it feels like it is just an add-on for me to make a decision, but it's not fully driving my decision.