Building the machine that builds the machine (Interview)

28:02Like, where is that? Oh, we don't, we haven't thought about this. What about how it touches that? What about second order effects over here? It's like, okay, you're right. Wow. We have such clarity here, but we don't have full clarity because there's a lot of touch points. You just forget to even think about because you're kind of excited. You got a dopamine rush. You're just going, you know? Yeah. Well, so for example, there's this new set of capabilities we're working on right now, which we do intend to release. And there was a core change that I had to make towards the end of last year that was based on a workload that we had seen that we weren't able to handle.

28:41And I was like, okay, I need to make this change and I use Claude to make it. And I didn't actually then take the time to go through and refactor a bunch of other spots in the code that kind of were impacted by the change. So we got a few more weeks down the line and everything seemed to be fine. And then we started doing more extensive testing. And basically there were these weird failures that only occurred when we were under load or under resource constraints or certain timing issues. We didn't know what it was. Right. And last week, you know, we had multiple engineers, myself included, like trying to figure out what it was.

29:20And of course, all of us are like using agents to try and like help and investigate the problem. And basically by like last Thursday, I was like, man, we're just like spinning our wheels here. It's like everybody is in the AI casino pulling the levers on the AI slot machine, slop machine, hoping, you know, hoping for manna from heaven to come down. And triple sevens, gimme, gimme, gimme, I need. Right, exactly. And the problem is like none of us were actually getting anywhere. And I was like, okay, I'm just going to pull the plug on all this. And I'm just going to go back to basics and audit everything myself by hand.

30:00like personally. And I think on reflection, I think this was due to a few different problems. But before I cover that, that experience tells me, well, like if we are going to take 60 ,000 lines of vibed up code or it's not totally vibed up, right. Cause it's not like there was absolutely no supervision. It wasn't like build me an app and I have no idea what's underneath the thing. but still it's 60 ,000 lines of kind of vibed up code and an engineer has to support it not that it's bad code but Rust is inherently hard and so if you've got 60 ,000 lines of even harder code than Go code, Go code is a little easier than Rust for example yeah it's like if there's a failure and the AIs can't actually solve the problem then an engineer has to dig in and get deep with it and whatever.

30:57And it's like, how do you determine what things you want to support at that level? Because we only have so many support tokens, engineering support tokens that we have to hand out. And the thing is, AI will help scale that, but if you get to the point where the AI can't help and you actually need to do it yourself, then there's that limitation. But how do we... I guess the question for me then became like, how do we avoid this kind of thing? So one, like I said, I'm an absolute believer in a solid verification suite. And I think a verification suite is much, much bigger than unit tests or even integration tests.

31:44It has to be more comprehensive. It's basically what I would have a QA engineer do back in the day. I got my start as a contract tester on Windows 2000 back in 1998 and 1999. And I was a QA engineer. And essentially, I spent my time writing code to create QA environments to test out new builds and test out these different scenarios and stuff like that. I think that's going to become a lot more relevant for engineers this year. Right. They're going to spend a lot more time if they actually want to get the benefits of, you know, agentic coding velocity. They have to spend more time building QA tooling to make sure that what they're cranking out is actually good and robust and all this other stuff.

32:36So I basically like we have a bunch of testing internally, obviously, and we have multiple teams doing it, but we need more. And then the other piece is kind of just like a matter of like functional code organization. So the problem I was working on spanned across, you know, probably like 10 different files in the code base or more. And it spanned across, you know, maybe like over 100 ,000 lines of code. But when you start up Claude or Codex or whatever, they won't read an entire file if it's more than whatever, like 2 ,500 lines. 25 ,000 lines, yeah. No, 2 ,500. Oh, 2 ,500. It's like 25 ,000 tokens.

33:20There's like some sort of number. 25 ,000 is a number it might extrapolate to. I mean, definitely. So any code file that's like 20 ,000 lines of code, it will not read. So basically, it'll sample little pieces. and I think what that leads to again this is just my theory of like I think like the result isn't as good because you only get a sampling of the thing and basically it doesn't have the context it needs to make good choices so I still think you need a lot more you know professional developer involvement when it comes to architecturally how do you organize the code how do you kind of like break down the problem?

34:00Like if you have this big problem that you're trying to solve, how do you break it down into smaller chunks so that you don't have to keep the entire thing in your head at once? Which is important for developers, but it's also important for agents too. So part of this is code files that just, you don't let code files get that big. You break them up ahead of time. You make clear what the invariants of a system are and what the expectations are. And you make it so that the agents can read those out quickly and easily to get the context for anything they're working on within a system. But for our engineering team this year, my expectation...

34:44Everybody's using these tools. And my expectation is we're going to spend a lot of time building QA suites and things like that, that essentially run on developer laptops, but also in the cloud. And all of it is going to be designed such that an agent can kick off the run. An agent can look at the results. It can run validation steps. You can give it like we're building command line tools. They're designed not for humans to run, but for agents to run so that they can validate signals from the database running. Some of them are like black box signals that are accessible via public APIs. some of them are reaching into object store and looking at file states and actually like reading binary files out with these tools.

35:30So yeah, basically building that so that the agents can kick those off, do the validation, do the inspection, make code changes and deploy it and run it again, like in the cloud without an engineer having to do every single one. But also at the same time, making it so that any engineer on our team at any time has a single command to kick off one of these specific tests and look at it and iterate with an agent. This is the year we almost break the database. Let me explain. Where do agents actually store their stuff? They've got vectors, relational data, conversational history, embeddings, and they're hammering the database at speeds that humans just never have done before.

36:16And most teams are duct taping together a Postgres instance, a vector database, maybe elastic search for search. It's a mess. Our friends at Tiger Data looked at this and said, what if the database just understood agents? That's agentic Postgres. It's Postgres built specifically for AI agents, and it combines three things that usually require three separate systems. Native, model context protocol servers, MCP, hybrid search, and zero copy forks. The MCP integration is the clever bit. Your agents can actually talk directly to the database. They can query data, introspect schemas, execute SQL. without you writing fragile glue code, the database essentially becomes a tool your agent can wield safely.

37:04Then there's hybrid search. TaggerData merges vector similarity search with good old keyword search into a SQL query. No separate vector database, no elastic search cluster, semantic and keyword search in one transaction, one engine. Okay, my favorite feature, the forks. Agents can spawn sub-second zero copy database clones for isolated testing. This is not a database they can destroy. It's a fork. It's a copy off of your main production database if you so choose. We're talking a one terabyte database, fort in under one second. Your agent can run destructive experiments in a sandbox without touching production and you only pay for the data that actually changes.

37:48That's how copy on write works. All your agent data, vectors, relational tables, time series metrics, conversational history lives in one queryable engine. It's the elegant simplification that makes you wonder why we've been doing it the hard way for so long. So if you're building with AI agents and you're tired of managing a zoo of data systems, check out our friends at TigerData at TigerData.com. They've got a free trial and a CLI with an MCP server you can download to start experimenting right now. Again, TigerData.com.

38:51roadmap, let alone internal tooling to support the work we're doing. But I think we're now in an era where maybe QA is the most important level to hire for to build internal bespoke tooling to confirm and verify and code review and code quality the plethora of code we can create. And that's going to become the bottleneck. And then obviously supporting it with support. But QA, that process is all about the right bespoke fixtures or, you know, not actual tests in the code suite, but an actual, like if I was a QA agent and I built a tool to test it, let's build those things. And now they're versioned.

39:28They're there in perpetuity. Maybe they're even hosted. Maybe they're in an on-prem situation or a home lab like situation, you know, in the office or maybe even the cloud, if that's what you need to do. But that's the era I think we're in is this bespoke tooling to test and verify and code quality what we're building. Yeah, I think so. for sure. I mean, and the nice thing is like the agents are more than happy to create all that tooling for you. It's just easier than ever to create that. Do you want to test it? Yes or no? And I was like, let's test it. It's like, yes, tab complete that and enter please.

40:02Right. But again, like I, it's funny. Like I've also had the experience where I have some tests and I tell the agent to make some change or whatever. And it does this and then it will change the test so that it's actually like it makes the test pass but not because the code does what it's supposed to do but because it had to change the test expectation so i still think you have to like keep your eye on them yeah 100 i think your engineers will become your qa in terms of i don't think you necessarily need separate people for this because they're well positioned to actually craft those tools. As you said, it's no longer boring plumbing work because you're not doing the work yourself anyways.

40:44It actually can be kind of exciting and fulfilling to create a qualitative analysis tool that can be used across the world. I would agree with that, but I don't think that opinion is widely shared by software developers across the industry. Yeah, I mean, I think the problem is like software development is changing significantly right now, right? And it will this year. And I think it probably will go more towards what you're talking about, which is software developers more and more are going to be responsible for overseeing what's going on with the agents and doing QA and validating that the results are good.

41:24But the thing is, if you talk to most software developers, you know, we got into this industry because we actually liked writing code. And it's for a lot of those people, it's like, they don't want to do that. That's not why they became software developers. They don't want to spend their time validating what somebody else's code is doing. Most people, if you talk to them, what's the thing they hate most about their jobs in terms of being software engineers? They hate doing code reviews. Now it's like the agents have turned us all just into code reviewers. We're just reviewing agent-written code.

41:56Well, I was going to ask you if you included code review in that because you didn't mention anything about code review at this certain point when you have so many validation tools. You know, do you do randomized testing like they do with drug tests? Like, do you have to code review everything? Because that's going to be such a bottleneck. Whereas if you have your 1100 integration tests and then you add onto that thing after thing after thing to verify at a certain point, do you need to do code review? I know today you do. Yeah, I mean, I would love to get to a point where, you know, you're able to triage code and you say like, ah, 70 % of the code doesn't have to be reviewed by a human.

42:34It's enough to have it reviewed by the AI. And if it passes the verification suite, then you're good to go. And then it's just a matter of like you have an AI triage the code as it comes through the pipeline. And it's like, okay, here's the 30 % that actually needs human eyes on it. And not all humans are created or focused on the same areas. So it's like, it's not just it needs human eyes on it. It's like, oh, it needs, you know, this person's eyes on it because they know this area of the code base, which is right. Like code owners kind of thing. But even that, I don't know that we'll get there because the problem is you have to, you have to build up trust in the tools over time.

43:17And really it depends on your risk profile, right? Like what is it you're creating? and the lower, like the things that aren't as critical, you're much more likely to be like, yeah, we'll just, we'll just wave it through. And that's fine. And that's what you're seeing. I think out there in the world is like, you know, people are just creating fun things for themselves or whatever. Yeah, zero users, you know, demo it off. Nothing to lose. Yeah. Yeah. I totally get that. As a software developer who hates code review and always has and probably always will, you know, I'm ready for that one to go by the wayside.

43:49I always like, I have a mixed opinion, a mixed feeling about code review because sometimes I feel like, you know, a lot of times like people are, it's just like people are just hitting okay or whatever. Or they're reviewing it for like stupid, like syntax stuff or whatever. Like it's not actually like reviewing the big picture of like, does this actually do what we want it to do? Not everyone, but like at some point, like sometimes, yeah, even for me, like not all my code reviews are great. Like some of my code reviews, I'm just like, it's on this day. I'm just like, ah, let's just get this through.

44:24Yeah. You know, have you ever seen the meme where it's like, you submit like a 10 line change and like a hundred nitpicks on your code review. And then you submit like a 3000 line edition and it's like LGTM, you know, merge it. Wave it through. Yeah, exactly. I get that. I think there's some truth to that. Yeah. I mean, I, I, I think it's, it's tricky because like for better or worse, like software engineering is changing. And I think people are just going to have to like deal with that. I think for people who was, somebody wrote this the other day. It's like for people who are like, maybe it's Andre Carpathi.

45:00It was just like for people who, you know, fancy that who always loved software engineering because they like to build things. Those people are fine. But for people who like software engineering, because they like the details of writing code, those people are kind of hosed because, you know, those days are are are limited at best right yeah it's weird i guess i'm an em of two minds because i've i've been both people i've i do enjoy crafting a function and like the you know the thought process and the toil and the the win at the end of it where you're like you know what that's a well-factored thing that works as a court and i built that i don't totally love that but i think at the end and i'm but i'm okay not doing it too like i actually kind of just want the end result.

45:45And so I guess I've been on either side of that fence. That being said, I do think developers enjoy making internal tools for themselves and for their colleagues. And so we talk about verification tools. I hope that we'll enjoy the process of coming up with really cool tests and tools and building those at a fast rate without any of the toil. I actually have a theory that I just thought of for that. Because I agree with you. I think developers love building internal tools for themselves and stuff like that. I think one of the reasons for that is because there's so little friction for doing so.

46:22You can create it. And when you toss it up for code review or whatever, like people don't nitpick it because the stakes are lower, right? They're just like, ah, it's internal. It's not going to go out to a customer. That's true. You know, so the problem is when you raise the stakes, when it's like going out to a customer and you really have to worry about it, then it becomes harder to ship. Right. And then, then like your enthusiasm degrades because you're just like, do I really want to struggle with this and get this through? You procrastinate for a couple of days, maybe a week or two, who knows, maybe you don't ship it at all.

46:57You go work on some internal tool instead, you know, exactly. Yeah. Just go do something fun for a moment. It only runs in Chrome, but that's fine because that's the only browser I'm using, you know, like that feels good. I don't have to do all the extra stuff. Yeah. Yeah. I think ultimately sweating those details is still going to be important. Even if all the AIs are doing all sorts of code and stuff like that, it's still important for somebody to sweat the details and make sure everything is smooth and works as expected and all this other stuff. The idea of editing and curating and fine tuning and having taste is of the utmost importance now.

47:35Like you said, if you have engineers who really enjoy the process of creating software for its sake and not just simply laboring over the function or the particulars of the language, you're going to see that be a less thing. but I feel like PMs, product managers, folks who were sort of like developer adjacent, but still sort of developer, their job wasn't really around writing code. It was around helping developers who write the code to understand what to build for the company. I think those folks are well positioned to become engineers, wouldn't you say? Yeah. I mean, I think the people who are like in the best position to take advantage of all this are either product managers, like every product manager is in a great position right now.

48:24Like they've got to be living their best lives because they can actually do stuff. Before they had to wait for an engine, they had to ask, you know, try and sequence everything through the engineering team and wait for an engineer. Now they can just like do things. The problem is whether or not that actually anybody will accept that and get it to production. But I think also like engineers who are like product minded or product focused engineers, like those, those engineers are the ones who are just going to, you know, just run wild with this. Right. The, you know, and I, I think in that case, if you give like product focused engineers, the ability to actually like freely create things and iterate on it, what you'll see is like a small team having a productivity level.

49:08That's like a hundred or a thousand times greater than what you get before. And you'll have teams of like two to four people producing what used to take a team of like 100 engineers, right? Much more time. It's crazy what I think is possible. I have a lot of faith that actually there's still going to be a couple of improvements this year. Even if they just make tool use a little bit better, maybe a little bit longer context. All this stuff is going to keep stacking up. It just gets better and better. So when it comes to building the machine that builds the machine, as you so eloquently put it in your blog post, do you think that's something that every engineering team is going to reinvent inside themselves as they reinvent themselves?

49:58Or do you see vendors coming in and solving these problems? Or do you perhaps see the big model providers, the open AIs and the anthropics actually just leveling up their tool so much that it is the machine at certain point what what are your thoughts on on like where does this reinvention come from so I think all of the above happens just on different timelines okay right so I think right now people like forward-thinking engineering organizations are already building a bunch of custom tooling to do this stuff. And they're either building up a whole set of tools and stuff like that that's designed for agentic use, or they're actually working with the APIs directly and they're creating their own custom agents to do things within their own infrastructure, to do things within their own code base and their own product delivery cycle.

50:49There are also a bunch of startups who are trying to build these things. The most obvious ones are the code review startups and all that kind of stuff. Or on the all the way opposite side of the spectrum, pure vibe-coded application builders, like Lovable and Replit and all these things. But ultimately, I think that the big model providers are going to just continue down the pathway of producing better models, right. That, that have, that are able to process a larger context, ideally, hopefully a larger context. I feel like it's been a little while since we've gotten a good upgrade there, but a bigger context, but they're also going to continue to build agentic tooling, right.

51:35Just because it's so obvious that, that that's what the people want. Like, you know, at this stage, like, you know, Claude code keeps adding more capabilities along the way. Every single one of these agent programs keeps adding more capabilities. So at some point, it's like you don't need to write your own tool because it just does that. And you get to the state, I think you'll get to a point where essentially you have like a knowledge base within a company. It could just be defined as markdown files, right? And the knowledge base says like, here's how you do these things. And you can define processes.

52:10You can define all these different things. And that's consumable by agents. It's consumable by humans. But yeah, I think this year, a lot of people are going to be creating their own and they're going to be a bunch of startups trying to create stuff. But I think within two or three years, maybe a lot of that gets eaten by, you know, Anthropic, Google and OpenAI. Right. Hence the SaaS software apocalypse, you know, that's happening. happening well you gotta feel good as an infrastructure provider at this point right like influx tv that's infrastructure so you're yeah you guys are sitting pretty right uh i mean i feel good in the sense that i have so much uncertainty i mean i i'm generally an optimist so okay and i like i i'm like i'm just excited about all the stuff that's currently possible and i just like want more time to play around with these tools and kind of like create things but I think, yeah, as an infrastructure software provider, we're in a position to not be completely displaced.

53:19But at the same time, we're not necessarily in the AI hype cycle, growth cycle, whatever, right? Because what we're creating is not something that we're not on some sort of weird AI growth curve. Well, you do probably have a lot of folks using InfluxDB. And I know that in the past, you've done a lot of, you've had your own conference, right? You have an annual conference, I think you do a lot of workshops that you participate in and have. So you're educating folks to use InfluxDB in great ways. I'm curious how, you know, while you're not in that AI centric, however you just phrased it, how your CLI and how your API may be reflecting this new world.

54:05Yeah. I mean, so there's this idea I have, and this is like a prototype that I built over the holiday, but the idea I had essentially then, which I still have now, which is, you know, before early days of InfluxDB, when I was thinking, you know, building the thing. And And one of the things I wanted to do was optimize for developer happiness. I was like, if you optimize for developer productivity, make it easy for them to use, easy to create things, the software will win. And that was informed by my experience using Ruby on Rails in the early Rails days. I could create a web app in a fraction of the time that you could with other frameworks.

54:53And Ruby is a language generally, it was designed for developer ergonomics, right? Because Matz cared about creating a language that was a delight to work with. So developer ergonomics were the thing. And for InfluxDB, that's what I modeled. And I took inspiration also from MongoDB because I think MongoDB really nailed developer experience early on, right? They had a data model and a way for working with the database that just made it easier for front-end JavaScript developers to create applications on top of that database. So that's how you win, right? As you focus on developer ergonomics, you make something easy to use, whatever.

55:36So the thought I had in December, which is basically the tail end of last year, is like, I don't know if developer ergonomics matters much anymore because developers aren't going to actually be writing the software. What matters is actually like, how easy is it for an agent to use? And it's funny, actually, Wes McKinney had a post probably like two weeks ago where he mentioned this. He mentioned instead of designing for developer ergonomics, maybe you should be designing for agent ergonomics because that's what's creating the software going forward. And he used as his example, he's been lately writing a bunch of Go code, but he doesn't write it and agent writes it.

56:20He's never been a Go developer before. He's a Python guy or a C guy. So I think that's very true. And when I was thinking about InflexDB, I was like, well, what does it look like if an agent is writing most of the code? One. And two, you want to make it so that people can quickly use whatever code they're having the agent create. And the idea I had there was like, well, then you want the database to essentially be like a platform for running agent written code. So how do you do that? well You need some sort of safety in there. Forks, sandboxes, stuff like that. So the idea was like WASM. WASM is great because you can put a WASM runtime in the database.

57:07You can sandbox it. You can limit the number of CPUs, the amount of RAM. You can limit specific calls that it can make. So I basically prototyped that. So we may experiment with that over the course of the next year. I don't know. Again, that's not getting shit. That's not, that was a side quest to see what's possible. I think what's happening too, though, I mean, you still have developers right here. I mean, developers still care about CLIs. They still care about APIs. And I think your notion of Wasm in a sandbox environment in the database is future thinking. I think you're thinking a year from now, maybe not so much the next six months, you know.

57:44And the only reason I bring this up is because I'm like just laser focused on this idea that, you know, platforms like yours need to pay great close attention to the CLI. Your CLI is essentially a gateway to your API, right? It's the framework that developers use to implement and utilize that. And if you're paying close attention to your CLI and how that's engaged with by a developer that's augmented by an agent, we still have developers. They're still coding by hand. They're still learning things. There's still people leveling up on InfluxDB and how it works and how they can utilize it. But you're also going to have agent-enabled developers who are writing GoCode for the first time, but they're a Python developer, you know what I mean?

58:21And we need to pay close attention to those interfaces and agentify or agentify, you know, agentify, I suppose is probably the right word to use, give them the right kind of tooling that helps a developer be augmented by an agent, utilizing those tools to get started faster, to take influx to new places inside their org. It's the great vertical for you, right? I mean, you can go much further into an org now because you can hand a much better agent enabled CLI coupled with your great API that's, you know, got all the right kind of tests and all the backing behind it. Now they're off to the races.

59:01You know, you've enabled them to go on their own roads at much greater speeds and not have to build new features. You just get to go further in, not the other way. Yeah, I mean, I think the thing is, like, we don't sell like a whole solution, right? Like, we're just like, we're always a part of a component that a developer uses to build a solution, right? Whether it's like in sensor data use cases or monitoring, system monitoring use cases or whatever. And I think where there's a lot of potential is we've often had customers or users asking us to build more application-level features and stuff like that.

59:41Sometimes we've tried to do that stuff, and other times we've said, no, we can't do that. I think all of the agentic tools and coding, what that enables, it makes it just much, much easier for end users to create all of that software to basically create a fully complete solution. And the solution is like, can be tailor-made for each individual organization that it sits in. Right. The thing is, all of that, I believe is like probably way too forward looking because that's not where we are. Right. Like most enterprises are, you know, they're, if they're using agentic tooling, agentic software development, stuff like that.

1:00:21My impression is that most enterprises are doing it just to make an individual software developer a little bit more productive. which is why you hear people talking about, ah, yeah, I got like 10 % gain, 30 % gain in productivity. People are just like guessing. And that's because they're using one session and it's helping them write code a little bit faster, but everything is largely like the same. I think that's going to remain true, but I think this year there will be just a continuing development of essentially like a separation of people who are like all in on it, where they build the machine that builds the machine, right?

1:01:00The machine is basically the software factory, the delivering software. And if you're building the thing where it's like, okay, we can have not one developer running one agent, but we can have one developer managing 15 agents. That's just like writing a bunch of code and delivering everything in the background. Again, I think you'll see a real difference between engineering organizations that are delivering at that kind of velocity versus ones that aren't. But again, I think along the way, what we'll also see is spectacular failures. You will see big security compromises. Like Clawbook? Yeah. Yeah.

1:01:47You will see big like infrastructure failures or, you know, production failures and stuff like that. And where it will be. My bad. Maltbook. Yes. Maltbook. For open claw because it's not MaltBot anymore. That's right. No. That was so last week. So like, that was like, yes, last week. Forever ago. Yeah. So yeah, I, I don't know. It's wild. It's crazy times. Are you thinking about this from a CTO level only on how you enable your engineering department? Because like the question that I was trying to pose there was how do you how does your product change so that your tooling enables the users better, not just your engineering department better?

1:02:31How do you think this changes how you interface with your customer and enabling the users of Influx? I mean, a lot of it is still the same. Like, well, you know, we've already put like ask AI features in our documentation website. And we have like, we're starting to thread this through our support organization. Right. So what I expect is like more and more tooling to allow AI to basically just solve customer problems along the way. But in terms of actual database features, other than making sure that the command line interfaces that we're coming up with and all this tooling that we're coming up with is accessible by agents, at this stage, we're really thinking it's on the individual organizations that are adopting InfluxDB for them to use their tooling.

1:03:22because we're still not trying to, you know, create an agentic platform for creating time series applications, for example, right? Because then we'd be trying to like bring in a model and do this thing. And it's like a bunch of, a bunch of that stuff. So mostly what I'm thinking about it from is like the perspective of like a CTO trying to optimize a software engineering organization. But I do think there's downstream impact. It's just that I don't think we have any particular talent in being able to innovate in that area faster than the big AI companies like OpenAI, Anthropic, Google are going to actually come out with new agentic tools and stuff like that.

1:04:09As those tools and those models get better, they're automatically going to be better at using InfluxDB and better at using our stuff as long as we have proper documentation in Markdown files that agents can consume, command line tooling that the agents can use. I'm not super bullish on MCP just because at this stage, if I can have the agent use a CLI, that's what I use. but yeah I think MCP is a there's a lot of reasons why it's good and a lot of reasons why it's bad I mean in certain cases it makes a ton of sense but in a lot of cases just like I want to just the CLI and make it more agentic friendly you know that's it's a it's a sibling to the CLI yeah I mean I I imagine we'll be adding like more security features to the database where the security features are designed around agents accessing the database versus humans.

1:05:07Because my guess is if you take out basic dashboarding, I assume agents are going to be the most frequent accessor of the database. So the question is, how do you want to lock them down? You want to lock them down so they can't mess up the data that's in there. Maybe you want to lock down visibility in terms of what they can get to. So, but those are all like security and access controls. But the truth is like a lot of that stuff is the same. If you have a very large enterprise organization with a bunch of humans accessing these things. So I feel like, you know, just like I said, like for all the software development practices that mattered before just matter even more.

1:05:50It's the same thing. Like all these features matter more. The only difference is instead of having an organization where you have, you know, a hundred or a thousand humans accessing the thing, you also have a hundred, a thousand, ten thousand agents accessing it and doing it much more frequently because agents can work faster than humans can. How has collaboration changed inside your org as a result of moving faster? Is it like bonkers in real time chat? How do you track tickets? How do you guys get together and do scrum? You know, are you practicing lean? How has your, I guess, day-to-day collab around what to build and how to build it and maybe even pair programming?

1:06:34You know, is that happening? How does it work? Yeah, so we've never been a pair programming shop. I mean, certainly like individual developers will sometimes like contact one of the other developers to like pair on something quickly, but not like, you know, agile pair programming kind of prescriptive kind of thing. initially like in july of last year we started doing like oh we'll have like a an ai day like once a month where it's like you don't have to worry about doing any sort of like product development or stuff like that it's just like pick up some ai tool and do something either fun or whatever and and share it with the rest of the engineering work we also created you know a Slack channel for AI stuff where it's like, okay, share your learnings with whatever tools you're using.

1:07:25And recently we've started sharing skills that are getting developed inside, skills for code review or other kinds of things. So I expect more of that. What I anticipate is we will build a collection of skills for individual tasks. So we have this pipeline for how customers get support. We have our 24 by 7 support org. Things come in. If they can answer it, great. If they can't, they log an issue in a specific GitHub repo that we have called EARS, E-A-R, Engineering Assistance Request. And basically that gets logged and then an engineer looks at it and whatever. So what we haven't done yet is built a set of agent skills to help process those and help engineers dig into stuff.

1:08:10so I expect we'll probably be doing that I'm guessing like a lot more skill development stuff around those specific kinds of workflows right support thing comes in do this if you're trying to do this in a production cluster do this right the kind of stuff where it will like preload as a you know context that's important in a set of tools and just make it easier but yeah I mean the collaboration part is tricky I still, I'm thinking maybe we should be doing like a, you know, like an occasional like engineering demo to each other. Like we also have like a Slack channel where we can post, you know, demo videos of things.

1:08:54But again, this is all kind of like ad hoc. It's not. You're trying to figure it out. Yeah. Yeah. Yeah. The demoing is pretty interesting because, I mean, showing off something because you're probably getting there faster. You probably need more feedback. I'd imagine while you're not vibe coding necessarily like a traditional, maybe even a negatively connotated version of vibe coder is doing it, you're still vibing with the agent. How are you defining those plans? Is there a centralized place for specifications or whatever plan the agent is working on? How do you riff on that yourself to discover and learn and then define and feel secure in it?

1:09:37but then also share it and get feedback. Do you have loops for that currently? How do you do that now? We don't have loops for that currently. Right now, everything just has to go into a GitHub issue. So we use GitHub for all of our issue tracking and stuff like that. We use the Kanban board and whatever. So it's like anything you're working on has to go into a GitHub issue. What I will typically do is I will, before I start something, I will create potentially like a markdown file to track what I'm doing, right to track like the work and i'll come up with like whole design for the thing and whatever um but even that like the problem is like if you put that up so people can review it it is trivial to come up with like a plan that's like you know a few thousand words of with like words and some code snippets and whatever right because you just do some brainstorming with the pretty easy to do that too yeah yeah do some brainstorming with the agent you put together something.

1:10:35And the thing is like, if you actually iterate with it, like you could put together something that is actually like, I think qualitatively pretty good. Right. And it outlines in detail what you're going to do and how you're going to do it and what's important and all these other things. But again, the problem is like, I can do that with an agent in 30 minutes and it'll spin up like this, you know, 2000 word document with all this other stuff. another human to review that it's going to take them an hour or more to read it and digest it so the problem is like if every engineer can do that in a brief amount of time then again like you have this problem of okay it's too easy to produce this stuff and who's going to review it and who's going to like i almost feel like um the 37 signals model of like okay whenever we do a feature, we put two people on it.

1:11:30I think that's almost like the way to go. It's like you have two humans on something. They don't necessarily need to pair program or whatever, but then you actually limit the scope of, okay, you have somebody who can review your stuff closely. And again, like we don't, our engineering team doesn't have that. The thing is we have a bunch of different engineering teams with different responsibilities, right? Some of them are, you know, there's an engineering team that supports V1 of the product, the hosted version and the on-prem version. There's an engineering team that supports V2. There's an engineering team that supports the iteration of V3 that we currently have hosted.

1:12:05There's an engineering team building this new version of V3 that we have as an on-prem product and also hosted in AWS. It's a time stream for InfluxDB. It's actually like an AWS first party product. So every team is like kind of a little bit different in terms of what the scope of their responsibilities are and what they care about. It's tough times. I mean, because you, I was going to ask you too, like you just mentioned it, but your team size, because we just had a conversation with Amel, the same one of our past participants slash animal in JS Party. Sorry, Jared. We talked about just this idea of, you know, you mentioned 37 Signals in that conversation I mentioned getting real.

1:12:49And back in the days of getting real, which was 15 years ago, I'm assuming on the date range of the book release, uh, very pivotal for many developers out there to, you know, to get real, but it defined, I think a four person team plus either a PM or a designer. I forget the exact, uh, prescription, but it was somewhere around four or five was like loosely there. And then Jeff Bezos was famously, uh, calling the, the idea of a two, two pizza team, which depending upon the, you know, your, your appetite for pizza, it was like a four person team in my, in my brain. Right. And so I think you will see, do you agree with that?

1:13:29Like it's probably four people. I don't know. I think, I think Bezos had potentially a larger team in mind, like up to like eight or 10. Okay. Maybe, I don't know. But I mean, the thing is at this stage, I think any more than three is more than you need. I think it's like two or three and that's it. Well, yeah, because I mean, well, if you go back down to enabling one person, that one person can now, in your words from earlier, produce 100x more code. And so any, if you put more than two to maybe three in that kind of sphere, you got now 300x of code output possibility, which I think is great in throughput, but just so much to overwhelm our feeble human minds.

1:14:13You know, like we still only have so much cognitive load we can handle no matter what kind of superstar you are in, you know, what your sleep was like the night before, you know, is your diet on point? Did you have an argument with your spouse or, you know, whatever, you know, did something go wrong in the, in the carpool line, dropping off your kids, like all these things impact our ability to comprehend our day to day. And so I feel like any, any more beyond two, maybe three is probably pushing it. Like I wouldn't do a four person team anymore. I feel like a two person is probably pretty solid.

1:14:42Yeah, I think two or three, mostly just two, though, where you can almost do a PM and an engineer, that kind of thing. Yeah, you need accountability and responsibility. Most of the work at this point is trying to figure out, okay, what is the problem we're actually trying to solve and defining that well? Then you've come up with a rough structure for how to solve it, but then the agent just does all the work. And then the other side of this is verification and iteration. right like verification is like does it actually do what's supposed to do and then the other piece is like is the user experience what we want it to be and then you just have to like tune it and make sure and polish it right because there there's still like a difference between the output of an agent where somebody is iterated with it multiple times to produce a more polished product versus the one-shot example.

1:15:37Interesting times to be in for sure. Have you figured out a way since you've penned this post on building the machine? Are you actively doing anything to build a version of a bespoke machine for you all? Do you have any ideas that you, any side quest plans, any specifications you've aligned? Can you give us a behind the scenes at all if you've got this in motion? Do you think it will be something that somebody builds and delivers to everybody? or will it be bespoke per team? Right now, I think it's bespoke per team. Well, I think it's bespoke per team using whatever agentic tool they want, right?

1:16:12And it's almost like when you hire somebody, a human, you train them on what you want them to do, right? Because whatever they're, software developers, right? You can't land a software developer in any organization and expect them to immediately become productive, right? You have to go through training. They have to familiarize themselves with the code base, whatever. I think the same is going to be true for people taking advantage of agentic tooling, which is the agent lands in an organization and you have to build tooling, build documentation, whatever, to make the agent effective. The difference is with hiring humans, you hire one human and that's it.

1:16:53You've got that one human doing it. Once you've optimized your process within an organization for an agent to deliver things, you can copy the agent a million times. You can spin up a hundred of them, a thousand of them. So what we're doing internally is still right now, we're still very much in the we're building tooling. That is like infrastructure tooling and command line tooling so that both humans and agents can do more verification of does the software do what we expect it to do and how if we have a customer issue that they're reporting, how do we reproduce that? and then put it, roll it into a regression suite that gets rotated.

1:17:40And then as we, as that becomes more and more mature, what I will, what I want us to do is like to build agents that can run in a loop to try and solve some of these problems. But again, like, yeah, that'd be cool. Just like a regression suite that like an agent actively has bounds on and sandbox and like, if it solves it, cool. If it doesn't solve it, then no harm, no foul, right? Just spend money on tokens. And it's now it's a cost consideration, not so much a, you know, it's not the worst case that solves the problem, right? Like, it's a good thing. Well, so for example, like the thing I've told like our engineers and like I told, you know, we had our QBR a few weeks ago with our executive team quarterly business review.

1:18:28And basically I told the execs, I was like, look, like everybody inside the company, has a superpower that people outside of InfluxDB do not have, right? Which is they have access to the code base, the proprietary code base that we have that we sell our customers. If you pair that with like Cloud Code or Gemini CLI or Codex CLI, you can actually do a lot of things. You can answer a lot of questions that people outside the company might have support questions. So basically where I want to get to is, you know, if there's something that comes in for support from a customer, we are able to actually answer that within minutes based on agents doing the work.

1:19:08And you still need support engineers and regular engineers to review things and to build the tooling behind the scenes. But ultimately, where I want to get to is all of that stuff is executed automatically when things come in. And they can use both the code that we have internally that our customers can't see, right? And all the documentation and internal knowledge base that we have, either in previous support requests or issues that we have in closed source repos or documents, markdown files, and code. And they can connect a lot of those dots if you have that corpus of past ticket history, regression history, customer use case, even if you're pending the versions of InfluxDB or your API, for example.

1:19:58So, I mean, that's a lot you can really give it to do a lot of cool automation stuff on top of that or just make the human faster. Like imagine if you can answer a support ticket and 10x the time, you just now solve the problem so much quickly that customer is now more enabled. They have more faith in you and more trust in you. And that's a good thing ultimately. And I'd be remiss not to ask you this question before we begin to tail off the show because we've talked to you in the past about your thoughts on open source licensing, open source in general. You've been a big believer and contributor to and champion of open source in your career.

1:20:39And I'm really curious how you think this shapes or changes the landscape of open source software. From this bespoke world, and maybe it makes more sense just to build the thing we need from scratch and not lean on maybe a library or whatever might be out there. How do you think this will enhance or degrade, if at all, open source software? I mean, I still think there's a ton of value in libraries that are written and tested and hardened over the course of multiple years. So I don't think that's going to go away. I think it's probably, the more complex something is, the more likely it is that you're going to want a library to do it.

1:21:20So like a SQL query engine, for example, that is a very, very complex piece of software, and you would be best picking something off the shelf that's actually written. You're not going to ask an AI to write the thing from scratch, right? Although it certainly could try and probably could produce some fairly passable result, but it's not going to be as performant and as battle-hardened as something that's been developed over the course of years. So I don't think that changes. I think as the AIs get better and better, the scope of things that they can one-shot and just give you improves. So in the various code ecosystems, like NPM or crates or whatever, there's this huge long tail of libraries where it's a few functions or whatever.

1:22:07The value of those is gone. But I think the bigger problem, based on what I've seen from people online is there's now like a deluge of, you know, AI slot pull requests that open source maintainers are having to deal with. And they're basically getting to the point where they have to start shutting down pull requests, like they are accepting them at all, right? They have to be, you know, we'll be open source and whatever, but we're not open contribution because they can't deal with the fact that like, they're just getting inundated with these pull requests that have no thought or time behind them, right?

1:22:46Because it's just too easy for somebody to spin up an agent to say, like, do this and issue a PR. So, you know, maybe that gets better and GitHub actually adds some tooling. You know, I think we've seen a lot of like GitHub anger over the last like few months because they don't seem to be too on top of this at the moment. But yeah, right now that feels like the bigger impact. but ultimately I think open source software is still going to be a thing. It's still going to be important, right? You're still, you're still going to want to adopt soft, well-developed software rather than just try and write it from scratch.

1:23:25Right. Like, you know, who is a cursor may have been able to spin up like a, you know, a browser from scratch, getting the agents to do it, but it's not, it's not chrome it's not firefox it's not standards either yeah i think a lot of that is like you want some of those chrome developer tools uh especially some of the headless chrome stuff that's available like why would you fight that battle why would you recreate it because now you're just having one more surface to to maintain or or to support even yeah i mean i i think you know there's it's really difficult for open source like software companies at this stage i I would expect to see fewer open source software companies, really.

1:24:09Like if people are going to do something, they'll do source available. Or it just doesn't make sense to do open source, really. Like you do open source if it's not like the actual product you're trying to sell, right? So if it's like, you know, if you have a social network and you want to open source these libraries and do whatever, that's fine. but I think more and more for people who build infrastructure software who want to sell it they'll probably be doing source available licenses if anything because you know it's just like there's it's too easy for anybody to pick up the source code and then like run wild with it and do whatever so yeah if you have good docs a good api and a good cli and open source well you just give them all the keys to your kingdom basically like uh the agent can scoop all that up and do most of it for you.

1:24:58What you do buy is, you know, obviously the support mechanism, maybe the uptime, maybe just the burden or the responsibility of uptime, or if it crashes or backups, it's just one more surface area you have to now maintain. But if you have a pretty good ops team, then that's like, sure, let's just support one more thing. Why not? We're already doing 50 other things we're doing over here, a hundred other things. Why not add one more thing to it that important. Yeah, I don't know. I mean, it's funny because like before, I feel like over the last 20 years, you know, a lot of developers did open source in the early days, specifically for like junior developers.

1:25:38They would do open source because it was a way to like get some practice, right? Get some examples of like work that you've done to help like raise your profile, to help you ultimately like get a better job, right? These are baseball cards, our stats. Right. And the problem is like that is kind of taken away. Like that motivation is taken away because, again, it's so easy to produce the code. You can have an agent do it. And the maintainers don't want to take your code anymore because they're getting like slammed with a bunch of like garbage from whoever. so I think what we'll see most likely is a lot of like open source projects that again are like open source but closed contribution and they're developed entirely by one person or they're developed by maybe a very small group of people and if it's good like what I expect is you'll actually see like very complex pieces of software that are very good that have a high quality that are, again, just a couple of developers working with agents to produce a result that is better than what you get if you just asked an agent to write it on its own.

1:26:49So maybe the rise of the single developer open source project. Yeah, I just saw Mitchell Hashimoto was threatening to go close contribution just the other day because he's just fed up with it. Yeah, he was the one I was thinking of for sure. But I've heard that from other people too. I actually have to, I haven't talked to Andrew, Andrew Lamb, who is the PMC for Data Fusion and Parquet and Arrow. He's on our team. And basically like he runs those projects and he's definitely made like rules for PRs, rules for like using AI in your PRs along the way. But I haven't yet heard him complain that he's just getting completely, I mean, he's getting slammed.

1:27:36Before the ASTF, he was getting slammed just because those projects are like, you know, he's so on top of them and he's getting so much like momentum with people contributing. Right. Turns out that produces more pull requests, not less. Yes. Yeah. Kind of the backwards, the backfire of even email. The more you answer your email, people respond back and then they have more email. So that's right. these things compound stoking the fire that's right that's right fire yeah it's interesting uh what what makes someone want to contribute more to to ghosty for example like what what's the groundswell there why is he getting so much why is he inundated with change or suggested change i don't get it why because you get your pet feature you want your feature in yeah and it's easy to have an agent code it for you and you don't have to put a thought into it besides well I tested it real quick and it works so right PR open a PR start advocating for your little pet feature to get in I mean that's my assumption yeah I think like one it is it's like as a project it has like super high visibility right it's super super popular and it's an end user like an end user project right people are using the terminal themselves so yeah maybe they're just trying to submit features that they want maybe they're trying to get recognition or whatever to be like oh yeah the calling card to be like oh yeah i got a pr landed in ghosty and everybody knows this project so yeah yeah there's lots of motivations now the curl bug bounty actually was ended recently because uh too many low value prs around bugs and you know because they're offering real money if you find certain things.

1:29:22And so Daniel Stenberg finally had to just turn that thing off because of similar problems. It's just kind of perverse incentives at the hands of, you know, available co-gen is like, well, I can just co-gen up something and hopefully get some money. I mean, people are going to do that all day. Yeah, I mean, what angle could be, you know, set an AI loose on that bucket and just say, give me all the cream off this crop, so to speak. But at the same time, it's like, well, what's the point if it's just a bunch of, or if it's mostly noise. I mean, if 1 % was signal, would you still want the 1 % given?

1:29:55Yeah, does it become like email spam filtering, right? Right, yeah. Yeah, exactly. Robots checking other robots work, you know. That's where we're headed. That's definitely where we're headed. We're in a wild world. What's over the horizon for you guys? What's happening at Influx that is, you know, this show goes out next week. So we've got a week's time between now and then. And is there anything particularly happening or anything in particular that you want to mention before we close out the show? Yeah, so we're busy at work right now on the 3.9 release of InfluxDB3. The goal, there are a number of things we're launching there, but we have a beta, hopefully a beta of some really big enhancements that we were hoping people to test out.

1:30:41basically that will enable wide sparse tables, many, many thousands of tables like we had in v1 and v2, and better query performance and a bunch of other things. So that's kind of what we're focused on right now is getting that release out the door and then 3.10 after it, which should come towards the end of March. So, yeah. And then later this year, like a whole litany of features that are building on top of that core. Maybe some PromQL? We'll see. We'll see. I'm not making promises on that one because it's easy enough to let AI make the thing and it's hard enough to see whether or not It's like, can we support it?

1:31:34I'm confident by then you'll solve the problem though. I mean, at this stage, it's like, you know, can we support it? Can we give our customers and users like a good experience? Because we don't want to like just toss something out there in the world that isn't going to give them a good experience. We're trying to be a little bit more measured these days. I wonder if you could feature flag that to some degree or take the folks who have asked for it and say, you know what? we're going to give you access to this with these conditions because you want to get this into production, but we need some willing participants, so to speak.

1:32:09Some, some testers, some production testers that are willing to understand there may be warts and we, we don't know. We're just not sure yet. I don't know. Just think out loud here. How has your move to rust been for you? Are you a happier team organization products better this last year? I mean, I love Rust as a language. I mean, I still like Go, even though I haven't really touched Go since 2016. You know, I love Rust as a language. I think it's great. I think it's the best option for creating high-performance server software. I think it's also good with agents, right? Because you have linters and you have the compiler and stuff like that.

1:32:46It's able to do stuff. You can ignore dead code if you need to. It's like, that code is on purpose. Okay, I'm leaving it there. It's a future feature. Just shut up about it. No, it's good. It's good to tell the agent, you're not allowed to use the allow dead code flag. You can't do that. Well, see, I'm not improv with this stuff. I'm just playing with this. So I let it go. And in certain cases, like I got like five dead codes in one of my projects. I'm like, that's cool with me. I know what they're there for and I'm okay with it. Fair, fair. Yeah, I love the clippy lints. It gives you, I think lints are also a great signal to give to the agent.

1:33:27to make improvements. Clippy Lentz. I agree. All right. Anything left here? What's left on our plate? Anything at all that we can say? Have we said it all? There's so much to go out and do and figure out. I would love to have Paul back on in a year and just see how much they've accomplished and things have changed. Because I feel like inside InfluxDB, there's going to be some serious machine building going on. And I'd love to know where they get in a year's time. But other than that, no, I don't think so. I agree. The machine building and when you officially launched PromQL, I feel like you can come back on here and celebrate that with us.

1:34:05Like what was the journey to take a fully AI generated feature you've never thought of to use a test suite to build upon? I mean, that's, I'd love to like look back at how you went from not fully, full conviction to full conviction. Yeah. Yeah. I don't know. It's going to be an interesting year. Like I said, all the engineering leaders I know are kind of like having the same struggle internally within their companies. It's just like very, very turbulent right now. All right. We'll leave it there. We'll see you in three months, maybe six months, maybe one year. We'll see. We'll see, Paul. We'll get you back on soon the last time, though.

1:34:44We'll see what happens. Yeah. Maybe I'll have something interesting to say in another three to six months. Who knows? There you go. Who knows where we'll be? Who knows what the A &M is taking off our plate? Who knows? Who knows? We have lots of free time. All righty. Thanks, guys. Thanks, Paul. Thank you.

1:35:04All right. That is our Change Log interview for this week. Thanks for listening. And thanks to Paul for stopping by the show again. So much hard-earned wisdom, and it's clear he's on a similar journey that we're all on. It's nice being on this journey together, isn't it? Kind of like that line from Piano Man. They're sharing a drink they call loneliness, but it's better than drinking alone. Okay, that got somber there in a hurry. Let's thank our partners at Fly.io and our beatmaster, Breakmaster Cylinder. Okay, that's all for me. This one's done. But come back on Friday when we are joined by our old friend Brett Cannon.

1:35:37A conversation that's supposed to be about Python and stuff like that, but starts out with a deep dive on. Mmm, Star Wars. Talk to you then.

1:35:59Thank you.

1:36:27Game on!