Brex’s AI Hail Mary — With CTO James Reggio

27:29Yeah, and it's a secret fourth pillar, the platform. Yeah, exactly. That is the thing that ties it all together, exactly, is the platform. And I think what's been really nice is that even though the platform is kind of a loose term because it consists of a wide variety of technologies, as I said, we haven't been too religious or dogmatic about everybody needed to be on one particular thing. what we've seen is that by making a variety of sort of ergonomic options for building with loms available it like really has made it easier for for us to make a quick leap forward on operational ai like as soon as we put our mind to it we said like look no we want to hit 80 percent automated acceptance rate for all all startup and commercial businesses that apply for brex like we want a decision within 60 seconds it's fully touchless no humans involved we were able to break that down and then actually build the agents, build the tools on top of that platform really quickly.

28:26And a lot of those tools are the same tools that our product AI agents use as well. I was pretty sold on the conductor. I don't know if this is under exactly that bucket, the conductor one provisioning command. I was like, yep, I want that. Yeah, that was actually, I'd love to talk about that. So that's actually on the corporate side. And I think that this goes back to maybe another intuitive, but I'd say like bold decision that we made, which is that we're not going to try to pick winners in the horse race between the foundational model providers or the agentic coding tools or basically anywhere where there's an active horse race.

29:00What we do instead of trying to pick a single solution is we will procure a small number of seats, multiple solutions, and then we'll give employees the ability to pick whatever one they want to use. And so, for instance, like we allow employees to basically go to Slack and use Conductor 1 to get a chat GPT, a Cloud or a Gemini license. And basically you can just like build your own stack where you pick your chat provider as a dev. You can pick, you know, between like cursor, windsurf, Cloud code, credits, like, and you can basically craft your stack to your preference and easily switch between them.

29:38And what that does for us, too, is when we're going to, like, obviously we have sort of enterprise agreements in place for all of them for the sake of like the, you know, the privacy and non-training guarantees. But it's fun because when we go to renew these contracts, we can basically resist the need to like do a wall-to-wall deployment. We can say, hey, look, like usage trends, our employees are voting with their fee, they're voting with their dollars. And, you know, maybe your tool isn't as hot as it was a year ago. Does it give you a dashboard of what people are choosing? Yeah, actually, we were looking at that as we're going into budgeting over next year.

30:12It was very interesting. I would love to see that. What's, you know, anything that's really up? Anything that's really down? It's fascinating how different the landscape is every three months. And I think one of the interesting challenges we had early on was getting folks to just, like, try these tools, try to incorporate, like, authentic coding. You know, like, early on, I say, like, 12 to 18 months ago now, like get folks to just take the time to try a new workflow and now at this point i think what we're seeing is like even if you know uh a new model hits the same like um when when codex uh came out and everybody's like oh codex is is better at uh code jam but it's a little bit slower like i find fewer folks are like kicking the tires on new things because like the they're just so comfortable with the ergonomics of their current workflow that that um you know uh some folks are I'm just like, I'm going to stick with Cloud Code because I know it now.

31:08I've been working with it for like nine months, so I don't need to keep switching. I don't feel the incessant need to keep trying new things because I'm an iPhone person, and I'm just going to stay with an iPhone even though there's some really sexy Android hardware out there. Do you have one of the big numbers, like 80 % of all of our code is written by AI? But how do you measure it internally? Yeah, no, not really. I mean, what we do is we'll measure the attributions on the number of commits that I have them co-authored with. And we pull some of those stats, but I don't index heavily. In fact, I don't index on those at all.

31:45And honestly, I don't know how I honestly calculate that number. Yeah, I agree. Yeah, and so the thing that we're really just, you know, we're at the point now with our AI agentic coding journey where now we're trying to solve the second-order effects of a little bit too much slop, maybe a little not enough, yeah, exactly, not enough rigor and code reviews. We're trying to, the adoption is there, and now we have to figure out how to mature in our usage of these tools so that quality or long-term maintainability doesn't suffer. As well as maybe one of the other facets of being able to generate a lot more code more quickly is the drift between team members as far as understanding of the code that's in their services increases.

32:38Everybody's moving faster and more independently. That is another risk that we're starting to see, like an incident response where folks don't know. they don't know a service as well as they used to because it's changed so much in the past couple months because everybody's moving more quickly. Yeah, this has been a major topic for me this year on code-based understanding and slop because obviously it's so much easier to generate code, but then now we have to review it. And to some extent, you can't really fight AI with more AI. You can't just be like, oh, just throw an AI reviewer on their AI code and you solved it.

33:10And so you do need to just scale human attention. And I think that's something I've been pushing a little bit in terms of like, well, you're just going to, like every engineer is just going to own more code, period. And be parachuted in and be expected to ramp up and be productive and also fix bugs if you're on pager duty or whatever. Just because, I mean, everyone's going to try to be more efficient and you're supposed to see ROI productivity because if you don't, then what's the whole point of this? Exactly, exactly. And I think it's funny going back to the point of, you know you could you could add ai on top to solve the problems that ai introduces and there but you just keep you that's like an endless chain uh and so but i mean the the the code rabbits of the world the the graphites of the world would say yes actually you can and so that's the little bit of the tension there yeah you know i i uh i've been thinking a lot about how the craft of of engineering is evolving and and i will say that i feel further away from being able to predict what it looks like than I did this past summer when I spent a bunch of time.

34:13I actually basically went on leave for a month and joined the team that, the AI team that we were building just to go and build alongside them. I felt like it was really important for me to deeply understand the problems in the tech. And so that was me. I was writing Pushing Code effectively 996. And I went through so many different moments of realization of like, oh my God, this is going to change everything to, oh my God, this is just amplifying all the good and the bad in the industry to, oh my God, engineers are not going to have a job anymore. And so I felt like I had all the predictions back then.

34:54And at this point now, I'm just very interested to watch the phenomenon continue to unfold in front of us. And I will say, I was chatting with a bunch of really bright college juniors and seniors at a dinner we hosted last night. And while these folks are about to enter the industry, basically having kind of come up in the era of agentic development and LLMs. And I asked them, like, what is your workflow when you're like building a project? How do you use agents versus like when you decide you're going to actually just write code by hand? and I was surprised to hear the consensus was that most people there were using agents to collaborate on building a design document and collaborating on the architecture of the solution that they want to build and then maybe asking it to emit a doc or an implementation plan, but then they'll go and write a lot of the code themselves still.

35:46So it's a little bit more of the rubber duck co-architect use case that was most prevalent in that group. I was very surprised by that. I'm impressed. The kids are all right. Yeah, I know. They still want to actually write the code themselves. It's interesting. Yeah, what we hear from the Gen Zs at OpenAI, they just YOLO everything into Codesk. Yeah, I would say most of the code I generate. But I spend a lot of time on the doc. It's curious, when you're younger in your career, you don't really have all the mental models of the different patterns to instruct. I feel like there's over-reliance, especially if you're doing the design doc.

36:23I feel like most of the senior engineers will spend more time on that. It's like even things like, you know, what columns should you index? Depending on, you know, what queries we usually run on this table and things like that. It's hard for any AI to know that. Right. You know, and it's like, I feel like the role of like the more senior engineer should actually be more of this. It's like spending time teaching the AI and then the AI can teach the junior people in a way. Yeah. Yeah. And everything looks like mentorship and management. at the end of the day, right? It's like you're breaking down tasks, you're supervising work, you're giving feedback.

36:59Like it's basically management. Except that there's agents are really bad at memory still. Like they basically have zero memory. And it's the end of 2075. What's going on? Yeah. Yeah, what's your internal stack for like preferences? There's like kind of like, you know, explicit preference you can use with, you know, agents.md and all that stuff. there's implicit preference with linter rules and things like that in a way where it's like it just happens you don't have to tell it how do you structure that? Oh no you're talking about for agentic coding or memory or linted platform? Yeah yeah for like the coding specifically it's like and then we can kind of talk about you know the whole Brex platform Yeah just nothing special just a lot of explicit rules that MD files Yeah I know we have and we in linting we still have like traditional linters in place for the couple of different language tool chains and then we're big fans of Creptile and we use them for basically all of sort of the smarter than linting like agentic code review.

37:59That's been the one solution that we've aligned around that has served us extremely well. Yeah, Creptile. Yeah, no, we're huge fans. They've built something really impressive and I think the thing that constantly blows my mind about it is the way that they're able to just have a really impressive signal to noise ratio. like the comments that it leaves are very very high signal uh like never i never regret going through all like 65 comments it leaves on my on my diffs because it catches so many things yeah i found the codex review to be really good i don't use codex for code generation but like the review product yeah it's like very good for some reason um i used to have when i was working in rails there was like this project called danger systems oh yeah it was kind of like a semantic linter exactly i feel like there should be more of that now it's kind of like the rules are one thing a generation but i want something in my ci that is like enforce these rules and call out where they're broken and then i can just copy paste that uh in an agent but yeah when we when we started building this this new agent um coderase like because as i was saying like we were answering the question what would you do if you built uh you know a brex disruptor today and it's like it wouldn't be to pick kotlin and elixir as the back end and uh and so we actually went with the full TypeScript stack and we were building on all public interfaces and really trying to make sure that this agent layer was arm's length from the good and the bad of the core of our product.

39:27And one thing, I think what we did early on, and I don't actually know if this is true because again the team keeps iterating, but we were having good luck using Cloud Code in a GitHub action to basically go and do more of that danger style like code review. So I have a prompt for it that went through all of the different facets that were more conceptual versus rigidly enforceable by a linter and have it leave a big comment at the end with your conformance to the idiomatic coding patterns of the new repo. I wanted to spend some time, you said you wanted to defy on operational agents. Customer support, onboarding, KYC, fraud, link on account, disputes.

40:08This is, I imagine, the bulk of it, of the work. anywhere where there's a good story about maybe when you started out it was going to be this way and then you discovered through building or through customer contacts that it had to go a different direction and so that difference in beliefs is something that people can learn from but the thing that immediately comes to mind is that we uh we believed at the beginning that using rl for credit decisions would actually be a like would be the way that we would end up for credit and underwriting, like how much of a limit should we give to this business, that reinforcement learning would be the way that we would go about building a model that effectively would decision in the way that a human underwriter would.

40:55And it turns out that it was, we made this big investment, we were working with some outside, like a company that specializes in this, and the performance we ended up getting was inferior to just building a web research agent. And so I think what we took away, what has been most evident in operational AI is that in operations, you need to be able to break down problems really granularly and be able to form SOPs that humans can repeatedly follow and thus can be audited because so much of the responsibilities and operations is to have auditable, repeatable processes that help to ensure that we're operating in a compliant manner.

41:39And that actually translates just so cleanly to LLMs that we haven't needed to use too many sophisticated techniques in operational AI. It's been relatively simple, like a few tool like agents, or maybe even a lot of problems can be solved with just like a single turn chat completion. And so the fact that we didn't, well, we did one sort of attempt to over-engineer and use more sophisticated techniques and we discovered that in fact the solutions are a bit more plain and less technically sophisticated. The challenge is really articulating and refining prompts to reflect the execution of the SOP and reflect all the sort of institutional knowledge that isn't written down so that agents can properly replace the humans or the contractors we would have making these decisions.

42:31How do you decide what is worth spending a lot of time building versus what you think? Some of these models are just, because some of these tasks are so generic, they're not really about Brex. Yep. You can assume the models will be good at it versus some of them are like very specific to you. We kind of prioritize like the tasks that are most common for the broadest number of customers. And some of them are fairly intuitive, like being able to research a customer to look to assess like legitimacy of the business and whether that business would fit our ideal customer profile for onboarding because there are certain types of businesses that we either legally cannot serve or we are not comfortable being able to serve.

43:15So that's the type of really kind of basic research and like a relatively straightforward problem that isn't hyper-BREC specific. The things that are a little bit more specific to us or companies in our sector would be preparing documentation for a network card dispute. Like if you go and dispute a transaction on your personal card, you will provide evidence to your card issuer. The card issuer then has to put together like a three or four page Word document that goes to the card network and then eventually goes to the acquiring bank. And all of that is like much more specific to our business.

43:54It's a huge operational overhead for us. And that's something that we decided to automate later because it's not on the critical path of serving the vast number of our customers. Disputes are expensive, but not very common operational process. And so they're lower on the stack. And I think we're getting there right now. But this year has basically been us just kind of looking at every single process, just kind of stack ranking. And I will say the thing that got us started down this path was we wanted to expand our ideal customer profile to support a wider variety of commercial businesses, which tend to be businesses that aren't growing as quickly.

44:34So they're not like tech startups, which have a lot of growth, and they're not enterprises, which also tend to have a lot of growth. It's more like a lawyer's law firm or a dentist's office. These types of solid businesses that we should be able to serve and underwrite, but the cost to onboard them and the cost to serve if you have all the humans in the loop make them ROI negative And so that was the first sort of use case of AI within our ops organization that then led to us really understanding we could automate much more than that. Is this Berks going back into SMBs? Ah, that's a good question.

45:09Yeah, yeah. So never let that die. I think the way we've thought about this is we want to always offer our product to customers where we believe we have an offering that is well-suited to the needs of those businesses. And I would say that still for very small businesses, our offering isn't, it's not built for that. It's built for companies that have some degree of scale, typically have at least sort of one person, if not a couple people in their finance team. So we consider these to be more like the commercial segment. And so it rhymes with SB, but our approach back then was a little bit more naive.

45:54And I would say we also, we were just going for a volume game there. Our internal controls were not as strong. We didn't have as much experience underwriting those businesses. And so it was really ended up being a huge burden for the business, almost existential for us to have those tens of thousands of customers that all were ROI negative. and so we're trying to basically scale to serve more businesses outside of tech and outside of like the out market segment but um but do it thoughtfully so i think right now our um our minimum threshold is is uh like a million dollars a year in recurring in annual revenue or um or like ten thousand dollars or more per month in in card transactions as kind of being like the low end of our icp which is obviously not what you would think when you think of a small business like small businesses tend to still be smaller than that.

46:47Oh, wow. That's really small. Okay. Yeah. Yeah. Mid-market. Yeah, exactly. And it's funny. It's just like the, the, the names of these segments, um, you know, it's like, what we consider it. I don't know. Yeah, no, I think I think like that's, it's like, yeah, it's like lower mid-market and it's funny though, because when what we call enterprise may be another, you know, what sales, what we call enterprises, uh, business that Salesforce might call a mid-market, right? Like, because it's just, it depends on the scale of yourself as a business when you use these terms. And all of these things are built in the Brax agent platform, like all of these automations that people build?

47:18Yes, exactly. Yeah, and in fact, most of the operational AI is running on that original platform that we have. And we built it, one element of it that I didn't mention is that it also, most of the UI UX for this platform is built in Retool. And so like you can basically go into Retool and there's like a prompt manager, a tool manager, an eval manager. And that's sort of where much of this was built. And the goal with that was, again, to make it more accessible, more ergonomic to get started. But what a secondary effect of having a more visual set of tools for this is it's enabled members of the apps organization to go and do prompt refinement themselves.

47:58So you don't need engineers to go and refine the prompts or even test new foundational models when they come out. I think that that's another fun thing when a new model drops, folks will go into the platform and basically run the evals on the new model and kind of see, can we get better performance here? Does this have different latency or different cost characteristics? Yeah, you want the domain experts or the people directly using the tool, not the engineers who are somewhat removed from the tool. Yeah, I do want to highlight to listeners that a lot of the BerksAgent platform are just things that every company should have.

48:35basically. Problem management system, which we talked about where the domain experts are doing it. Multimodal testing, evaluation, and benchmarking frameworks. API integrations for automated workflows. NCP-based architecture shared with Brex's external AI products. This one is obviously very Brex-specific. One thing I did want to highlight that I was semi-impressed by because nobody, people very rarely talk about this, is knowledge base for understanding Brex's business. Yeah. So, do you want to expand on that? Yeah, and this is an area where we've only scratched the surface here but but a big a big challenge that that we face is that the world knowledge or the knowledge that's built into the model about uh about you know what gpt5 thinks brex does and how it thinks our business operates is actually quite different from what our business offers today or how our product works and so we've had to to work on building a corpus of sort of product documentation, process documentation, and like curate this set of information to basically ground a variety of our LLM applications, including like that Brex assistant, which is like the, you know, the assistant that employees will talk to.

49:45It's like, we don't want it to hallucinate features that we don't have or like give wrong information there. And similarly, like some of the operational agents need to be grounded on what our ICP is. Because if you ask ChatGPT5 right now, what types of businesses does Brex on board or what types of businesses does Brex serve, it might not give an accurate explanation to that question. It might say, hey, we're a corporate car for startups, which is what we did seven years ago. And it might say, we only serve enterprises. And so that has been an interesting challenge. And I think what we've been trying to do there is I'm actually going to be spending time with folks talking about this next week internally about can we refresh our strategy and kind of unify it because we have a lot of product documentation that's internal for our operations and go-to-market teams.

50:37We have a bunch of product documentation that's external for our customers. We have a lot of go-to-market sort of enablement material that's more sales pitchy. And we have documentation that is put into Sierra, which is the chat assistant that we use for frontline support. All of this ideally could draw from the same source, but right now it's a little bit fragmented. It's just something that we're trying to invest in, though, because I think at the end of the day, the duplication of efforts is wasteful, and it's absolutely necessary to get this right. Just to deduplicate, Sierra meaning the Brett Taylor startup?

51:17Yes, exactly. I would expect that you have, you build so many other agents, that's one you can build yourself. That's like solving problems that are not differentiated enough for us. I think what's interesting about the Sierra that has been really helpful is that, again, it's really easy for, like the UI and UX of basically administering a Sierra agent is something that's really accessible for the ops and CX strategy team, which are like, it's much more low code and more sort of workflow. and DAG oriented and we have engineers kind of going and giving it tools to take actions but for the most part like it's nice to not have to build build the UX for somebody to manage something like that and I think the fact that Sierra speaks the language of customers yeah exactly speaks the language of CX they can do all the reporting and the telemetry and stuff that that our you know VP of CX would like to see just you know it's just one fewer thing that we have to build What about evals?

52:14How do you build evals? Who manages them? Well, it depends on the application. So on the operational AI side, those evals are basically baked into the platform around every prompt or every agent. And for the most part, I think most of these use cases kind of come online, like the V1 of our commercial underwriting agent or the V1 of our startup KYC agent are co-developed between a subject matter expert in ops and an engineer. and they're going to kind of co-develop an initial eval set. But then from there, generally in Ops, you're always doing QA, be it like on humans or on the LLM decisions. And so whenever, like as part of our QA feedback loop, whenever there is a mistake, that's usually almost always going to result in like another eval being written as like a regression test.

53:08So all of that within Ops AI is pretty straightforwardly managed. on the product AI side that's where it starts getting a little bit more challenging because the multi-agent network is quite challenging to evaluate and so what we do there is we try to adopt some of the state of the art for multi-turn evals where we will basically have an agent embody the user and like have basically the end user agent is given an objective and then we basically have it run a multi-turn conversation and then use LLM is judge at the end to do all of the different facet assessment. The one other thing that we do technique-wise that is interesting is sometimes you want to, you don't want to do like, you know, I think these multi-turn evals are kind of like integration tests.

53:58They sometimes test more than what you want to assess. And so sometimes what we'll do is we'll also pre-can like an initial preamble to a conversation or maybe a couple turns will be handwritten And we'll basically set the eval to start and we'll see if we're able to isolate certain behaviors. So it's still like a work in progress. And I say like at the end of the day, a lot of the just periodic human review and like looking at cases where we've detected as we go to like summarize. like what we'll do is we'll reflect on a conversation after a certain amount of time has passed where we'll summarize it, like extract facets like did it seem like the user accomplished their objective and we'll just manually a lot of the cases when that's failed and decide where I don't have an eval for it Are all the evals supposed to pass or do you have a set of evals that are like someday the model will be good enough and like how does that change over time?

54:58Yeah it's interesting I don't know if we have any that are like, oh, someday I hope it'll be good enough to do this. But it's like there are the evals that are blocking because they would indicate like a regression, an unacceptable regression. So these tend to be just accuracy related evals. But then there are others that are more about like tone and coherency and these types of things where they're more subjective. And we were just looking at those over time as a metric. But the team is actually interesting. I think we're going to get a big update on like how the team is thinking about evals tomorrow and like our Friday, uh, our Friday review.

55:36So it's, this is an area where I'd say the largest challenge, like the largest change we needed to make and how we were executing sort of as like a lab or an incubator back earlier this year to like where we are now, where we've, we've shipped and like, we're trying to, to increase the rigor has been around, uh, like avoiding regressions and having more and more increasingly robust evals. Yeah, I work with a company called Veris AI that does user simulations and I think that's what's been interesting. Some of these things they just don't expect, like the customer does not expect the model to do, but they want to track the saturation of the model in a way, if that makes sense.

56:15And I feel like most companies know what they don't want to happen, but it's almost like they cannot quite articulate oh, I want in the future the model to be able to do this. They can do it today, but I'll keep running this eval. That's actually really, really interesting to me and I'm going to take that away and start thinking about this because there are going to be certain I mean we already seen this where users will ask the assistant for help with things that we don't support yet or we haven't implemented yet. Those are opportunities actually for us to build effectively write a test that's going to be failing for weeks or months and eventually will go green but as a way for us to actually kind of show the progression of sophistication of the assistant.

56:57I really like that as an idea. I wonder how you also catch hallucinations of things that it doesn't have. That's usually the problem. It'll pretend like it can assist with something. One thing that is really annoying that has been tough to prevent is that the assistant, because it is used to speaking to other agents that can support it in accomplishing various tasks, if you ask it to help with a task that it thinks it probably should have an agent to work with. It'll just hallucinate that it's like, oh yes, I'll reach out to the finance team on your behalf to pass this question along. But it's not doing anything.

57:38There's no finance team, there's no way for it to do that. This is something that comes up a lot. It's like, would you like me to ask the finance team? And there's no actual tool for that. Do you put your card reels for that? Yeah, yeah. That was something that we had to... Like a regex? Oh no, we don't... I think we've been able to just beat that out of its system with a system prompt. But we don't have as many guardrails in place right now, just around a couple of potential things that could get us into trouble. Yeah, really, it's surprising when I, I guess two years ago, was first kicking around the idea of all these things.

58:14I would have said that probably guardrails would be more prevalent, especially in finance use cases, but surprisingly they're not. Yeah, and that was actually part of what we, that was like a feature I believe we built in the LLM gateway early on as like the sort of last chance like hard got it yeah exactly here's some red jacks yeah exactly or just you know in the way that like if you go away a field on chat GPT you just get like the inline 500 error it doesn't even tell you that it can't help it just like craps out like we kind of built a couple of those circuit breakers or like the ability to put those circuit breakers in and I don't believe we're using them for anything.

58:51One last thing I want to get your thoughts on was AI fluency levels, which you guys have a framework of user advocate builder native and everyone goes through it, including Camilla. And I just think it's interesting. I think it's a model that other people are thinking about adopting, but they're worried about rolling it out. That everybody's going to be bad. And also like, how do you have like this in-house training course that you keep up to date? Like, you know, just tell us more about it. Yeah, so in the operations org, they're actually more ahead of even engineering on this front as far as, like, trying to create, like, learning pathways for this.

59:32And I think that part of the reason why they're ahead of us is that in operations, they're much more, they have to be able to operate training at scale. Like, training is a very big part of how people build aptitude around their job function within ops, whereas in EPD, a lot of it is sort of getting hands-on, building experience, going along, getting mentored, getting code review. But it's been really neat because I think we've really created an environment. We managed to, by speaking openly about the transformation that we saw would happen in this industry towards AI, sort of displacing a lot of the operations and CX roles, and we were just honest about it.

1:00:14And I think what But in the same breath that we said, hey, a lot of these job responsibilities will go away. We also said we don't anticipate that meaning that your job has to go away. It's just that your job has to change. And so the fluency framework and then the training and support and the positive sort of culture where we celebrate people making progress has been really helpful for avoiding a culture of fear. or like, oh, you have to do this or you have, you're going to get, it's going to go in your performance evaluation. I think the... It does. Well, it's not like as rote as like, oh, like what is, you know, what is the, like how much are you using AI and is it enough?

1:00:54It's more, I think we've built a pretty like positively framed culture where we'll do like spot bonuses for people who have like particularly novel uses of AI in their day to day. And our company, all hands every two weeks, we'll do an AI spotlight. And it's very rarely somebody in EPD for the most part is folks in btms ops uh finance the people organization showing off like how they're building agents you know in chat tpt or on glean or how they're they like just found some new use case that they thought was helpful so we're trying to create create like um i think at the end of the day like we've hired a bunch of really smart people who like i have full confidence that that this type of work is in within the reach of anybody who's motivated to like sort of challenge themselves.

1:01:39And so we've done that. Then in engineering, there's one other thing that I want to call out because I think that this is kind of fun is that we adapted our interview loop to be more AI sort of agentic coding native. So instead of we had like a coding and a system design question that we basically have revamped into a project where we'll give you like a brief before you come on site and then like an additional sort of spec when you do when you start, We expect you to use agentic coding to complete the task. In fact, it's kind of impossible to get all the way through it if you don't. And so we're evaluating your knowledge.

1:02:13We're kind of watching how you work. We're evaluating whether you understand the code that's coming out. We're kind of probing at you as you go. But what we did in order to kind of bootstrap the process of all of our existing engineers getting familiar with agentic coding is that as soon as we had the interview ready to ship, we started we said everybody in engineering including all the managers are going to have to go through this interview and so we re-interviewed everybody internally and it's like it's one of those things where it's like it's not a we didn't like keep a score like or like you know i don't have any data on like who passed or failed or what they what they scored but what we found is like as people would take it it would actually cause them to have moments of realization where it was like oh i i can up level my skills around sort of like i have like i want to be better at this And so we're trying to find a variety of techniques to kind of push the culture along.

1:03:04And I think as I reflect on the year, because this is the year where we really put all the effort into it, I'm really satisfied to see the descent to which everybody's leaning in on a daily basis. Going back to even, I was shocked when we were looking at our cursor logs that the number one user is an engineering manager on InfraOrg. That is super cool to me. It means that folks have taken this to heart and found ways of doing their job differently. I guess I had a closing question or I guess a parting question. And this is broadening out from Brex. Yeah. And this is just you interface with other engineering leaders all the time.

1:03:43Did we not cover anything that other CTOs are having as top of mind today? Like their number one problem is underscore. for the thing i find myself discussing with with folks that and i i don't want to shy away from like scary topics uh in fact we're just just kind of on one that was adjacent which is like how do you evaluate somebody's like progression towards being more ai native the the the cousin to that question is it's like will we need as many people to operate our businesses like are there layoffs coming are how are you how are we thinking about like um headcount growth junior versus senior junior versus senior yes exactly like level mix um and i still have more questions than i have answers there i think i think what has been really interesting is that i view agentic development as being something that amplifies all the all the good just as much as it amplifies all the bad and it amplifies uh uh sloppiness poor architectural thinking um uh misunderstanding of of the requirements like there are for all of the the acceleration of good outcomes it also accelerates bad outcomes and i think what has been interesting is that there has been when you sum that all together there's less of a obvious um like capacity increase it's it's more it's more nuanced than that so i'm not looking at headcount planning uh as we think about it next year as being something You know, like, oh, well, because AI is giving us so much more leverage, we don't need as many people.

1:05:17We've actually, the thing I'm really proud of in my tenure as CTO is that we haven't grown engineering at all. What we've done is we've grown the business significantly, but we've been able to build like greater efficiencies in how we execute, like how we think about building, how we roadmap, what we choose to do and what not to do. that we're able to serve significantly more customers with more lines of business without needing to grow engineering headcount. I think that that's kind of the way that we're going to just continue on this road. It's like, I like having 300 engineers. Like I would love to just, you know, a year from now have 300 engineers, but we're still, you know, 30, 50, 100 % more efficient.

1:05:56That is the thing that comes up with other engineering leaders. And the other part of that conversation is like, how much is AI getting blamed for just sort of ordinary performance-oriented realness. You know, like if Microsoft is letting go of like 4 ,000 people as a business, what, they have 150 ,000 employees, I believe, is that really like AI causing that or is it them just using it as a way to avoid some harder like perf management decisions? I'm not entirely sure, but I'm listening more than I'm speaking on this topic because every time I feel like I have a pretty firm point of view some new anecdote or experience comes in that kind of challenges or invalidates it.

1:06:39Yeah. Well, you know, I take these signals as it's my job to go find people who think they have answers and surface them. And you may or may not disagree, but at least you have something to use as a strongman in your work. Exactly. Exactly. And I think as an industry, it's just early innings on this transformation. So I'm looking forward to seeing, you know, listening to this podcast episode a year from now and seeing, you know, what we got right, what we got wrong and what's different because so much changes quarter over quarter. I do think AI COE is a very well established pattern. I think internal platform is very well established pattern.

1:07:16And this fluency thing is something that people are figuring out that I think you guys are hit on. I'm happy to hear that. That'll be my feedback. Yeah. Any final call to action for things that you want to buy? Like what should people build for you? Like problems you're trying to solve that you would love people to reach out for to help? The call that I'd make is for folks who are interested in multi-agent networks to get in touch with us. Because I do feel like this is something where we're innovating in service of our customers. And where I feel like the frameworks, the tooling, and the research is there.

1:07:53There's actually quite a lot of interesting papers and things that we lean on, but I would love to see more of that encoded in what's available at large in the industry. because I feel like my intuition has been that trying to craft LLMs into deterministic workflows and DAGs is kind of underselling the power that they have to actually plan and execute in a more sophisticated, fluid way. And I just want to see the industry lean in more on these agent-to-agent interactions. Okay, so I'll dive in a little bit here because I have a minor opinion. you keep using the word networks is that a reference to a specific paper or it's your term for it?

1:08:40It's our term and I think that's actually the term that master uses as well for it initially we used to call them agent runtimes internally and then we just switched to networks. And then I think the other thing I wanted to get a clarification on is is it mostly a full agent talking with a full agent or is there a kind of like an orchestrate a boss agent talking to a sub-agent. And I think that does matter for a subset of people who are building all these things. Because when you say multi-agents, sometimes people don't agree what that means. Yeah, so it's a tree more than it is a graph. So it is like, yeah.

1:09:17When you say network, it feels more of a graph. Yeah. But it seems more directional as a tree. Like there is a hierarchy. There's a hierarchy, yeah. But there are some violations of that. Like one of the interesting use cases, And this is where the power of having an assistant for every employee plus having agents that run and embody members of the finance team is really powerful because there's this interesting use case that we brought to market, which is that one of the finance team agents that we launched is an audit agent. where like an audit agent kind of embodies the work that a lot of larger finance teams will do to look for patterns of waste, fraud, or abuse or like systematic avoidance of policy that isn't as obvious with a single expense.

1:10:08Like you can evaluate a single expense and the metadata around it to see if it's within policy or not. But what if you start seeing an employee often make a large number of like$74 transactions when receipts are required at$75? or what if you see certain things like, oh, okay, there's actually a fair number of like DoorDash expenses during business hours from this individual, like on days that an office lunch is provided or maybe you see like ride share patterns that are where you have to look at a broader context. So we built this audit agent that can like ingest your SOP and look also ingest your...

1:10:44This is a Bikes' customer's SOP. Exactly, yep. And what it does then is it's basically always looking for potential violations. And what it does is it is extremely zealous. It wants to have a minimum number of false negatives. So it will raise a large number of potential violations. And then a separate agent, a review agent, will then apply the wisdom of like, is this important enough to follow up on? Is the dollar amount in question high enough? Does this user seem to have a high compliance behavior more generally? It makes a judgment call about whether it's worthy enough to take that violation and make it into a case.

1:11:23Then once it's made into a case, generally what happens is that you need to get more information from the individual. So if humans were doing this, there'd be some outsourced team that's looking for all the potential violations. Then you have some full-time employee on the finance team who's looking at all the violations. Oh, these are the ones that are important. We need to follow up on it. Now what they do is they hand it off to somebody who will go and slack that employee and be like, hey, what's going on here? And so what we have is like the audit agent looks for violations, The review agent decides whether it's worthy enough to turn into a case.

1:11:52And then from there, when the case is filed, that will trigger an event to the Brexit assistant for that employee. And any additional information about the business justification can be collected. Or maybe the assistant already knows because in its conversation history with the employee, knew something about why this expense looked out of policy. of policy. And so you start having the network becomes interesting when you have the finance team agents communicating with the assistant or various employees and then behind there you have other sub-agents. And so then you start seeing more of a graph emerge.

1:12:29But when you look at just what serves the employee it looks more like a tree. Amazing. Well I didn't know you were going to go into that level of detail. Yeah, yeah. Don't worry about that. No, no, no. I'm actually really glad I asked. That is very impressive and And I hope you do more content about that. Yeah, absolutely. We're really excited about it. I think it's been good to finally figure out a use for agents and have the technology be as robust as it is to start realizing this vision because it's something that we kind of dreamt of a couple years ago. To your earlier point, the tech just wasn't there when we were trying to make a similar concept work with the GPT 3.5.

1:13:06It was like, nope. We were hallucinating tool calls back in that day. um awesome man thanks so much for joining us this was fun i really enjoyed it happy holidays guys thank you for having me thank you