Inside the AI Stack of an $8.3B AI Company’s Product Team | Together AI

0:00It felt like after a while that the new party file was flooding your co-workers' context window. We all just start launching slop at one another. What if we could put all these linings into one place? And so that became this Together product repository. If I had to do this manually, this would have easily occupied half of my day, if not more. There are 19 tickets that were filed in the last two months alone. So this is a pretty heavily requested ask. So I kind of get a pretty good sense of what the customer problem is within like five minutes. instead of maybe a full day. Together AI just raised an$800 million funding round valuing them at$8.3 billion.

0:37I got their product team to show you their exact repo they use to automate their work. We've all discovered ChadGPT in 2023 and were horrified at the output and started handwriting. What is a good PRD in the AI era? I come from Amazon. I used to work at Amazon. So we are used to rising like 20 pages PRDs. What replaces the bulk of that is actually a prototype. We're all going to live in a multi-model, multi-harness world. That's essentially the new bar for UX for the kind of products we build. A lot of these things don't get magically better with AI. Where does the line of product manager end and developer begin?

1:27of their paid plans. Check out bundle.akashg.com for an entire year of Bolt.new, Airtable, Speechify, Descript, Magic Patterns, Linear, Dovetail, Arise, and Mobbin. And now into today's show. Charles, so I'm fascinated by how you guys work and how you guys have been using AI. What have been some of the big unlocks for your team in productivity? You know, Together AI is a platform for powering AI applications and agents. And so we saw going back to the rise back when Cursor was starting to hit its growth spurt, how what an amazing superpower an AI agent could be for any kind of knowledge worker in in addition to and including product people.

2:19But it made me think a bit about what would a world be like full of product people and engineers that could all generate as much code and content as they liked. And it wasn't clear to me that the sum of all that was actually necessarily forward progress for us as a startup. and I imagine it's probably the same for a lot of other organizations. So what we set out to do as a team is ask ourselves, what would it mean to use AI, but not to use AI to just make ourselves individually productive, where we all just start launching slop at one another, but rather AI to make us collectively more productive, that the code and the content and the research that we did actually advanced the company as a whole.

3:15And that led us to a bunch of decisions about what we wanted to do centrally and what we wanted to lead people to be free to do individually. And that's largely what brought us to where we are today. So how do you coordinate and centralize all this information? Hi, I'm Nicolina. I'm a product manager at Together. And a few months back, We all we've been talking about the way that we use AI in the hallway, but we never really sat down and discussed how we actually do our day to day work. And when we did, we found that a lot of people had unique workflows that made a lot of sense to the area that they were working in, you know, checking to see the reason behind a node failure, working with customer support, drafting reports.

4:01But then the rest of us were doing, on the day-to-day basis, we were doing things that were largely the same, researching customer needs, creating PRDs or one-pagers. And so we thought, you know, if we're really being thoughtful about how we're constructing these skills and how we're thinking about agents versus skills and now like, you know, loop engineering, what if we could put all these learnings into one place? And so that became this together product repository. It's composed mostly of markdown files, some YAML files as well. And what we put in here are both contexts that is useful for us across the board.

4:41So we have this context document or markdown file, sorry, this context directory. and it has a number of groups of the products, the product grouping that we have. So customer intelligence, sandboxes, but we also have the result of strategy meetings that we have on a regular basis where we divide up product by mission, our strategy, what we want to accomplish, milestone over milestone, quarter over quarter. So that when we're thinking through our research and we're composing these documents with our agents, we can pull this context as it's relevant. And it's also useful to be able to look at the context from other people's sections.

5:24So if you're building something that is touching or going to create a joint workflow, so if I'm creating sandboxes that reinforcement learning users are going to make, I can, of course, sit down with the product manager from the model shaping team, or I can look at the documents that she's already thoughtfully composed about their entry points and how they use SDKs and how an ideal integration might look and kind of work out a really decent proposal before I then put it before her and try to do that legwork. I'm taking advantage of the work she has done. On the skills front, it's a mix of things that are useful for anyone outside of a code repository.

6:05So everything that's relevant to specific code really shouldn't live here but if you want to do something like figure out if we are serving all of the generative media models that all of the users are interested in on artificial analysis, I can have that skill here. So I could run that as somebody who supports generative media, but someone else could as well and they can see the output of that in the terminal or things that we do on a regular basis. So this top line status one is one that I really love at the end of With every sprint, we go through all the projects that we work on and we create an update of what's been shipped, what's ongoing, what's coming up next.

6:45On the skill, you can just decide what areas you're going to pull from. For sandboxes, there's a lot of linear projects, but for the SDK and API, that pulls from a lot of projects. You can pull directly from a repository, see what features got pushed, and then it'll form an update based off of the sources that it was fed and then you kind of edit it add additional updates that it missed or context that would be useful for leadership but that's something that used to take me I don't know anywhere from like 10 minutes to 30 minutes depending on what I'm sleuthing through with the engineers that now takes me you know a handful of minutes just to type in the the prompts and then put the answers into the the right location there As you can see, this is all Claude.

7:34Originally, I think we, for the most part, we adopted Claude code, but a lot of the harnesses that support open source models are used to looking at Claude markdown files and being able to run them just as well. And so we've started to move more over to using our own models. One, because they're fantastic and it's really nice to use our own models. And two, it's more affordable because Clog is, I think, like many teams are learning across the globe, that Anthropic and all these other models are starting to become quite pricey. And so this is an example of using OpenCode. It's the same repository.

8:16This is one of the skills. It's just a news report that I run in the morning. It looks at our competitors, other model labs, just to collect information from what happened over the last 24 hours and I will read it over my morning coffee. It takes a minute so I ran this earlier but you can see it's going to read the skill and kind of look through all of the listed sources that I wanted to find news articles for and then it will create a nice report that i can then click into and build out or read through any information that i want to learn more about that day so if we return to the context like who owns and maintains this because the worst thing is if you're going to have like out of date context so is that one is that like let's say charles sets the overall product strategy is it his deliverable to maintain the strategy document that is in a good format for any AI harness?

9:18Or how do you guys manage these types of behaviors and responsibilities? I think for the areas that, you know, the sandboxes or the customer intelligence that I own, it would be my responsibility. And then on the general, it has been kind of updated based off of larger events. So if we have a strategy or planning session, then someone would sit down and decide like who's going to sit down and translate that into what material. So there's typically a lot of documents that have been written in the lead up. And so it would be a matter of just translating that into a markdown file. But, you know, if the CEO will write letters or write documents of direction and that's a really it's a very clean moment, you know, when direction is being set.

10:04And I think that speaks a lot to how well our leadership communicates direction. But you know that this is a pivot. They make it very clear. They kind of lay it out for you. And so that's kind of your signal. And I think as a product team, it's just a matter of us. I tend to update this, I think, quite a bit. But I think if anyone else makes a PR, it's like any other PR. Like if somebody takes the initiative to create a PR, make the update to the markdown file, then, you know, that's that. But as soon as you want to pull from it and you realize it's missing, then you go ahead and make that change.

10:41And going into the skills, so how do you maintain, like, what's going to be a team skill and an individual? Maybe somebody has, like, a different way of writing a PRD. What is the sort of prescription or guidance? You should be using our team PRD. You should be customizing them. What's the line? Yeah, I think when it comes to updating skills, there's a rule set on where each skill should live. You want to keep it closest to the work you're trying to do. So if it's a skill that is referencing code or is housed within a set of work that a team is working on that's always in that one repository, then keeping everything whole located makes a lot of sense.

11:26And then all the rest of the skills need to go somewhere. So you're either going to have a personal repository or you're going to have a shared repository and potentially you create a branch where you test out a skill and you say, okay, first, I think this is relevant. I think this is repeatable. I'll see if I use this a few times. And if you do, then you can push it up to main. And that's been my approach so far. And if it's something that I think is super niche, then it might just stay in a branch that never gets pushed. Makes sense. And so everybody can connect into this harness using whatever model they want, but the harness primarily is these context and skill files.

12:05Is there any other components that people need to know about? So there's the, the repository and then there's the harness and the harness. There's many open source harnesses. Open code is one of the primary ones that we've, we've been watching as it's evolved over time and supported a lot of the models that we support as well. But HairMaze has also recently published one that I've enjoyed playing around with. And as you figure out which harness you prefer in your day-to-day work, or if you're looking for models that suit a specific workflow, you know, I think a lot of product managers are starting to become a bit more familiar with what models are suited to different tasks.

12:49But as more open source models are coming out, I think it's really interesting to see, okay, Like, how is GLM for coding? How is it for creating these product prototypes or Kimi for doing analysis? And I really enjoyed that experimentation because as these models evolve, you feel like you've got a lot more control over pairing the right model for the capability that you're after at the time. I've been building a lot of AI products lately. My job search OS has 16 different agents. My newsletter has a recommendation engine and I kept running into the same problem. I'd ship something, it would work in my testing, and then I'd get messages from users saying it's hallucinating or picking the wrong tool.

13:31The issue wasn't the prompts or the tools, it was that I wasn't actually evaluating anything. I didn't have a way to see what my agent was actually doing step by step. Every tool call, every decision. That's where Arise comes in. Let me show you. I'm going to open CloudCode and install Arise with just one command. npx skills add Arise AI, Arise skills skill yes. Now CloudCode already knows how to instrument my agent. I tell it set up tracing to Arise and it automatically analyzes my code base, figures out where the LLM and tool calls are, and adds instrumentation automatically. Now I can see everything, every trace, every span, every decision.

14:11And more importantly, I can evaluate it. That's the shift. Trace what's happening, evaluate where it fails, then fix it. This trace right here, my resume feedback agent was supposed to pull the company's tech stack from the job posting, but instead it hallucinated that they use React when the posting said Python. I never would have caught that without seeing the trace. And here's the part that blew my mind. I asked Cloudcode to look at these traces and tell me what I should be evaluating. It came back with four eval criteria I hadn't written. Things like picking the right tool and staying grounded in the input.

14:42I wrote the evals, ran them, and found that my agent was making the same kind of mistake about 12 % of the time. Claude pushed a fix, I re-ran the evals, and it dropped to under 2%. That whole loop, trace, evaluate, fix, took me about 20 minutes, and now it runs automatically. If you're building AI products and not evaluating them, you're shipping blind. Try Arise free at Arise.com and get a year free, a$1 ,260 value with my bundle, Arise. Check it out. It's one of the top AI evals platforms used by all of the top AI teams for a reason. Amazing. So Pavneet, can you show us what this is like in action?

15:22How does someone day-to-day use this for customer interviews, writing PRDs, and getting things into a state that they'll hand it over for engineers? Yeah, Kash. I'm Pavneet Alwalia. I'm the product manager for infrastructure at Together AI. So how it really comes into action is basically based on two processes. One is the initial discovery and aligning process. And the second part is the build and ship, which is more automated. And really what this ensures is that as a PM, when I'm shipping a PR or I'm writing some code, it's not noise to engineering. It's actually grounded in our best practices in architecture.

15:58So let me show you this in action. So I have my cloud code here. I'm going to first start with like see a feature that I want to work on. Right. So let's say I want to do some research on a potential feature that I heard some customer complaining about. So one that I'm looking into right now is how many customers are complaining about shared storage resizing. And what this feature is going to do is it's basically linked to pylon, which is where our support tickets lived, to linear, which is where all our project and engineering execution is tracked, and to notion, where some of our internal product documents live.

16:38And it's basically pulling all the information from there to show me how big is this as a customer problem, how many customers are complaining about it, give me some verbatims and some tickets that I can go and deep dive into. So you had mentioned that certain parts of the product process are more automated than the others. Which do you feel like are more automated? Yeah. So the way we see it is the part which is where the decisions are being made, right, which is defining the feature, defining the API surface area, the abstraction layer, those are very human in the loop, right, where I'm hands on with Claude.

17:13And the code writing part, the execution part, is actually the part which is automated. And how we would typically do that is, like, I would basically trigger a goal, right? And say, give me production-ready PRs for a feature to resize. So, exactly the feature that we are going to work on, right? Resize shared volumes for a running cluster without destroying data. Right? Just hypothetically. Now, typically as part of this goal, I will also give it a few other tasks. For example, give me a design doc that I can review with engineering. And first, let's verify on a POC cluster. So we would typically deploy it on a cluster just as a POC to validate everything that's working as intended.

18:09And this is where, because goals are amazing, this will spin off multiple subagents and actually start tracking all the pieces. And it's going to prompt me for information along the way, for example, like a POC cluster or more information about the feature. And that's where we'll feed it the PRD or maybe a UX prototype that I built along the way. So very, very powerful engineer skills. It sounds like you have a lot of confidence and the engineering org has built skills that you can just give a goal command, like get a production PR ready and it will go pull the relevant skills. It will be able to read your code base and get something actually production ready.

18:49Absolutely. We actually have a full engineering repo where it's just our architecture details, right? It's a bunch of skills which tells it for things that we've done in the past, how to redo it, like runbooks, engineering, architecture, design documents, which Claude has access to through our shared repo. And that's why it's able to build high fidelity PRs in like a first attempt, really. Wow. I didn't realize just how good these had gotten at AI native companies. So that really then, like you were saying, in the human and the loop parts, gives you a lot more time to focus on those in depth. Yeah.

19:27So if you see right now, it's actually spun off a subagent to look at how is the one data plane working? And then another subagent to check how is the control plane working? And now a new one to look at docs, right? Is there existing internal or external documentation on how we might do this? And do you have a point of view like you're using looks like cloud code in the terminal? What's the easiest way to use a shared team repo harness? Yeah, so actually, if you look at this, it's currently running in my OS directory, which is actually pulled a lot of the shared repo artifacts. So, for example, Nico shared a few artifacts around research.

20:09So it's actually this skill, the research, feature research skill, which is actually a shared skill that everybody uses in the team. Then there's another one, which is my personal favorite, which is PRD writer. This is actually pretty cool. Let me just actually trigger it off. Right. So resize shared volumes for existing running clusters. So what this actually does is it does a turn by turn interview. of, it'll interview me basically on what are the decisions I want to make in this feature. And if I show you the skill, so this is the actual skill. It ingests, it can take a prototype. So if I already have a UX prototype, it'll ingest that.

20:53If I have a running POC, it can take that or nothing, just like a simple prompt, right? And then it has access to all the information, the shared context that Neko shared about, right? It's pulling all of that in. And then it's going to ask me questions, turn by turn. And it's specifically instructed to push me, to challenge me on my assumptions and basically the decisions I'm making. And then it goes into drafting a PRD. And we've given it a question bank, basically, to kind of ground it in what type of questions typically we need to make. For example, what are the tradeoffs? Is there any one-way decision we are making along the way?

21:31Amazing. So we've spinned off. I think we have currently three agents working for us, right? Research, PR, and PRD. Yeah, so this one is actually done. So the research agent came back. It's saying all these customers have asked for this feature in the last six months. There are 19 tickets that were filed in the last two months alone. So this is a pretty heavily requested ask. I have the source, so I can actually go into these tickets if I want to dig deep exactly what happened and why did customers ask this. And what was Pylon again? Pylon is where our support tickets live. It's the support platform.

22:08And then actually it figured out that we had started working on this feature because it has access to linear as well. And it was partially implemented, but it's broken in some way. So we stopped working on it. So now I'm not duplicating. I'm not like replicating work. I can actually build off somebody else's work or actually talk to that engineer and figure out, hey, what happened? What issues we ran into? And is this hooked up to the live code base so it could go check like in case there's some discrepancy between linear and the code base? Yes, because it's in my personal OS directory. It has a skill which gives it access to all the internal GitHub repos and it knows what's what can map that.

22:49By the way, Kosh, later on I can show you like a God's eye view of like what this looks like across all products for repos as well as across all customer feedback. Okay, I'm excited for that. So, Puneet here, we've got the research report. What's your take on this research report? Is this a, like, how long would this have taken you? What do you give this out of one out of 10? Yeah, so if I had to do this manually, this would have easily occupied half of my day, if not more. And this is actually pretty good. It gives me the brief snapshot that I wanted to know, which tells me the feature passes the smell test.

23:23There are enough customers asking for it. I know most customers that actually want to increase their storage volume, right? So I actually know what's my like MVP use case. And then I have a source so I can actually dig dive into some of them to actually get a better sense of the customer pain. And it gives me verbatims as well, which kind of helps. So I know, for example, here it calls out a customer called Moonlake. And we have a separate tool called Customer Insights where all customer calls live. So let me go into that. So this is the customer insights tool where every customer call that our sales team is getting into, all our support tickets are getting summarized and cataloged.

24:02So I can actually go into this and actually see what exactly happened with Moon Valley. So they wanted a 50 terabyte, but there's a lock icon in the UX, which is preventing the action. And they are basically they reached out to sales team and they basically want to sell the workflow. Right. So I kind of get a pretty good sense of what the customer problem is within like five minutes. instead of maybe a full day. And this customer insights tool, so you guys have built an MCP server sitting on top of Gong and this has been visualized by V0. That's what we're looking at right now? Exactly. It covers both Gong, Slack, Pylon.

24:39Yeah, at least those. So it's actually more than just Gong. And who maintains this? Sales ops? Hassan, you want to speak to that? It's largely automated. Yes. Me and my team maintain it. So my name is Hassan. and I run the developer experience team here at Together. Me and my team have built a bunch of this kind of stuff. We maintain it, we continue to have new features, but it's largely kind of working by itself at this point. We have like a daily cron job that runs every day that grabs all of the calls, all of the pylon tickets, all of the like Slack channels, everything that happened in the last 24 hours and it adds it to the database that this is running on.

25:17And yeah, what Poudnita's showing is the daily screen that like where we show like five to 10 kind of insights from customers every single day based on all these calls. We also have an MCP server. We have a chat where you can ask it anything. There's a lot of kind of facets to this tool as well. But yeah, my team continues to maintain it and kind of add new features here and there. Okay. So developer experience maintains like various MCPs. What other MCPs are you maintaining? Good question. We have this one. I mean, some of them, a lot of these tools are exposed differently. This one has an MCP server.

25:53For example, another tool is called Orchestrator. Charles is going to show it off and talk about it momentarily. That one has kind of just a UI and is a wrapper on top of all our GitHub repos. And yeah, really my team just experiments with a lot of this stuff. We build things that we think may or may be useful for the product team and the rest of the company. And some of the internal tools kind of flop. and we're like, well, you know, this is not used very much. And probably our two or some of our most used ones have been this like customer insights, MCP and app and orchestrator and agent evals, which I'll also show off.

26:27So those are like the top three that have been, that people have been using. Sweet. So we get to see all three. Awesome. And Bhavneet, how's our PRD and our PR agents looking? Yeah. So it's now asking for evidence. So this is where typically I would run this in a single chat because this is sort of somewhat sequential work. But typically then I would take the output of this research, which is pretty detailed for like a starter starting point. And I would actually give it to the PRD agent to run with. So now it has a sense of the customer problem. So it can ask me more educated questions on, hey, what are the tradeoffs?

27:08Should we build as an abstraction? Should we self-serve? when do we pull in support those kind of questions so almost like managing different agents that feed each other okay so now it's asking me hey do you want to give me a figma or a running poc or some screenshots or i can just say no run with it like let's do it uh turn by turn by discussing it uh we all have gone through phases of using ai for prds you know we've all discovered chad gpt in 2023 and were horrified at the output and started handwriting. Then we discovered Claude and said, okay, maybe it can write well. And then it almost feels like, you know, people were overloading our colleagues' context windows with long documents.

27:51What is a good PRD in the AI era? Yeah, so that's actually a question we thought a lot about. So where we really use PRD, like historically, like companies started using PRDs as like a gating document where everybody, all stakeholders need to come together, align on it, and only then we'll start building the feature. We see it a little bit differently, where it's a tool to trigger ideation and problem solving. That's it. And it's short. It's usually one to two pages. It defines the customer problem well, so that everybody has a shared understanding of the customer problem we are trying to solve.

28:24It lays out some solution options, right? It doesn't have to be fully thought through necessarily, but at least, like, what are the different ways we could go solve this problem, right? And then a sample user journey. Based on the option we want to go with, it will define either an API-based journey or a UX-based journey to show, hey, these are the three, four steps that a user would go through in the solved world once this feature is shipped. And that's it. Like that's enough to actually have a detailed discussion around, hey, how should we build this feature? What should engineering design look like?

28:59Is it even worth building or not? Right. And that's it. That's what the PRD template that this skill uses builds off. And actually what replaces our typical historical six pager, like I come from Amazon, right? I used to work at Amazon. So we are used to rising like 20 pager PRDs. What replaces the bulk of that is actually a prototype. So we built a separate skill. Let me start a new cloud window.

29:26And there's a separate skill called UX prototype, which actually you give it that one pager that we just discussed. And it will give you a very detailed prompt on how to build it in like a Figma make or any design by code to that. You can use it in VO, you could use it in cloud design or even Figma. Right. And that's where a lot more debate will happen, because now everybody can visualize it, visualize the solution, see what's how it's going to interact with the user. And that's where we find like a lot of the best critical feedback comes from from engineering, marketing or anybody else. Do you find you're living outside of cloud design then a lot?

30:04You're getting this to generate the prompt and then giving it to Figma Make? Yeah, the only reason I use Figma Make is because it's easy to share a Figma Make link and others can iterate on top of it. It's easy to click around. Cloud builds like HTML files, which it's hard to keep version tracking and stuff like that. Figma is just like more shareable. That's why we use Figma Make. Makes sense. So how do our PRD and PR agents look? Yeah, so, okay. So now it's asking me questions, right? So it's saying, hey, the evidence splits into two different directions. One is multi-tenant substrate quota bumps, and the other is dedicated conversion expansion.

30:40Which one do you want to go for? I actually want to go in a very specific direction. Just looking at the feedback we got from customers, I want to say, hey, we should focus on increasing the attached storage volume sizes for tenants. It's tenants here is basically a customer cluster, right? So tenants in a running cluster. So here's where your domain expertise, your human judgment is coming in. Yeah, it's not a substitute for me, but it really does two things. A, it speeds up the whole PRD writing. It solves the blank page problem to a large extent, but it also catches a lot of things that I might not have thought of.

31:27I often find it's asking me questions that I might have missed if I'm not spending a lot of time thinking about the problem. So ignore VECA issues. I'm just giving it a sample to focus its energy and to reduce the scope so it's not trying to boil the ocean in the problem space. So a 10 out of 10 that comes out of it, it's going to be kind of that very short document that accompanies a prototype. Yeah. If this all goes well. Yeah, so I actually did this and I have that PRD. If you want, I can show it to you what that looks like. Yeah, let's take a look. So this is what the final output looks like, right?

32:06So first of all, it's telling, like, where's the evidence? It looked like we had 20 pilot tickets across 14 customers, right? So straight up, you know there's evidence. And this is what it looks like. It first defines the customer and the business problem, like we talked about. The 20 confirmed customer issues, it's name dropping some customers just to give us a sense of which customers, right? Is it our top big customers? Is it like a long tail of customers? And then it gives you a sense of like, what is the actual pain, right? It seems like we cannot update this in the UI. And that's basically the ask.

32:37They just want to self-serve the volume resizing. And that's like pretty obvious from the customer verbatims. And then you go into goals and non-goals. This is important for scope creep, like, which is like the number one problem I feel like PMs and engineering faces. So it gives you some goals. It's giving me a goal that, hey, if done well, this should basically handle over 90 % of resized requests. It works in both directions, upwards and downwards, like scale up and scale down. Billing should adjust immediately. See, this is something I did not think of, actually. When somebody reduces the size or increases the size, we need to make sure that the billing matches up and they're not over or undercharged for it.

33:18And capacity block requests. What happens when we don't have enough capacity? If it's a fully self-serve experience, when do we actually need to get supported more? And then it defines our goals as well. User stories so that whoever is reading this and when we are doing a discussion in engineering, everybody understands the key use cases. And then actually the proposal overview. It's giving me some API specs. It actually gives a detailed API design as well. So we are going to add to our existing API some changes on how to basically allow for resizing. This is the user story. sort of, right? Like what are the different API specs and what is the API calls?

33:56And then UX as well. End-to-end user flow, happy path, what are the steps that a user will follow? So I feel like it always gets the headers right, but sometimes the devil is in the details. Is it nailing the details? So let's look at this one, right? So admin sets a new size in the console, API validates. This is some internal detail actually. So maybe this part not necessarily needed in the user flow. this is actually internal detail and then it shows the status it shows that it's available and billing rate is updated so it got the user journey right but it bled some internal detail into it as well so it's still going to need a little bit of editing you can't just immediately take this and start sharing this with your colleagues yeah absolutely actually at the top it says this is a draft one page of prd it's not meant to go wide distribution for everyone And that again kind of goes back to the process that we were talking about, where in this initial discovery and design phase, it's human in the loop.

34:58So the expectation is that the PM will go and make changes, maybe make some changes to the API, maybe change the scope a little bit before we circulate this widely. Very cool. So we've got the repo. We've gotten to see it in action. We've got a little preview about an orchestrator agent. Charles, can you walk us through this? how do you get to see this bird's eye view at the top of the product ladder? So we built this great internal tool, which we call Orchestrator. The basic idea is each of the individual product people are sort of married up, hand in glove with their corresponding engineering team.

35:33And Pavnik kind of gave you a good example of how that works, where he wants to be able to go pretty far into the definition and implementation for some of the things he works on. and the specific engineering team he works with has given him the kind of skills connected to their code base that he can go do that. From my vantage point, what I need to do is be able to get a check on where things sit in all kinds of different things that we're building as a company. And I don't want to have to build the equivalent of PubNeed and Hassan and Nico's environment. So I have this nice tool here called Orchestrator, and you can see that essentially all the major repositories we have as a company for all of our products are represented here.

36:18So if you see, I'll give you kind of a quick example. So you have like, let me find PubNeed's T-Cloud. Yeah, so you can see here we have the repo that PubNeed was just living in T-Cloud, but you can see that it coexists along with many other repos. So for sake of argument, like one of the other products we have, we call model shaping, which is basically the ability to adapt the behaviors of open weight models. And I can decide how I want to interrogate what's going on in that code base, what's going on in that product area. I can pick my harness. I can use either open code, clod, or cursor. I can pick my model.

36:57So in this case, like I tend to be a, you know, GLM open code kind of guy. And I could ask some question like, what's the most recent model we have enabled or supervised fine tuning? So we basically have more than 30 models that you can adapt and fine tune, but these are changing all the time. I don't want to have to go understand the entire world of the product lead for model shaping. I just want to understand what's going on in this one specific, you know, to answer this one specific question. And you can see that essentially what we'll do in this case is we'll actually create a sandbox. We'll call in the repo.

37:41All of this happening in the background. In a minute, it's going to basically interrogate that portion of the code base. And it's going to go research, like, what are the most recent changes and let me know what happened most recently. You can see it kind of cogitating right now. Like the proverbial cooking show, I have kind of a synopsized version of the conclusion. And in this case, it looks like the answer is the last model that we enabled for supervised fine tuning is the new NVIDIA Nematron Super 120B model. So I can do this exercise essentially across like any product, any portion of the code base for that product.

38:30And it's not just limited here. I'm just asking questions. But if I also find some part of the product that annoys me and it's something small, I can actually generate a pull request from here as well. Okay. So you're mainly living in the orchestrator, not inside Cloud Code and the team repo? I use Cloud Code and TeamRepo for if there's some requirement I'm writing myself for some part of the code base, then I would do that. But if it's me shopping across all the different products we have and it's like some small UX change or things like this, then I would sooner use the orchestrator because this is basically not just pointed at all the different repos we have as a company, but it's inheriting all of the skills and or mcp servers that are local to each of those repos so i don't want to have to build all that into my local uh open code just to make one small change it's a lot simpler uh to just to just use orchestrator and it's i know it's going to have uh the latest greatest skills in mcp servers for that portion of the code base got it so who are the other users of orchestrator within the company like what who is this product exactly built for it's sort of intended for casuals right it's intended for so so it's intended for like uh let's say pub need wants to uh investigate something in inference or let's say uh necoline wants to make a suggestion on on uh our infrastructure as a service um they don't want to build and replicate all of the specific skills and context native to that particular portion of the code base, it's a lot simpler to just look at, uh, just to use orchestrator where everything is going to maintain server side.

40:14Hmm. Okay. So this is like for whenever you're casually working, maybe like trying to learn something about another team, this is also creating that connectivity. Yeah. One of the things that we talked about as a team was like, like, like it goes back to this point about how much is shared and how much is individual, right? And there was a point at which I thought, wouldn't it be great if everybody knew what everybody else was doing? And there was sort of like, we all had the same context and everybody else at all times. And in reality, most people don't have a whole lot of motivation to want to understand all the depth and nuance of the context of somebody else's area.

40:52They want to know just about they need to get one single question answered or to get one single problem resolved. And they're really not interested in the rest. So we sort of set aside the idea that there was one big, broad, flat set of context that we're all going to swim in. And it's much more like a context hierarchy. And some of us belong all the way down to the bottom of the depths of that hierarchy. And some just want to traverse the top. Fascinating. Okay. So if a team company watching this, they wanted to replicate what's being built in orchestrator how are they going to spin up their own version of this yeah so i mean this like from what i from what we see in terms of our own customers this is becoming increasingly common like i was at a ai conference in in paris the other week and i saw like a like a like a digital um uh services and marketing company called process and uh this idea of essentially, they basically already built their equivalent of this same thing, where they have like every portion of the code base you want to change is all in one place.

42:05Everything can be spun up as a sandbox and generate its own unique pull request or its own questions. So I think this set of tools is like not that out of reach for most software development organizations. and this idea of like we're all going to live in a multi-model, multi-harness world. And the main endeavor then is how do you organize shared context? That's kind of, I think, what everybody is starting to build. To give a more specific answer, I mean, Hassan, what was the total time invested, would you say, to build Orchestrator? I would say a few weeks of work, maybe a month of work to build it, of engineering time.

42:53Yeah, and a lot of it was kind of just like trying to figure stuff out fairly early, trying to figure out the right architecture, the right tools to use, a lot of that stuff. But as Charles said, I think this has kind of been replicated by a bunch of other companies. There's a few open source versions of this as well that exist out there. So it's kind of easier than ever once you have the architecture. But yeah, for our team, it took a few weeks. So like if we're just drafting the PRD for an internal orchestrator for yourself, what are the key things? It needs to hook into like all of your code bases and your different repos.

43:24It needs to inherit all the active MCPs and skills from the different team repos. It needs a sandboxing mechanism because you're basically going to essentially like each individual thing you're researching or each pull request you're going to propose, that all gets done in a sandbox environment. So you're basically cloning a fraction of the repo, right, the portion of the repo that's necessary in a sandbox to generate that one pull request. Got it. You also are going to need some form of a model gateway or router. So you notice I have like a whole choice of models that I could use. So that's typically another piece of the puzzle.

44:02But as Hassan mentioned, every one of these components I've mentioned so far there are either like either it's relatively quick to build there are open source uh tools and libraries um that let you do these things and there's commercial shrink wrap software if you don't feel like doing that either the previous two things fascinating so we're walking through the whole product development life cycle the last step evaluating how these things are going live hassan how are you guys doing agent evaluations great question um as you said this is kind of the final step that we do. I'm going to share my screen.

44:36I'm going to go over this tool that our team built called agent evals. And just to give a little bit of context on agent evals, you know, I think the whole world is moving to this agent centric way of doing things, right? You don't, you don't kind of manually write code anymore. And kind of, it's moving up the stack with like, you just give your agent stuff to do. And so, you know, my team is called developer experience i think charles is this close to renaming it agent experience um and because it's something we're increasingly uh thinking more about and that's very very top of mind we've re-architected re-architected our docs in kind of this agent first way where it works very very well for for humans obviously but also for agents um you actually on some pages on our docs like agent when when an agent is reading them we'll inject something extra um if we think it'll help the agent um we worked on it together mcp server we worked on together skills and so we're exploring all of these ways that we can make it as easy as possible for agents to use our product.

45:33And agent evals is kind of the tool that ties them all together or the tool where we can actually say like, okay, like we're pretty confident in how this works. So I'm going to share my screen and we're going to go over this. Fantastic. Okay. So agent evals, there's a lot here, but agent evals is the tool we use kind of at the end of this product development lifecycle. We have the product. It's shipped right some kind of some major feature or a product or second version of the product And now we need to validate that it actually works well with agents So what we do is I tend to work with a lot of the product managers In each of the different areas and we write a series of tests of like the main Like here are the main ways that we think users will use a specific product.

46:18So Let's actually take fine-tuning as an example or we'll take Yeah, let's take fine tuning as an example. So if I click on fine tuning, this is a prompt that we give for this is one fine tuning task that we're testing. So here we're saying, hey, we gave it a data set. And we said, hey, run a fine tuning job for this data set. And when it's done, spin up this new fine tuned model as a dedicated endpoint and evaluate it, right, where it's going to send it some inference requests, and then delete the endpoint when you're done, right. So we gave it, and this actually touches on a few of our different products, it touches on our fine-tuning product and our inference product.

46:54And so some of these tasks are a little more simple, some of them are a little more complex. But the point is that the most important part is to define a series of tasks and give it to this tool. And what we do actually has a very similar architecture to Orchestrator, which we talked about. So what it'll do is it'll spin up a sandbox, it'll spin up right now Cloud Code, and it'll give it this prompt, right? We'll give it together API key and then it gives it whatever it needs for the prompt. In this case, we gave it actually a data set and then it just watches cloud code do its work, right? It's just It lets us do its thing and then it evaluates it at the end and for us This is like the best test of like can an agent actually do stuff in your product?

47:35We try to make sure we try to make everything in our product We try to make it so that you can do everything in our product for example in our UI as a human, but also as an agent through our API or CLI or SDK. So in this case, it actually got it correct. But we go very, very deep here. So we run a lot of different runs. Some of these runs are just using our docs as contacts. Some of them use our MCP server, some of them use our skills. And we can click into every one of these and we can get a lot of information on the run, how it worked. We can get some high level info. We can get improvements, which this is one of the most important pieces of this.

48:13And this is something we see a lot where we'll ship a feature or we'll like in the validation stages of a feature and we give it to agent evals. And then it's like, oh, well, I had trouble doing this thing, right? I wasn't able to, like for example here, I said it wasn't able to discover what fine-tunable models we have, right? Obviously this is a big problem if it can't find this. And it said, hey, like it can't, it couldn't find a docs page listing currently fine-tunable models. And so we should do this. And so we make a ton of improvements to our docs based on this. In this case, actually what happened is we had a page that listed all the fine-tunable models, but the agent actually couldn't find it.

48:49We were able to go into the transcript. You can go to the transcript and you can see the full transcript of this agent and what it did, right? We gave it this initial prompt and you can see all of the different terms. We can see the agent started exploring the repo. It started looking at the data set. It like did all of this stuff, right? Like the full transcript. And we looked at it and we saw like, actually had trouble finding that one, um, uh, that one page in the docs. Cause it's not linked in our main fine tuning quickstart. So then we're like, okay, well, we'll just go submit a PR to our docs and add that page to our fine tuning quickstart.

49:21So, um, the agent evals has been responsible for, for like dozens of like docs fixes that we've done. Um, and, and it's been really, really valuable to really understand like, uh, and get this bird's eye view from agents on like how good are agents at using this particular tool or this particular API that we just shipped. and Akash like I just wanted to like add on I mean you think about any sizable feature or product that you'd build in the past and the idea about like well did you get the design right like did it actually you know can the can the outside user whether it's like someone technical like a developer or non-technical uh be consistently successful using that feature and think of how long you had to wait to validate that as a product person before.

50:10You'd have to wait till you did a bunch of user tests and then you're going to get a bunch of conflicting signal or you're going to have a bunch of developers evaluated and they're all going to say, well, I don't think this is sufficiently Pythonic or I don't like the way you did this syntactically. And so it's really hard to get timely signal and it's really hard to get objective signal and here essentially at this point for a lot of our products agents are already the majority user and so the idea is that the nice part about that is now did the feature work from a design point of view this is something that we can validate immediately and continuously so it's really powerful to sort of always know where you're at and it's really powerful to have the confidence to know that whatever your documentation says your product is, you can be sure that it actually is that because essentially every few hours, we have agents revalidating that.

51:15And across time, what we plan to do is keep expanding the range of harnesses and models that we do this for, because that's essentially the new bar for UX for the kind of products we build. Okay, very cool. So if your product isn't used by agents yet much. Does this still have value? Is this able to kind of simulate what a human is like and should people still be setting this up? So if for us, it is because our users are developers and so they're going to use our SDK. And this is basically validating that there's like valid, successful paths using our SDK. If we had a more GUI, like, you know, web front end intensive product, we would have to adapt this tool to use more, you know, like a visual reasoning model that could actually do the click throughs and like interpret the screens and see whether or not that was like naturally intuitive to the model.

52:16So it could be extended to human centric examples. But for our case, where agent use is already so popular, it works as is. All right. So we've been able to cover front to back from customer research through to actually evaluating how agents would be using your feature. This is kind of the whole product development lifecycle. I'm curious where this ends. We kind of drew the line here at create the PRs, create the prototype. Where does the line of product manager end and developer begin? I want to think how to best respond to that without just repeating what Pellney just said at the beginning of our whole thing.

53:03I think that the essence of what the product person does versus the essence of what the engineer does, ironically, is probably not all that different than what it was before AI. In the sense that the most valuable thing the product person can do is bring a unique insight about the market that's been well validated by lots of internal and external context. And that's essentially what you still saw Pavni do. And the essence of what an engineer does to add value to the company is to arrive at a design that is the most efficient way to meet a need. that is also, that is differentiated. The essence of the engineer's job is to arrive at a design that most efficiently satisfies a need and is also something that you can maintain and extend and contributes to sort of the long-term architectural strength of the product.

54:10So like these two, the essence of each thing, I don't think is actually all that different today than it was before AI. What's different is the convenience with which the product person can reach into the engineering world and accomplish small and medium-sized tasks. And the inverse is true as well. The degree to which the engineer can reach into the product management process and answer their own questions. So just like it's possible for me to interrogate the code base and make small pull requests, it's just as possible for one of my colleagues in engineering to use that customer insights tool and do their own analysis and have their own standing query for whatever the customers have been asking for in the past month in their area.

54:54So we sort of like it's sort of easier for each side to reach into each other's area to do small things. But in terms of like what each person is supposed to contribute, that brings their unique talents and perspectives. I think that's actually the same core as it's ever been. Amazing. So I've been preaching to people, get your PM OS, get your team OS, get your company OS. You guys just demonstrated are actually living that reality. So you're at the very top of the AI adoption curve. And I heard a really interesting observation from Chamath Palhapitiya. He said, you know, our token costs are something like doubling every 70 days, but our actual engineering productivity is just up like 5%.

55:36When you look at it from a product lens on that, like how are your costs on AI growing? And are you seeing a tangible, like some sort of tangible productivity gain that you can point to from it? Yeah. So, I mean, we never were in a place where we were doing like story points or other kind of velocity measures. So I can't prove it on that level. But I would definitely say that our velocity is gained more than 5%. I find claims of 3x to be very suspicious. I don't have any, like when you get a team of, let's say, a dozen engineers and a product manager, there's so much of building software, which is discovery, debate, kind of reevaluation, coordination.

56:29And a lot of these things don't get magically better with AI. So even if you compress the research and even if you compress the coding and the testing, I think that's worth a lot in terms of velocity. I don't know that I would say our experience has been that like 3x or some huge multiple like this is the case. And as far as expense goes, I think we went through the same surge that a lot of folks did. It was easier for us to mitigate because we can use our own open weight models and they're a lot less expensive. but the other part is it goes back to this first point which is I think if you're focused on making teams collectively productive as opposed to yourself feeling individually productive by producing lots of output I think it's unlikely that you wind up with these crazy token budgets of three times people's salaries or things like this I don't know that we ever reached that type of peak I like how measured you guys were in selling the benefits of all of this, where none of it was overhyped, but we got to tactically see how it helped both individual ICPMs like Bovneith and a product leader like yourself, Charles.

57:45All four of you guys, thank you so much for dropping so much insider knowledge about how TogetherWorks. Thank you for having us. I hope you guys enjoyed that episode as much as I did. They have been kind enough to literally open source the tools, the repo, the orchestrator, how they built this. So check the link in description below if you want to get your hands on how together actually built this and replicate some of this in your own company. I highly recommend you do that. Podcasts are one thing, but actually implementing it, that's when you really get the ROI. So I hope you go do that and we'll see you in the next episode.

58:20I hope you learned as much from today's episode as I did. If you can do one thing that's totally free that would help the show, it would be to check that you're following on Apple and Spotify podcasts. Check that you've left ratings and reviews on those platforms. Check that you're subscribed on YouTube. Leave a like and a comment on this video. And then share it with your friends. We're trying to make better and better podcasts. After two years, we think we've gotten something pretty good going. So let us know what we can do to make it even better, who else we should interview, and we will put on the best shows we possibly can.

58:53Finally, don't forget my offer for the bundle. You get an entire year of my paid newsletter. Plus, my favorite AI tools, Bolt.nu, Airtable, Speechify, Descript, Magic Patterns, Linear, Dovetail, Arise, and Mobbin. That's$27 ,000 worth of value for just$150. So check that out at bundle.akashi.com if it interests you. And I can't wait to share our next episode soon.