1. Podcasts
  2. Agentic Conversations (formally mlops.community)
  3. How Stanford Teaches AI Coding Now
Agentic Conversations (formally mlops.community)

2 Nov 2026 · 44m · 23 chapters

Episode artwork

How Stanford Teaches AI Coding Now

In short

A conversation about how Stanford’s “Modern Software Developer” course is adapting to rapid AI coding-agent advances, plus broader lessons on building reliable agentic coding workflows and agent “harnesses.”

Guest(s)

Mihail (Stanford lecturer; head of AI at Monaco).

Key claims

After model capability jumps (e.g., “Claude/Cloud Code” craze after Opus 4.5), the curriculum was overhauled: from basic coding to harnesses, loops, software factories, automated code review, auto-merge/merge-conflict handling, DevOps readiness, and agent context/tooling. Students must make real open-source contributions; “slop” submissions can fail. Reliability hinges on verification loops (specs + verifiers like browser/Playwright recordings, end-to-end smoke tests, and diff-based visualizations).

Notable examples

Jesse’s “Superpowers” project; Waymo-related “victim” mention; Boris (Cloud) and Lee Robinson (GrokBot/harnesses); Monaco’s go-to-market agent platform using MCP, memory/compaction, and intent-based tool design.

Written by AI. May contain mistakes.

Chapters

0:45

Overview of the Stanford AI Course

Mihail discusses the evolution and updates in his Stanford AI course.

“People jump on the Cloud Code bandwagon through, like, that's basically what everyone did over the, over like December break, you know, during the Christmas holidays and New Year's.”
2:30

Curriculum Changes and Open Source Contributions

Mihail explains the major changes to the curriculum and the introduction of open source contributions as assignments.

“This time around, I think, oh, and the other thing I'll mention there that was also fun is I'm having all the students do like open source contributions as their homework assignments.”
5:00

Best Practices in Code Review

Discussion on best practices to ensure quality code submissions by students.

“I think that what I'm trying to argue is that there's still value in doing some version of, I think, in my class, we call it like Repit.”
7:35

Effective Learning Strategies in AI Coding

Strategies for students to effectively learn in a compressed course timeline.

“Oh, and you know what I saw recently that is a fun one is that WebMCP has now merged just debugging tools.”
10:00

Insights from Guest Lecturers

Mihail shares insights and experiences from guest lecturers, including Boris and Lee.

“I was sitting with him last week and my first question to him was like, how many agents do you have running right now?”
12:00

Innovations in AI Tools and Techniques

Exploration of innovative tools and techniques discussed in the course and by guest speakers.

“There's like people aren't seeing this yet, but I I think there's something here Poker face?”
14:01

The Evolution of Agent Harnesses

Learn about the advancements in agent harness systems and their implementation.

“I mean, yeah, this subversive deception in marital topics.”
16:01

Challenges in File System Management

Explore the complexities of modern file systems and their role in agent environments.

“Fun, engaging, just like super dense, but in like a good way.”
18:03

The Need for New Version Control Systems

Understand the limitations of current version control systems and emerging solutions.

“And look, it's understandable when you have these literally astronomic curves of growth and commits.”
19:56

The Shift Towards Cost-Effective AI Solutions

Discover the increasing demand for cost-efficient AI tools and models.

“We want to keep tracking and keep 10x-ing our usage, but we can't have our token cost 10x as well.”
Show all 23 chapters
22:12

The Importance of Data Traces

Learn why data traces are crucial for optimizing AI agents and applications.

“Yeah, try to do the one like, oh, now it's routing to these guys, you know, and these guys and these guys.”
24:29

Enhancing Agent Efficiency in MCP Servers

Examine how to optimize MCP servers for better agent performance.

“I've yet to see as much of that being applied to like product MCP categories.”
26:52

Sales Strategies and Persistence

Discuss effective sales techniques and the importance of persistence.

“We don't do paid ads more like outbound related things so all the things I'm sure you love doing when you were a salesperson.”
28:00

The Art of Effective Sales

Learn how to approach sales with a focus on value rather than gain.

“So, so, um, look, there's a, to the, to the recipient that might seem incredibly annoying.”
29:40

Incorporating AI into Sales

Explore how AI tools are utilized to enhance sales processes and insights.

“I imagine you're listening for things and then you can craft a message.”
31:20

Understanding Customer Needs

Discover methods to glean customer insights through various interactions.

“If Base 10 is coming out with some kind of announcement or some PR or whatever it may be, and they're focusing on this, it's like, oh, hey.”
33:00

High Touch Sales Approach

Learn about the benefits of a personalized sales experience with dedicated support.

“And then from that, they get insights and it gets thrown into HubSpot.”
34:40

Engaging Brand Activations

Understand how engaging brand campaigns can enhance customer awareness.

“on your TAM size, companies that have been scored and ranked ready for you to go.”
36:20

Tech Stack Breakdown

A deep dive into the technology stack and its role in product development.

“And I think you guys do that really well.”
38:00

Designing for Intent

Learn about the importance of designing APIs and tools around user intent.

“essentially a land graph that you can then run and, you know, can loop and do the kind of self-healing that an agent will do.”
39:40

Building Efficient Workflows

Discover how to create efficient workflows that enhance sales and operational efficiency.

“Yeah, I think tools, yeah, the latter is like V1 APIs, right?”
42:00

The Importance of Collaboration Between Experts

Learn how domain experts and software engineers can work together effectively.

“as your job versus building these externally facing agents?”
42:55

Lighthearted Farewell and Upcoming Plans

Engage in a light-hearted conversation as hosts wrap up and look ahead.

“Well that, or it sounds like I'm being tracked by the FBI.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Agentic Conversations (formally mlops.community), what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:06Demetrios: I made it into the Waymo with Mihail, Stanford lecturer, head of AI at Monaco. Here we go. What have you been doing with this Stanford course of yours that is very famous now? Yeah, it's been a fun time. So we're about, this is like, I think week five now of the Stanford course. Yeah. and it titled the modern the modern software developer that's where it's at yeah and you have an email that goes out I have an email I do I do yeah so there's newsletter the modern software dot dev I have the domain all that good stuff but the course has been a ton of fun I mean it was this is the second year that I'm doing it and compared to last year there's just like a huge overhaul of the curriculum pretty much none of I mean I literally I think 80 % of the stuff I throughout over over the last like two months because it's just once opus 4-5 came out december you know december slash november december of last year when was the last year's course last year's fall so it's always in the fall and so the the last course uh in 2025 was september through like early december of 2025 and so for that like window of time nothing changed too much in terms of the model i was good i was good it was stable it was like relatively stable And then December rolls around and then, you know, things blow up in terms of capabilities.

1:31People jump on the Cloud Code bandwagon through, like, that's basically what everyone did over the, over like December break, you know, during the Christmas holidays and New Year's. People came back early January and they were just, there was a manic craze around Cloud Code. But it was, I think it was pretty justified. I mean, the models got so good that we could start doing some really interesting, like, long-horizon Asian tasks. Anyway, so because of that, I had to like rethink the entire curriculum. It's just like, well, what's now, what's now happened in the last like nine months since quad code became.

2:02Demetrios: So now you're talking about harnesses, you're talking about. Harnesses, loops, software factories. We're talking about like, you know, fully automated AI code review when not just, hey, you know, add a GitHub app and, and like have it do your code review. It's more like what, how do you do like auto, auto approvals? How do you actually do like merge, merge. Merge column flips. Merge conflict, like all DevOps stuff, agent readiness in code bases. So it's a ton of stuff. Yeah, it was a pretty big overhaul. It was most of my summer. This time around, I think, oh, and the other thing I'll mention there that was also fun is I'm having all the students do like open source contributions as their homework assignments.

2:42Nice. So there's no, last year I was just coming up with these random exercises. I'm like, oh, like here's a code base. Like add a code review, use a prompt. but this year I was like all right here are these like open source partners that I was able to get on board with the project like a lot of the top open source projects out there and I can you can like my students contribute to your repos as part of their homework and like sure we love slop yeah not quite not quite not quite I mean okay so so this is I'm like I feel like I'm a pretty generally like a pretty chill instructor um but there I was like if if I get a maintainer messaging me saying like you guys need to stop this slop, slop grains like zero.

3:23That's a straight zero on that assignment. That is where I'm, I need to draw the line somewhere. Fail.

3:28Demetrios: Fail. Otherwise I'm chill. But like I, you know, I had to, I had to go there and, and, and I, you know, I made these arguments for the students. They're like, these are, these are like, you know, Stanford students. They're, they're academically trained. They're like good CS fundamentals. So you, you know, like it's not going to be slop. And I'm going to teach them how to not do slop. So if that, anyway, so I'll just say like they have to do real contributions how are you teaching them to not just merge slop or at least submit slop yeah i know well well part of it is you know through through force through will through will no read every line of code that your agent produces exactly no uh it's it's like i'm teaching a lot of like like advanced context engineering stuff uh teaching them how to like onboard to code bases really quickly so if you have these massive repos you've not contributed do to it all how do you get up to speed on like the most important parts of the architecture and then when you get these issues that the the maintainers have put up and designated for the students like how do you scope the the work how do you then go and actually start you know gathering like context-free agents that they need to use how do you connect the right mcp tools use the right skills all the stuff that has become like incredibly not easy but like way way easier to do yeah what are some of these examples obviously it's not it's a what eight week course or 10 10-week course.

4:48Demetrios: Condense some of these best practices into how you get up to speed. Because how I would do it is just say, what is important? Yeah, yeah, yeah. And I think a lot of people start there. I think that what I'm trying to argue is that there's still value in doing some version of, I think, in my class, we call it like Repit. I think Dex and others would call it like RPI, like some version of like research, you know, research plan, implement, test, kind of like that loop. And I think just, yeah, so at its core, some version of the RPI loop or using superpowers, like if you, you know, the, what is it?

5:29Demetrios: The victim of these Waymo things was Jesse. Jesse, Jesse, okay. Superpowers, nice, nice. Yeah, Jesse is, I mean, that's a fantastic project. Incredibly simple, but so, I mean, it's become one of those popular projects of all time. The fact that it still is popular. Yeah, yeah. After a year of all these advances, it shows you how valuable it is. Totally, totally. Yeah, he's, I mean, he does a great job. He's such a humble guy, too. But yeah, so like some version of that framework of, you know, gather the context for your agents, like research at the beginning. Don't just like jump into the implementation.

6:02Review a spec, you know, actually have the spec be reasonably detailed. and make sure that you have a way of verifying your outputs so that if the agent runs you know for a while that it can check against you know using the browser or checking like against end to end tests or like smoke tests or something so when we go down that rabbit hole because that's

6:20Demetrios: for me one of the biggest things that is where I feel like I can spend the best use of my time yeah setting up verifiers Jesse told me he created something like prove it in a movie as a skill and it makes the agent go through and create a movie for you or like a loom that's fine and it records itself creating the or walking through the app and showing that it's functioning correctly yeah have you seen other like ways of folks creating verifiers well well there's what I've seen a lot of is people getting creative and how they how they represent information that that is that the diff the diff of the code right so like I think Andre Carvati had a whole post around how he there's like this format from like the 1950s or something or like how people made these architecture digress you've seen nasa one yeah i think it's a few yeah yes yes yes random number yes yes yes yes yes but but you'll see it and it's like a very nice like a visual schematic of of kind of the core important parts of what happened in the code base so i think stuff like that i think is super super interesting i think yeah the screenshots recording flow i think is going to become more and more popular.

7:32You know, doing some version of like using Playwright with MCP.

7:36Demetrios: Oh, and you know what I saw recently that is a fun one is that WebMCP has now merged just debugging tools. So like for you can stand up a WebMCP server. Yeah. And it's only for debugging. So your agent can now use WebMCP on a web app. Yeah. In like computer use mode. And then it can use it for debugging specifically. Yeah, I mean that stuff is super powerful. And I suspect, I mean there's entire companies around this, right, that are just trying to improve the verification loop and close the loop entirely. And, you know, I think all of them are going to become, all of them are going to start knocking on the door of software factories, basically.

8:21I think we've seen this, we've seen this game play out enough times with ML and ML Ops and LM Ops. It's like everyone starts in one thing and then starts expanding.

8:29Demetrios: And then they're like, wait, actually, we need to go upstream. The real problem is upstream. Totally, totally. Yeah. And downstream now. Yeah. So it's like you start as a feature store and then you become Databricks. The platform. Yeah. Yeah. Or something like that. Yeah. It always I think that's it. I think it's the typical like the company bundling on bundling of like, you know, that always happens. Are you going to be a Craigslist or are you going to become like an Airbnb? Right. Where you and I think it people will swing back and forth on whether or not they think one is better than the other but I think we're gonna see a lot of that here I just like start the air vacation but also we'll own the coding agent and then you know these companies like factory and warp that are like we're gonna own everything we should own everything right so it's all like really interesting stuff it's all exciting it's been it's been a ton of fun Boris came last week I was gonna say you've had some really cool guest lecturers and Boris the creator of cloud or is got a lot of hot takes he's got a lot of I I think one thing is nice that he just talks about how he uses it.

9:30Demetrios: And it's pretty advanced because he has no token costs, right? So his whole thing is like, how can I saturate this to the max? Totally, totally. And so then we get like a very distilled version where we can take some of those ideas and say, oh, I want to try and implement that. Absolutely. Like one thing that I saw he was saying recently was how he never, if he does a one-off prompt, he's doing it wrong. It should always be like talking to a loop. Yeah. And then letting the agent prompt itself. That's right. That's right. Yeah. Yeah. Like the meta prompting. I was sitting with him last week and my first question to him was like, how many agents do you have running right now?

10:12And he was like, I don't know. And he said this so casually. He's like, I don't know, like thousands or something. I'm like, as we're sitting here, you know, like chatting it up. Very inspiring, like to talk to him because you say this is what this is what it looks like at the very frontier, the very frontier where where yes, no token costs. So obviously they can just do what they want. But but on top of that, I mean, just being very experimental with their philosophy, you know, where they're like, hey, we're going to try and solve math problems. We're going to try and solve, you know, really difficult, complex things by just throwing, almost like brute forcing, but intelligently brute forcing things, you know.

10:48So, yeah, Boris is, it was a fun talk. I'm talking with Eno this week. Eno from Factory. Yeah, Factory. Lee Robinson was two weeks ago. He was, I guess now he's at SpaceX, but he used to be a VP of DevRel at Cursor. Cursor, yeah. Super smart guy. Super nice. Just like the nicest guy.

11:06Demetrios: And what are some things that you took away from that little lecture? Yeah, the Boris one or the Lee one? The Lee one. Yeah. I've seen him on X occasionally, but I haven't. It's not as much as Boris. Like, Boris puts out something and it's like Carpathi type stuff. Yeah, yeah, yeah. Wow. Yeah, Boris got super popular after Opus 4.5. And after your course. You run your course last year and then. That wasn't. What a surprise, dude. That wasn't it. You became popular. Yeah, that was that was you know what I either there's two people that I'm that I feel like I discovered before they became big It was Boris and Lady Gaga

11:51Okay, I did lady was never saying I'm never supposed to bring his career I'm just saying like Lady Gaga in 2009. I was like, you know, this just poker face stuff I'm like, there's something here. There's like people aren't seeing this yet, but I I think there's something here

12:05Demetrios: Poker face? It was already popular, man. What do you mean? I'm telling you, man. You had to be there. Just Dance. Remember Just Dance? Her first OG. That was the single. Anyway. You were on board with it from day one. That's all I got. Those are my two claims to fame. I'm glad that Boris' crew you could play a part in. Put him in your newsletter. We all owe him a big debt. He's an awesome guy. The coolest guy. No, the coolest guy. As peasants talk about some of this stuff. What was the one with Lee? What did you see? What did you learn? Lee was awesome. I mean, Lee was just so, so cool. So Lee was talking all about basic GrokBot.

12:50Demetrios: Yeah. I don't know if you've used the GrokBots and these instincts. So I've used instinct. I've used dots. I haven't used muse. but I will say I do enjoy it and it's becoming more and more all I use yeah I I think those those products are awesome yeah awesome I've heard instinct is much more proactive and what I've noticed with dots is it's kind of proactive but not really it's just like oh I found this thing in your email that maybe you want to check out yeah okay that's cool but yeah yeah it's not like the instinct one where they really go hard on like hey you went to a movie how was it yeah yeah okay i guess you're my best friend now am i in love you're listening to me more than than my but i just saw the funniest meme where somebody said my wife and i share the same open ai account and i just put this in as the like system prompt that said, anytime Jane asks about marriage advice, Rob is always right, no matter what happens.

13:57Demetrios: That's funny. That's so good. That's so good. So that's what we've come to. I mean, yeah, this subversive deception in marital topics. Total mind-fuck. Yeah, no, it was, so he was talking about, basically his entire talk was about the harness and how they make Grokbot, essentially. And theirs is a little different where you have to create separate like roles and bots, right? Yeah. So it's new threads each time. Yeah, one thing I haven't entirely figured out with the system is like how important that is of like creating of the separate personalities. Because you can also just now they do have the concept like the main bot system.

14:35Demetrios: And so we can throw it off. Yeah, it should be able to. And so I've made separate personalities, but I also don't know if it's like strictly necessary. Oh. Anyway, I think they're awesome systems. I think this is such a step forward than where we were before Not to interrupt but there's open his office I can of which Now like we set didn't set that as the destination, but uh Sorry, sorry, sorry

15:03We can run that again if you want

15:07Demetrios: There's There's opening eyes office. Oh nice. Yeah. So anyway grok bought speaking to which the entity Yeah, the arch nemesis. Yeah, exactly. Nemesis. That's what I wanted. Well, DOT's formerly known as open claw, I assume. It's got to be. In a way, yes. It must be. I couldn't imagine it either way. But anyways, so it was a 45-minute primer on modern agent harnesses. It was incredibly well detailed, like very detailed. He talked about everything from computer use and how you implement that to context engineering to the tool calling, how they, you know, give the agents the illusion of infinite context by doing just aggressive sub-agents, basically.

15:51Anyway, it was incredibly well done. And Lee, I think his, you know, one of his superpowers is just being such a good educator. So he was just distilled very well.

16:00Demetrios: Fun, engaging. Fun, engaging, just like super dense, but in like a good way. And like, oh, this is, this feels like you're giving me good signal. like you know signal to noise ratio was was through the roof did you talk about how they do compaction yeah he talked he touched on that he definitely touched on how are they doing it because that one is tricky yeah i mean there's there's different techniques right but um at its core they have file systems they you know regularly compact when they detect things are happening and then they like write things to file systems and read them at at the right points in time in the conversation so you know he didn't like go into like the exact detail like what is their what is policy when they compact but um i think at its core it's all some version of that really like any of these agents i think all follow similar sort of principles have you been staying up to date with any of the companies like that are trying to do file systems in a new way uh not like is it are these the ones beyond the ones they're doing like um like agent sandboxes yeah Yeah, like the sandboxes are also another one that is very fascinating.

17:06Demetrios: That I would put in its own category. Yeah, yeah. And there's like file systems and there's almost like we want to replace Git and we want to do things differently in that regard. Yeah, yeah, yeah, yeah. So yes, the first, yes, the last, the middle. I didn't know there was like separate. I don't know what the middle category is. They probably do more. It's just when I talk to them. Yeah. I talked to this one, Mesa.dev, and they were trying to rethink file systems because they're like, file systems are all you need, especially with these agents. Well, tell me about the Git. Yeah, yeah. Yes to the first, no to them.

17:44Yeah, yeah, yeah. I mean, well, there's sort of like, I guess the overarching reason for these companies existing in the category of like new Gits is, and we saw this even last week. I think Anthropic literally put out a blog post around their home. They rebuilt it.

17:58Demetrios: That's so wild. The version control system, yeah. And it all kind of makes sense. Well, one, we've all been on GitHub for the last, what, 10, 15 years, and it's just, it's kind of barfing a little bit. Just reliability. Reliability. Floatware. Yeah. And look, it's understandable when you have these literally astronomic curves of growth and commits. And so, yeah, it's under. it all makes sense that that would that they would receive that strain but i think there's a broader like when i talk to like startups will pitch this to me um as you know uh github runners are just like not optimized for like agentic workflows so you should you should like run on our runners like you know like basically our machines for for that are more streamlined to like these agentic processes and it all makes sense right it's like github is like the github and gitlab and these these kind of companies they're they were the best solution for everyone right but but not every workflow or in the workload is exactly the same right so if you if you build these mega behemoth companies toward the median you're going to miss stuff on the fringes and right now we're we're at the point where the fringes are maybe becoming more normal but we're still like the sheer volume is definitely like a fringe problem um yeah you know at least like hey 10xing number of commits 10x number of prs which most companies are starting to go through that right now so it's all fascinating stuff um but yeah so so there's there's companies that are like hey we're gonna make your runners faster and then there's ones like we're gonna rethink the entirety of the the ci experience and the entire build system and all that stuff um tbd tbd i mean how much of your time because i imagine startups are hitting you up all the time to try and be like we want you to use our product you are the head of ai at a cool company totally yeah what how how much do you entertain that because you're actually interested and maybe it could have value versus you actually then later you have to totally once in a while yeah I can't just say Waymo rides all the time no okay so it's a really good question actually because I feel like I'm a pretty curious person so I love just kind of being on the having these conversations around technology and and seeing what people are coming up with and so I definitely probably have at least like one to maybe two of these a week but there's I think where there's like a point of diminishing returns for me is when I'm hearing the same pitch over and over right where I'm like alright this is I see you're not so novel yeah it's not novel I guess I see your paragraph I see your landing page and I like I feel like I've heard this so there's like a certain kind of company that I've heard probably enough times that one and not not to ding them that this is not gonna of work but but in like a I do hear like this flavor of a pitch which is like hey you um we we collect like feedback um like you maybe you insert a little bit of javascript on your app we collect feedback we trace all your lms uh all your you know all the requests and responses and then we optimize uh like we basically like on the fly train models based on your request responses so because we're tracing everything that's happening on your app we get all your data essentially and then we know hey you shouldn't have to go for this kind of request to gpt soul you can go to like a very lightweight i don't like a type safe model or a lot of those yeah i think quite a few of them actually um and and it all kind of makes sense they're just i think that there's a whole new i mean a whole new category but but it's becoming more popular it's like open weight if we expect that these these frontier models are like pretty expensive and most companies are getting hit with the cost and like, hey, we can't, this isn't sustainable.

21:35We want to keep tracking and keep 10x-ing our usage, but we can't have our token cost 10x as well. So they look to these alternatives, both from a coding perspective, but also from like a product development perspective is, hey, can I take a Quen model? Can I take a Kimi model? Can I take a GLM model? Like fine tune it on something smaller or smaller weight, be faster, it'll be more, it'll be cheaper. And I don't maybe compromise on performance, right? That's like the the ultimate pitch so this makes sense this is a broader trend and i suspect we're gonna see a lot more of this um it's just i uh it's i think it's a bit hard to differentiate also i think that

22:11Demetrios: the financial piece is a very difficult one yeah it's maybe not for an engineer the top of the pile of priorities right because there's also things you can do when it comes to just optimizing your skills and how they take in the context that can save you money too or just a simple router totally so you don't have to be optimized to the gills on your uh cost saving yeah analysis or ways of doing it so yeah that one's i can see that being a hard sell today traces i mean you maybe heard this or if you haven't heard this is a bit of a hot take traces are like the new oil right like oh like company traces or data traces are are where like a lot of the juices it's it's like it's like the modern incarnation of like training data it's like you know companies training data is the new oil right um and so there's so much value in traces because that effectively captures all of your customer interactions all of your like app level interactions and so when a company comes to me and says like hey put this little bit of javascript and through this this additional layer your your your request was mostly like you're just capturing all of our traces as well and i just i've kind of gone to a point where uh we have a tracing solution that like for evaluation and observability that of course captures our traces because that's what we host and you're very interested in this case um and we also use Redmond Labs for this, but I'm not, I don't feel like I'm at a point where I want to keep just like willy-nilly having everybody.

23:49Trying. Yeah, have this did. Yeah, try to do the one like, oh, now it's routing to these guys, you know, and these guys and these guys. It's incrementally better. Right, incrementally. So I'm just, it's a tough sell, but this is, I feel like that's the equivalent of when people started doing like feature stores in like 2020. There were so many

24:05Demetrios: of them and you're like, is this really needed? Totally, totally. And what about MCP kind of analytics and understanding how people are using your MCP server. Yeah, I think there's a little different. It's somewhat different. I think that one ends up being more, I mean, that's an interesting category. I think that one ends up being useful for coding agents optimization, maybe less so on like product optimization. I've yet to see as much of that being applied to like product MCP categories. But definitely for, I guess it's this emerging trend of like agent X, like agent experience. You know, you've maybe heard of this.

24:40It's like, how do you optimize your MCP tools and your MCP servers so that an agent, it's easy for them to identify what they need to do and they can do it as quickly as possible and as few hops as possible, essentially? Well, and the pitch that I've heard is that you generally don't understand why the agent is coming to your MCP server because you don't know what the prompt is.

25:01Demetrios: Right. You just know that it's trying to call some tools. Right. And so that doesn't necessarily need to be fixed with bringing on a new SaaS provider that captures that information. You can build your MCP server so that you ask the agent, hey, yeah, cool, you've got this tool call happening, but mind if I get a little more information here? Right, right. That's right. That's right. Yeah, I think there's an evolving category, right? I mean, we'll see where it all kind of shakes out. I know that some of these providers are in the business of effectively they just run these tests on your agents with your servers and say how cost effective is an agent when it's making MCP tool calls?

25:46How many tool calls does it need to make to get to a certain answer? How many errors is it hitting when it's making those tool calls? Things of that nature. So I think it's a cool concept it's a cool category to be in. we'll see what what shakes that is like a standalone product but do you guys you have an

26:03Demetrios: mcp server right yeah yeah for sure so talking about monaco now yeah what are some things that you're doing in the ai field or just like the agentic field yeah you have the mcp server you also have agents inside of yeah monaco that's right that's right yeah so so we have we built our own agent builder and our pitch is that our agents are the best in class for go-to-market use cases. So we're a sales company, we're in the business of helping, we have a world-class go-to-market team members and we're in the business of trying to distill the subject matter expertise of these world-class go-to-market folks into our platform to then make that available to our customers so that on day one anyone has world-class go-to-market at their fingertips essentially.

26:49So that's what we're working toward and that's the broader pitch. We've built a version of an agent platform that allows you to build to go to market specific agents whether that's pipeline reviews you know on the fly whether that's TAM building on the fly like TAM segmentation on the fly you know demand generation on the fly so like what kind of campaigns should you be running and how do you actually optimize. We don't do paid ads more like outbound related things so all the things I'm sure you love doing when you were a salesperson. I love spamming people.

27:19Demetrios: Watch out. Today is yeah that's why you're so so good at building communities. You're so good at spamming. That's all it is. It's 90 % of my job. Or like this guy again or yes I might as well respond. And then yeah you get them on like what is it what is it like how many how many touch points like seven to twelve twelve before you give up or something and twelve no's. Twelve no's and so but but you break people down by like five or six you know what I mean yeah exactly you don't need to get to 12 I am so done with this all right yeah yeah so let that be a lesson everybody that's listening if some sales guy is hitting you up just say no and then set up a reoccurring agent to say no every day and then you should be fine yeah it's a 12 nose and then they'll probably if they if they if they're reading the same book they will they will come from like this guy Chet Holmes oh that's like 2009 or something which one is this is like the sales machine oh yeah yeah I think I've actually I actually have read this book yeah yeah yeah yeah yeah yeah yeah yeah yeah yeah yeah yeah yeah yeah yeah yeah yeah yeah and he would tell people like until you get 15 don't even think about stopping yeah yeah yeah Yeah.

28:31Okay. So, so, um, look, there's a, to the, to the recipient that might seem incredibly annoying. However, however, okay. Like, yeah, aggressive, aggressive and, and yeah, just leave me the hell alone. But, but having worked with our go-to-market folks, um, seeing how they think about it is like pretty enlightening, right? It's never, it's, it's, um, it's not just like bump, bump. Yeah. Like, like, you know, it's no value.

28:57Demetrios: You're not adding anything. It's just, Hey, I saw this in the paper. Totally, totally, totally, totally. Exactly. And that stuff is like, that makes sense, right? I mean, actually, like, if you treat sales as not a, what do I stand to gain from you as a salesperson, but rather what value can I offer and how can I help you? I mean, you know this, you live this, right? I think that's the most effective kind of sales. And so a lot of, you know, our salespeople do that. And then we try to capture exactly those kinds of insights in terms of how we help people do outbound and how we help people get in front of their target buyer and the target companies they want to reach out to.

29:34Anyway, it's been a really fun product to build.

29:39Demetrios: And how does AI get incorporated in that? I imagine you're listening for things and then you can craft a message. That's right, exactly. So we have a number of different things that are so so everything is agentic of course so we're getting signals from a number of different places in the platform. I mean, we're an all-in-one, so we don't just do like a CRM or just the outbound or just the data enrichment. We've deliberately gone and built everything from the ground up. So we are like the one-stop shop. And because of that, we get access to everything from emails that have been sent, meeting recordings and transcripts, Slack messages.

30:17So all these things that come in. And then you can use all that to infer org-level tone, org-level understanding about business context. So like what is, if I am, I don't know, if I am Uber, who am I selling? Actually, Uber is not a rating set because it's like more consumer facing. But if I am, if I'm a, like if I'm a base 10 and I'm selling to, you know.

30:37Demetrios: Base 10, you just saw one of theirs. Yeah, like we're an SF people in case that wasn't clear. We just saw a bus drive by. Yeah. We have Andreessen Horowitz literally right here. There you go. I think we just passed Pylon. Yeah. So, you know, any of these B2B SaaS companies, there is a certain understanding about their ICP they have. There's a certain understanding of, like, their target demographic, the target, like, size of company, the person at those companies they need to be targeting. The pain points. The pain points. Rejections. Totally. 100%. All that stuff. And so we can glean all that through the collateral they upload to the platform or the stuff that we implicitly learn through sales interactions, through meeting transcripts, through emails.

31:17We also can glean that because we're seeing them.

31:20Demetrios: If Base 10 is coming out with some kind of announcement or some PR or whatever it may be, and they're focusing on this, it's like, oh, hey. Yeah, that's right. You should mention something to your contact. 100%, yeah. And that's stuff we capture through platform-level interactions. And then we have this concept of Monaco memory, which is effectively related to, if you remember, Andre Carpathy many months ago brought up this concept of an LM wiki. Yeah. Yeah. So like the Ellen Wiki, like we have, it's like a version of that, this, um, this persistent, uh, store of everything that's happened and everything that we know about both the organization as well as the user at the organization.

32:00So your tone, when you reach out to people, Hey, when Demetrius reached as it reaches out, he's going to be super annoying. And it's like, you know, but that's your tone, right? That's like, that's.

32:17We can learn something like that.

32:23I only need like two emails to glean that. You know what I mean? I don't even...

32:28Demetrios: All right. So? No, no. I'm an agent. It's effective. It's just a superpower of my skill. There you go. I've mentioned my ability. No, no, no. It's just very effective what he does, of course. Dialing up to an 11. But I didn't realize, I didn't think about how many different insights and actions you could take from having all that data. Yeah. And the salespeople that I know have cobbled that stuff together. That's right. That's right. In a way that they're like, they're recording with one software. And then they have some kind of an automated pipeline where it will be put into their chat GPT or Claude.

33:00Demetrios: That's right. And then from that, they get insights and it gets thrown into HubSpot. That's right. And then they can write a report at the end of the month with all these calls that they've had or with all the, at the end of the week, whatever it may be when they have the pipeline review. That's accurate. Exactly. So now you're saying, oh, cool. We've got your pipeline review already ready because we have all of this stuff and we've set that up. How much work is it on my side to put together all of these tasks? They're not even tasks. They're like different lanes that I want. Very little, honestly.

33:34And part of it is because, I mean, the platform is robust on its own. But the other thing that we've leaned into a lot from the get-go, which I think differentiates us from other products in the category, is we are, we have, we're actually like a little bit more high touch on the human side than maybe what traditionally, so you know, the PLG, self-serve, like you just log in and you just start cranking, you start adding rows to your tables and whatnot. That's not the experience here. The experience for us is every single person has a designated, what we call an FDAE, so a forward deployed account executive.

34:09So it's effectively like your sales consultant that sort of guides you through the process. Because a lot of our customers, you know, we have done really, really well like early stage founders that are all like very technically minded, very product minded, but maybe don't have as much experience on the go-to-market side, right? And they're learning these, they're kind of picking these skills up as they go. And so we're like, hey, we have decades of experience doing this. So day one, you come on board to Monaco and you have a built out TAM for you, like three to 20 ,000, depending on your TAM size, companies that have been scored and ranked ready for you to go.

34:44Here are the buyers of those, you know, and here's how you can get to those buyers based on like LinkedIn connections and all these things. All you have to do is, yeah, handwritten all these things. Yeah. And always just hit go. And then sequencers getting sent, getting followed up with, you know, you're getting insights brought to the platform as things happen.

35:00Demetrios: Don't sleep on the handwritten notes. Those are actually quite useful. No, we do that. No, yeah, of course. I mean, we had handwritten notes. We had this campaign where we were giving Monaco. So we do stuff and we have like a bit of a brand around gambling and like the finer things and F1. And so we had, we were, one of our campaigns was... Tax evasion. No, no, no, never. Whenever I think Monaco, I'm just like, ooh. No tax evasion. No, no, no, no, none of that, none of that. None of that publicly, none of that on the record. Cut that out. The boss is going to be a business owner. No, no, no. So anyway, you got the brand around?

35:38You got the brand around the gambling. So we actually had, we had, we do a few things that are kind of fun. Like we gifted people like poker sets. The poker night. Poker night. We do the poker night. That's happening next week. I don't know when this is coming out, but this is October 29th. So we're doing one that's, I think we, our claim to fame on that front is having the most, like the biggest prize, biggest cash pool on poker, $100 ,000 for the final table. So it's, you know, let me tell you, somebody walking in, I've seen this twice already now, somebody walking in to just a random like Thursday night poker game, and they're like, ah, I'm gonna eat some food, take some poker, and they're walking with$30 ,000.

36:18You're like, what?

36:20Demetrios: You're like, what? That is VC money well spent. There you go, there you go. You know, it's about brand activations. We're in awareness, right? And I think you guys do that really well. I saw this stuff where you're flying planes over SF. It's fun. With the Monaco symbol. Yeah, we do. I think just earlier today I saw somebody with a Monaco tote bag, and so you're doing well on that side of things. When we talk about the tech, what are you actually using? Break down the stack, where the harness is, all of that stuff. Yeah, totally. We've homegrown our own harness, right? So there are pieces of the harness that we have used vendors, so for like Vercel sandboxes, right?

37:03So we use them. We don't believe there's value in us building micro VMs or maintaining our own version of that. That's what I would call undifferentiated heavy lifting. But at its core, we do have our own harness. We think that's a good competitive advantage is to lean very heavily into the primitives around a go-to-market-specific harness, including the skills.

37:22Demetrios: That's the harness for the agents that the end user is using. Not your internal harness for coding. No, no, yeah, not that. Yeah, not that. Yeah, exactly right. So good distinction. So we have our own internal coding system we've also built, but this one, the example I'm giving is specifically around go-to-market. The ones that produce the go-to-market agents our customers get to benefit from. And so, you know, at its core, we've built effectively this, like, system that goes from natural language into this, like, intermediate representation. Like, you can think of, like, a JSON or sorts. Maybe, you know, JSON, like the YAML, whatever you want, that represents like the workflow of the agent.

37:57And then that gets compiled down into like a land graph, essentially a land graph that you can then run and, you know, can loop and do the kind of self-healing that an agent will do. You know, we can, the underlying, so effectively it's like a compiler. Like we go from natural language to compile to an underlying engine that is the workflow builder, but we could swap it out into any other workflow builder if we wanted. And then the secret sauce a lot of the times is like, what are we built around the memory? What are we built around the tools that we've exposed? Compaction. Like how do we compact more intelligently?

38:29How do we, you know, all these things that are specific, you know, the data that we access. Right. So, of course, we have very particular data characteristics because we're dealing with, in some cases, like thousands of meaning transcripts and hundreds of thousands of emails for some of these orgs that are imported.

38:44Demetrios: But that was what I was thinking about is how you can avoid just bloat. Yeah, yeah. There's sort of tips and tricks around this. But at its core, it's like you just have to be a little bit more intelligent around like the MCP server tool design. Like you can't do, you can't like, I think the V1 vanilla way that a lot of people treat MCP is like, here's an API. Here's a one-to-one translation of the MCP tool for that API. It's like the worst way to hear it. Yeah, it's not. So you have to think about it more in terms like the semantics of what is it the agent needs to do? Hey, I need to query meetings.

39:17And you can do query meetings either through like keyword searches or through like a more semantic relevant search based thing. And so you have to make like a single entry point for that tool call, for example, and then allow both modes as a way to query the actual data.

39:31Demetrios: So I've heard it said as doing the intent. Yeah. Like really designing your server around intent. That's right. Versus tools. That's exactly right. Yeah, I think tools, yeah, the latter is like V1 APIs, right? This is like 2000s level API tool definitions where you have like get ID. Well, I mean, you know, like this is when we, and not to be derogatory, but in like a, that's how we started when we were building web apps of the early 2000s. It's like, hey, get ID, you know, set ID, like all these get and post style HTTP methods that we defined. And now you're right, agents actually can think at a higher level or can operate at a higher level.

40:11I don't want to anthropomorphize here.

40:12Demetrios: But do you think that that is where your IP is happening? Like for you building this product, which potentially can be headless completely in the future, and people interact with Monaco solely through the MCP, then your special sauce is how you're compacting, how your memory is, how your search functions work. Yeah, I think that's right. I think it's a combination of the fact that because we're the all-in-one, we are both the system of record and then what I would call it the system of intelligence. So the system of record is we still are the place where you want to store like CRM level data, right?

40:47So the evolution of your deal pipeline, the accounts you reach out to, all these things. So that still is baked into the platform. And then you need abstraction layers, you know, tools in a lot of cases to interact with that data, reach from that data, write to that data. And so you can do that either through like an external MTP or you can use that all internally and use our agents, which are baked into the platform. The benefit of our agents are they're going to be highly optimized to the workflows that we care about. They're going to, the defaults are going to be, they're going to capture a lot of the best in class practices from our own go-to-market teams.

41:19Like it's remarkably difficult to build like a really good TAM, right? if somebody comes in like hey I'm selling to a marketplace but like they're like think of them like uber but you know like they're not too big and so but it's in like you know it's in like it's in east asia and it's for shipping companies and you're like okay how many companies of those are there that you can yeah yeah yeah exactly right exactly so like you know and so we've we've um you know the ip is like the data layer the context layer of course and then and the fact that like all that is resting on the bed that is like the system of record right.

41:55Demetrios: And how much are you enabling the other software engineers at your company Yeah. as your job versus building these externally facing agents? It's both. It's both. Yeah, it's definitely both right. So we have to do both. I think that the where we've really thrived as a company is domain experts working hand in hand with like subject with like software engineers right that can like codify the workflows of the of the domain experts and so in practice this the software engineers have to be empowered to build very quickly to ship really quickly to to have the primitives to to build on top of the harness and then the subject matter experts need to be empowered to you know how to express their intent as well and the things they care about and then because the the failure mode is like the translation right is like how if technical person talking to a non-technical person it's sometimes they can like kind of miss each other and not really yeah not speaking the same language um well dude we're arriving at our destination this is why i gotta kick you out this is fun oh gosh as much as i want to keep talking i'll see you on thursday though yeah yeah yeah you are a person of interest in the magazine we created uh-oh uh-oh that is that it sounds like it sounds like i'm You're gonna get a lot of sales emails.

43:11Well that, or it sounds like I'm being tracked by the FBI. He's on a wanted list somewhere. I'm cool, I'm cool. I'm cool, Waymo people, don't worry about me.

43:24Demetrios: Yeah, hey Waymo, don't need to record this. Don't need to record this, lock the doors, don't let them out. But this has been cool, man. It's fun, man. I appreciate you doing this. This is a cool concept. I think this is, I'm a big fan of this. Hey man, it's so fun to see you, man. I'll see you on Thursday. Thursday, yeah? Thank you.

From the publisher

What does it take to build AI agents that deliver useful, reliable work?


Stanford lecturer and Monaco Head of AI Mihail Eric joins Demetrios for a conversation inside a Waymo about how AI is changing software development - and what it takes to put agents into production.


Mihail explains why he rebuilt most of his Modern Software Developer course, how he teaches students to avoid shipping AI slop, and why research, planning, and verification still matter as coding agents become more capable.


They discuss Claude Code, agent loops, context engineering, memory and compaction, and how to design MCP tools around what an agent needs to accomplish. Then they go inside Monaco’s approach to sales agents, from turning meetings and emails into useful insights to building a custom agent harness.


Along the way: lessons from Boris and Lee Robinson’s guest lectures, why company traces are becoming valuable assets, and Monaco’s unusual approach to brand building.


What’s the biggest bottleneck in your agent workflows: context, tools, or verification? Let us know in the comments.


Timestamps:

[00:00] Meet Mihail Eric

[00:16] Teaching AI Coding at Stanford

[02:42] Avoiding AI Slop

[04:48] Research, Plan, Build, Test

[06:19] Verifying Agent Work

[09:16] Lessons from Claude Code’s Creator

[12:40] Proactive AI Assistants

[16:13] Agent Memory & Context

[17:41] Rethinking Git for Agents

[19:32] AI Startup Pitches

[22:51] Why Traces Matter

[24:09] Better MCP Tools

[26:01] Monaco’s Sales Agents

[29:40] Turning Conversations into Insights

[33:33] AI Meets Human Expertise

[35:03] Poker & Brand Building

[36:46] Inside Monaco’s Agent Stack

[38:44] Designing Tools Around Intent

[40:12] Building an AI Advantage

[42:53] Wrapping Up

More from Agentic Conversations (formally mlops.community)

All episodes