Building AI Agents on the Frontend with Sam Bhagwat and Abhi Aiyer

26:52But at the end of the day, it's just like a rest endpoint that the LLM knows how to hit and get data, send data to and get data back from. My question is around that, you said that you're on there from day one. How does that relate to, I guess, the tools in Mastro. Is that the same thing? Are they different? Because I think of MCP a lot as like tooling, but should I think of Mastro tools in the same vein or are they different? One way of thinking about that is that an MCP server, like an agent is like a container for tools. But you could imagine if you have a bunch of tools, you could put them into an MCP server and you could just give it to your local windsurf agent or whatever and kind of, and sort of see like, hey, how does the Winserver agent interact with these tools when I put them in an MCP server as a container and give it to them?

27:40Great, that's what the MCP primitive is. But because it's this decentralized way of interacting with tools that you don't control, people have created these MCP servers that wrap tools to access GitHub or Notion or all these interesting third-party services. And because there are now all this, you know, we as devs, like, we have to write lots of integrations to do our job, but we hate writing integrations and nobody wants to write integrations. And so like in some ways, like these MCP registries have ended up being kind of like this decentralized integration hub that now your agents, your master agents can interact with because your agent can like take in MCP servers.

28:20And if you like feed in an MCP server as like a tool, essentially, it will be able to access any of the tools contained in that remote MCP server that someone else maybe wrote that you maybe had nothing to do with, but someone else created this functionality. And wow, now my agent is even more powerful because it can use all this third-party functionality that someone else just kind of built. Yeah, the low-key benefit of that is the MCP server doesn't have to be written in JavaScript. You could use tools from all over the communities that serve your purpose, but you're still connecting from your JavaScript written agent or application.

28:56But we do support you writing your own tools. of course. So you can write your own functions for tools and they work. We think of MCP like a better NPM now for this, but there's no governance there just yet. But yeah, you can also just use import from NPM and then make your own tools as well, but might as well just go on the MCP train. Yeah. Nice. I love that. It's like so much extension for free effectively for all of this. That's awesome. Building agentic AI apps isn't just about choosing the best LLM. Agents need short-term memory, long-term recall, and lightning-fast retrieval. Without it, you're left with clunky prototypes that never scale.

29:35You know Redis, the world's fastest caching solution? It turns out fast data is the key to good context, and good context is essential for fast, accurate memory. It's what makes AI agents actually work with your data. Redis for AI. The right infrastructure, the right tools, the only way to scale. Learn more at redis.io. slash genai.

29:57Have you tried building a text-to-SQL chatbot? If your AI agents don't understand your data, its definitions, queries, and lineage, they're forced to guess. And bad guesses mean risky assumptions. That's where SelectStar comes in. SelectStar automatically builds an always up-to-date knowledge graph of your data, capturing metadata like lineage, usage, and example queries. So whether you're training an AI model or deploying an agent, your AI can answer with facts, not assumptions. Stop the wrong SQL queries before they happen. Learn more at selectstar.com. The other question I had that came out of that, the discussion around your docs was, it was like treating docs as a second product.

30:42I have used the Mastra docs MCP immediately. Like I, it was like night and day difference adding it. And then like, it just knew everything and made it really easy to get, get up and going. And it knew what I wanted to do. And it knew what the capabilities of Mastra were like right out of the box. My question is, that's great when I know what I, that I want to use Mastra. My question is like around, you know, being a dev tools company and like having this new ish dev tool that may not be like, you know, in the memory of LLMs. like it might have come out after their training dates, right? How do you see that?

31:19How do you see marketing a dev tool in today's age as you're consistently getting more and more people that are not going to go to Google to ask, you know, what kind of tool I should use to build an agent. They're just going to ask ChatGPT or Claude. Is that something that you think about? We think about it a lot. I mean, honestly, like this is kind of our second time go-rounded. One of the great things about open source is the community and the fact that people can build things and share them with other things. Part of the community is also you can show up at AI Engineer in SF and it turns out someone else is running a workshop on Monstra and showing people how to use Monstra.

31:53And you're like, holy crap, like I'm showing up at a conference and someone else is running a workshop on this tool that we all built. This is crazy. This is amazing. We also like wrote a book. And this was just kind of like this idea that came to us one day when we were just kind of like, so the book is called Principles of Building AI Agents. it's again like it's 140 pages and kind of just walks through all the like what is rag what is prompting how you write good prompts how do you what is an agent you know what are the kind of like tool calling it walks through like all the kind of core core concepts that also just kind of came after we'd done a lot of these kinds of like whiteboarding sessions with other people that were trying to build agents and we just kind of had these interesting conversations and have these we'd kind of every time we had that we'd shared some insight to look oh wow this is really interesting.

32:37I didn't think that maybe I should be splitting up this like one LLM call into like 12 different LLM calls instead of trying to like have this one LLM call decide whether these 12 pieces of information, maybe I should just parallelize them and split. You know, so like we just had so many of these conversations and we're like, gosh, like it feels like everybody's figuring out how to do this. And maybe we should just like write a book at it. And I write pretty fast. And so we already had docs that were kind of like an outline and kind of a taxonomy, almost of those different concepts, except, you know, when you're looking at docs, it's kind of 20 % explanatory text and 80 % code.

33:12And so, you know, we just kind of took that, but then like flipped it where like we expanded the explanatory text where it was like 80 % explanatory text and kind of 20 % code and some historical background. And it was like a couple of weekends and I just kind of, we kind of like kept writing and kept writing and kept writing. Then all of a sudden it was like 90 pages and we're like, okay, like, I guess this is like publishable. So we published the book. And the reception has been really incredible. I think a lot of people have said, like, hey, I keep hearing about agents. But, you know, when I kind of read the book that you guys wrote, it kind of all made sense to me.

33:42And I actually understood how I would do this. I think the reception has just been like, we've been really kind of like humbled. And we try to pass out a lot of copies of books at like events and such now. Can confirm the book is great. I put a lot of pieces together in my mind. So yeah, plus one on that. For the LLM SEO technical bits, though. We do do LLM.txt and full text and all that stuff. And then, you know, it's pretty much kind of our game there. We're starting to get into like ChatGPT responses now, but that's probably just because we're kind of all over the internet right now. There are actually companies in our YC batch that's job is to help you get onto LLM's responses.

34:26Like that's their business. So there's probably a whole little playbook that they have to do there yeah one of the things too is just about the models basically scan the internet right when they're doing especially when you put them in deep research mode they kind of they scan the internet and they pick up what's on the internet so you know when your community members and this happens a lot right like tutorials about monster like that gets picked up we didn't have anything to do with that someone just got really excited about monster and wrote a tutorial and sometimes that tutorials in japanese or whatever because there's a huge industry community in Japan.

34:55That's been kind of also one of the biggest things that has probably helped us on that front is just like the community being excited and like writing things that gets kind of picked up. We've talked to other sort of dev tools folks, as you know, at Resend. And what basically happens is like when your tool has become the consensus choice among developers, the models can kind of figure that out just by like reading the internet and seeing how people are describing it. And so when your tool has become the consensus choice, you almost get like an amplification effect. And so I think that's great because like what that says is that you should build a great product that like people love.

35:32If you build a great developer tool that like people love using, everyone feels is like the right thing. And if you are successful in doing that, then like the models will kind of surface that to like people who are coming in asking them. That makes perfect sense. And so it seems like what we're doing to get your name out there for people to use it, like it's all amplified by these tools and they know how to surf the internet just like us. So that's good. And then getting into actually using these tools as new tools come out, you're positioned in the perfect spot with your own Docs MCP that immediately just like gives all of the knowledge to the LLM to help you.

36:06So that's great. One of the other things that's like really helped us a lot, we've kind of gotten the band back together from the sort of the core team that was at Gatsby. And I think we all working together and we all sort of like, we've done this before. And so we think we can work together pretty quickly to build great stuff. And so honestly, that's probably the biggest thing, like working in our favor is just having a lot of great folks that like working together and have like built an open source JavaScript framework before. Nice. Yeah. Yeah, for sure. One question I was ask is, Robby, you mentioned kind of like almost stumbling into MCP, like just being there right there at the right time to jump on it.

36:45I'm curious if there have been any other decisions or things that haven't panned out so well that you've had to kind of pivot from. So first thing we did, we tried building a prompt builder, like a utility function that helps you build prompts. This is when prompt engineering was all the rage and actually models at the time were okay with XML tags and stuff, but now it really doesn't necessarily matter. So we built something like that. And then that's when we really learned that we should listen to our community more because we like posted an experimental exploration. We got a bunch of feedback from our, and at that time we had nobody really in discord.

37:26I'm sure nowadays we get a lot of opinions, but like at the time we had like some early contributors that are just like, Like, this is why I would never use this. And it was like a bunch of valid reasons. And we were like, okay, let's just not, let's scrap this. We tried building like an MCP registry ourselves. And then we tried to get other registries involved early. But then with the lack of, Anthropic owns the spec and the governance, it's hard to try to like get in there and sort of shake some motion up. So we kind of just did our, you know, the registry registry thing. and then we'll maybe revisit stuff like that.

38:03The last big one that I remember is our initial version of workflows would never have worked for what we were trying to do. And so we kind of built it half-assily in the beginning and then we kept going on it because we were too far in. Much like many developer teams get into, you're too far in and you can't go back. And then you realize that you have to just go back. And then, so that was like a big time sink for us, but I'm glad we did. Cause now we have a lot of power that we can then use going forward. So there's been other times it's just taken us the time to find the right things. So like initially we knew we wanted to work with AISCK, which is the library from Vercel as a, as a model routing layer.

38:49But the first way that we did it was kind of, we were sort of republishing a bunch of their packages with sort of a little bit of a wrapper. The import structure was like too long. And then we were finally like, why don't we just expose their API? So we sort of ripped out our whole model routing layer that had been built on top of their model routing layer. Just said, like, use their model routing layer. While we were trying to figure out how to, like, with it, but it wasn't the right API. You know, that and sort of like this new workflow engine that we launched in, I guess, March. Like, those were the last two sort of, like, big re-architectures.

39:21Like, that one was in January. And then the workflows, like, we called it vNext at the time, was in March. Those are like the two big re-architectures that we've had to do over the, you know, nine months we've been working on this. Nine months. That just got me thinking about how different the world was nine months ago. Yeah. AI time is sort of like two to three or four times as fast as normal time. Yeah. I think. It's just something different is happening in this part of the region. It's like singularity, black hole, something. Time dilation. I don't know what it is. So you're focused on building these like primitives, right?

39:52And I'm sure that things have changed dramatically in the nine months that you've been doing this. Have you found that the choices that you've made have been pretty resilient amongst all of the changes in AI? And do you see that trend kind of continuing? Yeah, I would say the first generation of AI frameworks like LaneChain, what they kind of tried to do is they tried to do a lot of this prompt engineering stuff. Because at that time, the models weren't good enough. And then something flipped around June or July, about a year ago, where the models were good enough. And then the right thing to do was kind of build these like workflow based abstractions on top of that.

40:29You know, by when we started working on the problem, we landed on that the correct abstraction, which is build the agent abstraction on top of the models, the workflow abstraction on top of the models. The other big thing that has stayed the same over the last nine months or 12 months, but was kind of different than before, is that when you're building on top of APIs, you don't need to do any of this sort of matrix multiplication kind of machine learning stuff that a lot of the Python people are doing, which means that there's not a default language that you should be using. You can use any language, and why not use a language that you can build your whole app in?

41:01Why not use TypeScript? Even though things are moving quite quickly, there was a distinct inflection point. And then we started building on the right side of that. We ended up with the right architecture. Yeah, it feels like we got a little head start with then like the rest of the world by starting early. And like every week, we have like this channel in our Slack called Kindergarten, where we like share all these blogs and white papers that we should read. And I'll be named that because it was like, hey, we're all going back to kindergarten. Like we all got to learn this stuff. Like, I think one of the things that to stay up to date in the AI world, you have to stay humble.

41:37You have to like not think that you've learned at all because nobody's learned at all. And there's new stuff coming out every week. I think it's hard to not be glued to kind of Twitter X in that world because you're always just like scanning for what the new model releases. And, you know, there's just a certain set of podcasts that you like obsessively like listen to and whatever. But like you stay humble and you stay in kindergarten to know. Yeah. Yeah. And then when you look at the articles and stuff that are coming, and these are research papers and stuff, a lot of them are coming out before Sam Altman tweets about them.

42:08And that's kind of like the signal that everyone else is going to do, like come into the thing. For example, agent memory, we've been focusing on that since we started the company, essentially, because we were reading a bunch of white papers on that and it wasn't mainstream yet. And then what did OpenAI do? Chud, GPT has memory now, which now made people realize that this is a primitive that exists. So I think we were very fortunate that we had a little head start and also are in the mindset of always learning. Because we haven't been off yet on our predictions on what's going to happen or what thing is based on what we've read and stuff.

42:44Nice. Yeah, that's very fortunate. And I love the idea of just that kindergarten channel. I want to adopt that for sure. It's just like constantly take me back to school. We know nothing. That's one thing you mentioned the AI engineer world's fair, like talking to people there. It was so great because like, I realized that everybody who's like in the space right now, we're all just learning together and there's so much to glean and like so many ideas. Like I just left there brimming with, with ideas. And what an exciting time to have a great dev tool in that space. The final piece I wanted to touch on was your developer experience.

43:17I think that that's one of the major selling points of Mastra just in my playing with it day to day. And it's so easy to get going from the command line utility that like walks you through getting set up, suggests a model to get going with, and then like boom pops you into a browser based environment where you can see the agent, run the agency, what tools it's calling, see all of the workflows and all of that. How did that come about? Was that like through your experience building Gatsby and other dev tools that you just kind of knew how that was going to go or a lot of trial and error again? There's a funny story behind it.

43:53And Sam can add some color there. So the first version of Monstra, let's say Monstra negative one. And at the time, this is like October, right? So at the time, AI products were very much GUI driven. So the nature of having a playground for us, like Monstra was not code first. It was actually like JSON structure first. And you would actually create all the agents and stuff through UI and all that. And so we showed it to a bunch of people we respected. And they just said like, oh, that's cool. And if you're a DevTools builder, anyone says that's cool actually means go F yourself. Really like, you know, they're just too polite to tell you that you're doing something stupid.

44:31That's cool. Like, and I just knew I was, I remember Sam and I were together and Shane, I was literally distraught. all these people I respected just said that's cool literally means like they should tell me to jump off a bridge right so I'm just all distraught we're at the bar I'm like guys I think we have to rebuild everything and we did which was the right thing so it was good but the playground was like when we changed everything to code code first we were just thinking like hey this playground was pretty cool though so we should probably keep that in some way and then like Sam you're the one you and Shane kind of have the brilliance behind that so you guys you should take over the story.

45:05I think Shane was kind of the one that really pushed for it the most, but to get the playground to work, we had to do a bunch of like really kind of crazy bundling stuff. So what you do in the playground, right? You, you, the playground is kind of a visual chat tool. So like if, you know, monster is a backend tool, you're building the backend of the agents, but then you kind of want to see how it works and see how it responds to different commands. And so it gives you a chat. It lets you see the traces. So it kind of gives you like a tool playground where you can see inputs and outputs. It gives you a screen where workflow visualization.

45:36So if your steps have, you have branching or if you're chaining them together, it'll just show you them. And then like you can put in a sample input and it'll kind of run them. It's like that inner loop of development, right? You're like, oh, this thing is in the right direction, but I just need to tweak this bit. I need to tweak that bit. I need to add a step here. I need to tweak the prompt there. You know, I need to think about this edge case. And we just do so much of our development loop is that inner development cycle. And we just kind of were like, gosh, like, we were sort of like porting our old playground.

46:04Because initially, like, right in Monster Negative 1 that we had in, like, October, November of last year, the UI was driving it. In this case, like, it's a read-only UI. You know, you're not changing the code. You go to, you know, VS Code or Cursor or whatever to change the code. But, like, you can see how the code that you've written, what it does, and you give it this input and, like, what the output is. and then we sort of switched it over that one we're like you know gosh this feels like it's kind of cool and it like it really feels like this is something kind of unique and different but once we kind of like committed to it you know it was just like 10 000 little things is the ui details where it like highlights the step in green when the step is running and then it moves on to the next step and has the next step in green and you can see the input and output at each step and like what do you do when you have like a really big diagram and how do you represent like nested workflows and, you know, like, how do you like arrange the agent chat and like, how do you represent the tool call in the chat?

46:58So it like displays visually differently. And, you know, we just sort of went through the iteration. And then, like I said, like a lot of bundling stuff we had to do, because like you're exporting your workflows and your agents right in a way that like you could use it in other bits of the code, but like also you have this kind of dev server that's kind of using them and pulling them into like a different type of environment. So there was some like narrowly stuff there that like word from the team had to work on to get it right there. Yeah, that's a good point to touch on. Like the skills that we learned at Gatsby to pull something like this off.

47:29One, we became bundle, we call them bundle rooskies. If you think about it, there are not many few people left in the earth that can bundle JavaScript proficiently as we used to do back in the day. And so we used to do it back in the day. That was like our job. And we had to do it. How many people had to do it? So there are very few people on the planet who can bundle JavaScript. So that's us. Second is we used to create Gatsby's GraphQL server from all the BS framework code. So we're really good at making servers too. And that's kind of how the Playground comes about. We transform the framework into a Monstrous server, which becomes the REST endpoint for the Playground, which has some gnarly bundling stuff with bundling server and JavaScript.

48:10But then it allows us to do cool user experiences like that. It also allows you to run Monstra as a server, or you could just run it as like an import in your current application. So like being good at bundling allowed us to have two different modes of usage. Nice. I didn't realize that. And I agree. I've dabbled in bundling and just get frustrated every moment of it. I think that these tools are very important just from like also a learning perspective, Like being able to like put things together and then see it visually like that, like don't discount that. So I think that you're definitely on the right track of like just helping through those tools to easily like help people like myself who are new to all of this understand what we're actually building and how it all works together.

48:56Kudos on that. I wanted to ask about like roadmap or vision for the future. Things are changing constantly. And like, is it a lot of like reactive based on like what, you know, how things are changing in the world with, with AI, or do you have like plans for like new features coming to Mastra? Do you want to speak to that? It's sort of a variety of a lot of different things. So the core primitives are what our roadmap looks like right now. We know what our core primitives are. And like some proportion of our roadmap is like interoperability for our core primitives, right? We want, and like some of it is like, we're working on some new eval stuff right now where we have the concept of this score and like a score, which we think is much better than these like LLMS judge and evaluator kind of tools.

49:40But like, we've built that into the playground, right? And then we're, you know, the next day after that, we have a cloud product and we're building it into Monster Cloud as well. And so it's sort of like, hey, you have these changes in the core that kind of like, you know, ripple out across things. Like we're working, for example, right now on templates. and so like you know just walking through kind of like hey i want to build a docs chat bot here's a starter code repo the template code repo for you you know to install like a pdf upload these like really basic use cases and it's like well okay cool we built this like library now we're you know going to get the cli installed now we're going to run like a hackathon to get in the community to get the community to build some templates we're gonna like have get the templates working in playground and then the templates have to like go in you know in cloud so some of it is like that where you're like, I got to trace this feature across a number of different platforms, almost think about roadmap.

50:30Some of it is coming from sort of like externally to us. So like in the space and in general, it's kind of two of the biggest problems in like agent building are one problem we call like synthetic evals, which is how do you take this like raw traces that your agents and workflows are emitting and sort of turn that into like a set of tests, a set of statistical tests, these evals of for your agents, what's the loop that helps you do that? And then Then another one is like agent learning, which is how do you use those like, again, that like raw exhaust of kind of like traces and actually improve your agent performance.

51:03And given what it did in the past, you know, kind of learn from that data. And like those are sort of like two broad problems that we were just sort of like widely recognized in the space of problems. So we're like we're working on various, you know, approaches to there. But like that's coming from the space as a whole. We're thinking about this and there's like other folks thinking about this as well. and like somebody's going to figure it out over the next few months. And we'll either be one of the first ones to do it or like once it's accepted what the right way of doing it is, then it'll be really easy to build that into master like one of the kind of the two things.

51:35Broadly speaking, the other things that we're thinking about are sort of like agent authentication, agent sort of like security, you know, guardrails for like inputs and outputs. I was talking about our multi-agent primitive or agent network. We're working on an event-based execution system where you can custom-defined events and have them bubble up. Multimodal support, whether it's images and voice, we have some stuff there, but there's going deeper into voice, thinking about video. The frontier of what the models are doing is shifting so quickly. A lot of other bits are the integrations with other kind of, there are agentic front ends, like CopilotKit and Assistant UI, for example.

52:20And like, we have a lot of work that's like, work really well with them. And then a lot of the other sort of providers in the space that have like sandboxes for code execution and like browser use tools, web scraping tools, good integrations there. So I think like a lot of it is driven by the ecosystem. A lot of it is driven by the natural development of our own primitives. We got a lot of work to do. That's for sure. I mean, time moves fast, but I think like, yeah, there's the space is evolving so rapidly and it's like, yeah. Yeah, it's constantly changing, but you seem to have a good instinct for it now.

52:55So keep that up, keep it going. Is there anything that I didn't ask you about that you wanted to mention about Mastra? I think one of the lessons that everybody should have is that kind of something I'll be alluded to earlier. Whenever a new team member on boards, like it takes like a month or two for them to kind of learn all the agentic kind of concepts and how everything all works together. And we've now watched, you know, probably a dozen folks like go through this transition. And it's just a little bit of time and immersion. And I think like, you know, if you're new to the space, or if you don't feel like you don't know a lot, and a lot of what we said might be these like terms, you're like, what are they even talking about?

53:29Right? it don't be distracted or discouraged we just we've watched so many people go through this it's just try to immerse yourself in it however you can build something yourself and that's the best way of doing it you know hopefully you know just use monster like find some toy storybook type project where you can like build it that would be like cool and fun and interesting and like use that as a vehicle to learn the concepts you know read the book free copy at like monster.ai forward slash book just like immerse yourself in it for like a month or two months and you know you'll sort of go from like, I don't know what this stuff is.

53:59And what are these people even talking about to like, wow, this is like pretty cool. And I I'm getting the hang of this. Right. I think that's our biggest like advice or takeaway for, for folks. Yeah. We're really dedicated to the education part of this. So like every Thursday, there's a master workshop. If you want to learn something new, even things that we learn tomorrow, we're doing a lesson on background agents with cursor and codex and stuff. Sometimes it's not even about master. We just like teaching things. And on Mondays, I remember back in the day when I was learning JavaScript, JS Jabber was so cool.

54:30I used to listen to that stuff. So we also do like a live stream agents hour where we talk crap with each other about just the things. And that also gets people. We've had so many new people to the community come through there because you're now in the days where you're vibe coding and you're vibe listening to people, vibe code themselves. And it's like this whole world that's coming. And so I would recommend if anyone's trying to get into this, just come to those things and we're happy to have you. Nice. Yeah, I will try and be at the next one. That sounds great. How can folks reach out to you or learn more about Mastra?

55:06We're Mastra underscore AI on Twitter X. I'm Calc Sam, that's C-A-L-C-S-A-M on Twitter. I would say like you can also like NPM create Mastra at latest and like, you know, just kind of walk through that. And then NPM run dev and spin up the dev server and kind of just play around with it, you know, for yourself as well. Yeah, you can find me on X, Avi Iyer. And then, yeah, just come join our Discord. We interact with the community, so come say what up. Yeah, please do. It's an exciting time to be a developer. I think it's an exciting time to be a building. And there's like so many things that we can build now that we couldn't have built like even a year ago or two years ago or like two years ago.

55:46And that's one of the things that just gets me excited is like seeing all the cool things that like everybody is building. All the cool agents people are building these days and all the cool AI engineering people are doing. It's just, it feels like a world where like there's magic, you know, there's kind of like that sort of magical moment when I was learning how to program where I'm like, whoa, you know, I'm doing this right. And like, but I, I didn't have that kind of experience again until, until AI. And I think like we can all have that kind of experience again. Yeah. I'll definitely second that.

56:13That feeling came back to me learning all of this. It was very exciting and still is. So definitely I'll second your suggestion on that. Thank you for sharing all this knowledge. I love the DX of this tool. I love like all of the ideas that you have, like with, you know, the doc server. I really want to check out this teacher MCP. Sounds really awesome. But most importantly, I'm just like really excited about your excitement for teaching this and like being open about that. And so I think that everyone can learn a lot from you guys and I look forward to learning more as we go. So thank you so much for joining Software Engineering Daily.

56:45Abhi, Sam, thank you so much. Have a great day. Thank you. Thank you, Nick.