Building toward a bright post-AGI future with Eric Steinberger from Magic.dev

0:05So welcome to NoPriors. Today we're talking with Eric Steinberger, the co-founder and CEO of Magic. They're developing a software engineer co-pilot that will act more like a colleague than a tool. And Eric has a really fascinating background between work at Meta on different types of games, running Climate Science, which was a nonprofit focused on the climate world, and now, of course, developing an incredibly interesting AI model and system. So welcome to NoPriars today, Eric. Thank you so much for having me. It's great. So you have a super eclectic background. Could you tell us a little bit more about what you worked on in the early days, how that evolved into working on AI and sort of the path you've taken?

0:43Yeah. Yeah. Thank you. So I guess when I was 14, I just had my midlife crisis and thought I had to do something important with my life and spent a year trying to look at everything. It was pretty stupid. And basically I'd look at things like string theory and all the things a 14-year-old would look at. I mean, like, okay, what can I spend my life on? And eventually my mom got me a book on AI and I didn't read it. I'm sorry, but it was like the idea was sufficient. So I was like, OK, this could do anything. And so you should just do that. And then it does everything. Then it seemed plausible that you'd need to do reinforcement learning.

1:18So I didn't know how to code at the time. Then I learned to code over a couple of years. This was sort of in high school times. And then it seemed plausible that you'd need to do reinforcement learning, because otherwise you'd not be unbounded. So I sort of just started working on RL, played around with things for a bit. And eventually I reached out to someone at DeepMind to basically do it. So I was pitching this multi-page email. I was like, could you do a mini PhD thing where I'm a complete newbie, but if you can bash me, just please bash me every two weeks and tell me how to be a good researcher.

1:50And so eventually I got reasonable and then did some actual research work and worked with a few other people, including Noam Brown, who had meta on developing new RL algorithms to be more sample efficient and just generally better and faster or whatever was the goal at the time to solve whatever environments we were interested in at the time. So yeah, that's how I got into it. I have no background in language models when we started Magic at all. It just seemed, I just was totally not on my radar. I was like, oh, wait a second. Like if you take this and this and put it together, like maybe this works.

2:26And so then I sort of, it felt like this huge relief of uncertainty relief of like where AGI would come from because you just put those two things together and then it will work, was the sort of hope. But yeah, my original background is in RL and trying to come up with algorithms that sort of, yeah, just like have better structures to be more sample efficient and faster or better conversions. And a lot of the emphasis on magic is sort of twofold. On the one hand, you're doing a large-scale custom model, specifically in part focused on code, and then you're also building sort of at the product suite that can really help address coding and working on the software development side, how did you decide to start Magic?

3:07And why focus on that versus other aspects of AI? It sort of came from a place of working backwards from AGL. If your end goal is to have a system that can do everything, you can reduce that to building a system that can build that system. And so that minimal system is a system that writes code and comes up with ideas and can validate those by writing code and running experiments, which is still like in the same order of complexity as the full thing but at least we don't have to train Zorro and we don't have to think about 10 billion other use cases that everyone building general domain products has to think about we only have to think about code so it's a lot simpler in all aspects except compute and slightly simpler on the aspect and slightly cheaper on the aspect of compute I think it's not a lot cheaper I probably overestimated how much cheaper it would get on the compute site at the beginning but the other things are simpler I think And you've taken a slightly different architectural approach, right, than what a lot of people are doing in terms of just going with a traditional transformer model.

4:05Is there anything you could talk about there? And, you know, you also had a very early emphasis on long context window. I think you were the first model that was publicly announced at like 5 million tokens. And I think that was like a year ago or something, right? Yeah, that was a year ago. So I was just curious how you chose a specific architecture that you decided to focus on and how you decided to focus on context windows before most people thought it was a thing. I think you've been very pioneering on a lot of these areas. Thank you. Yeah, it seems important for models to have the ability to learn from long histories of their own and their collaborators' actions, as well as take into account a large amount of fast-changing data.

4:46And so if you imagine having 10 ,000 employees or everyone on earth having their own model and wanting to feed in all their data, you can now fine-tune everyone's model, maybe do some Laura tricks. But in practice, context just works better. And that's, like in context learning, is the magical part that came out of Transformers. Like this is what makes them great. I think of that as some sort of, as an online optimizer, in a sense that instead of compressing a set of data, you're trying to learn an optimizer. So the perspective we take on models is instead of, this is one of my colleagues put it this way, I find it very fitting, so I'm stealing his quote here, is instead of bringing the data to the compute, we're bringing the compute to the data.

5:33So you have a set of stuff and our model acts on that stuff rather than having like a giant model that you have to sort of work around. The whole system is designed for this. So yeah, like a year ago, we announced 5 million. And by the time this is out, we may have announced a larger number. The main reason being that you would want to deploy these things for very long horizon trajectories and you want them to spend a lot of time thinking and you want the model to remember all of that. And you can't really do that by fine tuning because you'd have to fine tune every like whatever many thousand tokens your context spent always long.

6:13I don't know if people would think of this as a trick, Eric, but like, you know, can you give us some intuition for why, you know, quality of output would be better than a retrieval based system? Yeah, I mean, so you can comment this from two perspectives. I can explain it mechanistically and I can, the other cheap way is to just point that Richard Sutton's bitter lesson. Retrieval selects a subset of data for one completion. Our model sees all the data all the time. Clearly, a subset of data for the whole completion is a subset of all the data all the time. So if retrieval was optimal, our system could learn it.

6:46And it just turns out that it's not optimal. That's the sort of mechanistic, I guess, explanation or logical explanation as to why long context will be better. You could make arguments around the quality of long context if it wasn't sufficiently high quality, maybe having a short context window and pulling in some data. So you obviously have to evaluate this. But in principle, yeah, on the assumption of Richard Sutton's better lesson, you would want the thing that can learn your heuristic rather than the heuristic. The other area of expansive exploration right now amongst researchers for better AI cogeneration tends to be test time search.

7:23So, you know, more compute at inference time. You know, speaking of a mutual friend in Noam, like, how do you think about this? Well, so you can think of model performance as some function of training compute times some function of inference time compute. Now, those are specific functions that are just scaling law things that you can model, but the general way to think about it is some function and some function. And then you would want to estimate how much inference you're going to do and what your total budget is, and then you would want to create the optimal trade-off in your allocation of money.

7:55You also want to consider the distribution of outputs. There'll be users who will want to spend less money and users will want to spend more money. And this is likely going to follow some sort of very sort of spiky distribution where there'll be like four users spending a million dollars and, you know, four billion users spending 10 cents and a curve in between. It seems strictly beneficial to be able to provide that choice. So instead of putting all the compute into training and having that$1 million inference performance be purely from the training compute, which is just hilariously inefficient, you can allow the user to choose their thing.

8:30or rather you can just deploy multiple things. The reason I can talk about this in a while is because everyone gets this. But basically you clearly want to be able to regulate the amount of compute you save this time. Now, it turns out this is actually not trivial. Doing this is hard. Finding the right algorithms to do it. That being said, people have done it in RL for a decade. And so to those, I don't want to name people. I know him obviously is one of them, but there are others at other labs as well. Like there's a set of people to whom this is like the opposite of a surprise, but it's still not trivial because it's the general domain.

9:05There is no game. Yeah, the analogy I've heard is sometimes when you're asked a question, you kind of pause and think about it. And that's almost like your inference time compute. So you're basically investing some sort of resources to actually consider the problem at hand versus just spot react to it. Correct. And there are things you have to learn during training. like if you are asked to write a piece of code in a and you've never learned coding you can spend the inference time compute you have to spend is ridiculous uh like like you're gonna have to at inference time learn programming which which maybe i actually think this is possible but uh i i it is also crazy um and um the this is like clearly this is shared among like everyone who like posts a query so so it's stupid not to make this into training even the best mathematicians in the world for the frontier of mathematics require a long time to solve the problem.

9:54So I would love to have a Terence Tao in my computer, but I would then still need to run Terence Tao for a year of human thinking time. And Terence Tao will not just token by token spit out a proof for Riemann or whatever. And so I think to achieve things like that through pure training compute deployments, inference time compute work would need to drastically fail simply because it is a shortcut. But you can also train this model for a quadrillion dollars. Maybe you can't actually because there's no data. But say you could. Well, what if I just need a billion? You get the idea. So I think it's this fundamental trade-off that you want to be able to make into.

10:37The humans can do this exactly as you say. This applies to all parts of the workflow that you ask your model to do. And I guess if the goal of the company is eventually to build AGI, how does that impact your choices from a design perspective relative to some of these trade-offs? Or is it more, we're going to iterate until we have a system that's very good at writing code and then it bootstraps its own next version? Or how do you think about your roadmap relative to AGI itself? Yeah, that's a great question. So I remember one of our first conversations. We were talking about sort of the step function relevance of safety risks, let's say.

11:08Right? Where there's a lot of stuff people are panicking about that really doesn't matter in the short term. for the grand scheme of society. And some people will get pissed at me saying this, but it's just what I actually believe. I just don't think this is like, the current complaints are similar to what we have seen in all other technologies and totally resolvable. But then there comes the evolutionary one. And I was like, okay, shit, humans succeeded in the world because we're smarter than chimps and we're smarter than bunnies. And bunnies do not rule the world. And that would be cute probably, but also it probably wouldn't be as nice for us.

11:44And here we are turning ourselves into bunnies and apes and creating this thing that we're all thinking is going to be way smarter than we are. This is insane. And so the reason I think the sort of recursive approach that you're mentioning here and in thing at, and then obviously you know this is what we're founded to do, is exactly that I think the only way to sort of reasonably approach this is to iteratively ask your model to solve alignment and safety at that stage. Surely you can also ask it to solve your product-level problems, but that's nice, but that's not the fundamental objective. The fundamental objective is to iterate towards AGI with a safety boundary, and there's just no knob like this.

12:28There is just no knob like this in the human world. You can say, oh, I'm going to spend X percent of my resources on this, but that doesn't indicate an outcome. You can actually control it. But if it's compute, you can somewhat do it. So I think it's just the most promising path we have, both on the progress towards better models. Because if you just have 100 ,000 brilliant researchers in your computer, that is better than having 100, especially if you can connect all their brains. And then if you can say, OK, we think this is safe to do. Let's go do it. And then you have the next thing. And then you're like, OK, maybe we should use that thing to do some more safety research.

13:08you can bootstrap. And so just very pragmatically, even if there was goodwill, I just don't know that we have the mechanisms to prioritize safety without automation. So yeah, that's the primary goal. Now, the good thing is, as we pursue this, if we're successful, if we're not successful, then I'm deeply sorry to all of our investors, including you. But if we do succeed, this will, especially if we are first or early to succeed, drop a beautifully profitable apple off a tree that happens to automate a good chunk of what we call work today, which is another thing that I think is a responsibility of our company to do it, is to say, look, if this type of thing works, it's not really an assistant.

13:50It's just like you just put it there and you talk to it and it's like a colleague. And that's great. I think the economy does the same thing, same outputs with less input or even more output with less input. this is fantastic for the world if we just do it well. Like, like, like capitalism and competition can be great. And like progress, the entire history of progress comes from this. Like we'd all be farmers if we weren't in favor of automation. It's, it's like, it's scary. I get it. And, and, you know, if you don't think about it all, like for the whole way, like it's very scary, especially if you're affected that you're not in one of the lucky seats I'm in, for example, but, but, but ultimately it's good.

14:25And so, so, so yeah, I just, I just basically like when we started, tried my best to think through this and and this seemed like the most productive path um forward and and again i think it's just like we're lucky into being a position where we both get to pursue an incredibly valuable uh product that that's a new generation of thing um that just doesn't really exist yet it's it just there is no like thing that does your work for you um i think chat gpt was one of these you know monumental moments of a new type of thing uh but you just the first AI assistant. This is mind-blowing to all of humanity.

14:58And this will happen once more, and then maybe once more when you can just query the solution to Riemann. And then that's probably it, at least in this domain. And so I hope we can be a part of the second one, maybe the third one, on the product side. But we chose the domain we chose because of what I said previously. Let's talk about those two pieces separately. You said you can be slightly cheaper than a more generic AGI effort? Like the last conversation we really had one-on-one was about how much compute you might need, right? Yeah, I was wrong. So like, I guess I contrast this effort to like a more generic effort because clearly the large labs, they're using a great deal of human-generated coding data.

15:39They care about the use case. So like, you know, what makes the efforts different? I think a lot of it is direct competition. Now, I think that like, there are totally things I could say here that sound like entirely believable as like major differentiators, but I think fundamentally are, and I'm going to talk about them, but I just also want to highlight that I think at the core, the wider world has understood that code is very helpful and there are ways to deploy compute to improve coding performance. And so therefore, because a lot of compute is deployed in this domain, the need for compute is large.

16:18That said, also in parallel with the release of this, we're going to be announcing what is going to be one of the largest clusters to ever be built. So we are correcting for that. I was wrong when we initially talked. You just definitely need a lot of compute. Now, there's still, I think, a notion of enough, but it might be a lot. It's interesting, right? Because there is enough compute for search, for Google search. which I don't think if you took Gemini 1.5 Pro and you put it into Google, and you did all the fine tuning properly, and then you swap it with Gemini 4, I don't think anyone would notice.

16:59Unless you go like Proof Reamon, right? But it's good. Like 99 % of users' use cases will never notice the difference. So this does the job. It's like AI overviews works now. And I think this is going to similarly be the case for each thing. Like there's going to be a model that can prove Riemann. You can make it a hundred times smarter. You have your proof, right? So for each thing, there's going to be improvements after a certain amount of compute. And I think I underestimated that number. And I underestimated how much others would focus on code. So anyway, just to clear up, I think our 101, you were right.

17:33So I guess there's the model side of it. And then there's sort of the productization of that model or the ways people access or interact with that. Is there anything you can share at this point regarding those types of things? as we have built the UX, each iteration of the next UX internally, we thought about launching it. We were at this interesting stage of it feels like an uncanny valley where, like, well, you know, clearly you can see signs of life. First of all, I mean, completion is a trivial one, right? We decided not to launch completions because it's just obviously going to get killed by the next thing.

18:07And then we're like, okay, it's going to take us a few months to get a prototype of the next thing. We got a prototype of the next thing. And then when I get looking at this, it was like, you can see signs of life, but you guys tried it when you decided to invest. So it's like, would you be using this to write your idea? No, not yet. But we can train the next model. And then that model can do it. But then that model can also do all these other things that would go into final shape. So you just enter this stupid recursive loop. Until the point we got to the point where we were like, OK, what's the final UX?

18:38Let's just make sure this never happens again. And so we're trying to meet the bar of that UX now, which I hope we will sooner rather than later. The closer you get to it, the sort of dumber it feels to launch the thing before it, because you're going to replace it in a few months. So if it's good enough for that, how hard can it be to add these last level few things? So I do think there is a difference between, like you can launch an extremely capable assistant before you launch full automation. That's fine. But launching a sort of mediocrely capable assistant, like we might do it. we have a deadline internally by which we don't have the like normally you know nobody has this right now like there is no amazing can do everything it just like feels like a true genius colleague on your team um but and if we don't hit it by that time we'll launch the other thing but um i would prefer hitting it it's just the honest you know the reality is um things are hard uh things are something some projects are uh going great and some are delayed and some are just, you know, there are a hundred fires all the time.

19:41This is just how every hard engineering project goes. Everyone who's listening to this and has ever worked on an engineering project was like, this is how it goes. I think we, you know, there are a lot of things we learned along the way and there's nothing we're stuck on. It's just things are, you know, oh, there's this thing we didn't think about. Okay, let's fix it. And so I feel very optimistic. What makes a assistant like more mediocre, versus amazing? Is it reliability? Okay. Like, do you just trust it? My engineers, when I have one of them write a piece of code and another one review it, like, looking at it, why would I look at it?

20:21It went through these two guys, you know? So, like, what's the point? So is the eval bar, like, I'm not going to do code review? So that's, like, the fully automation thing, right? And then you can launch something where you're like, I'm going to do code review, but it's not frustrating. if you have to do code review and it's like really taxing and you have to fix half of the problems like half of the PRs or whatever so I think the bar for this product is just high it's not that like we are like so ambitious and you know I think that too but I just genuinely think that there is a gigantic market that gets unlocked in a step function moment where users decide that they're no longer going to use VS code to write code and send it to their colleagues they're going to use magic or whoever ends up doing this well first to write their code for them and then briefly look at it and correct every now and then what has been done.

21:15And then eventually not, right? But that is a step function moment, I think. It's not like you're not going to use this for like 5 % of your tasks. You're going to use this for 90 % of your tasks or zero. You don't believe that this is something you can cut by use case, right? It will be trustworthy on some set of things. No, I think if you can, the leap to doing all use cases is small. Like you can build a UI builder and then it's like a normal UI builder or you can build a true great UI builder driven by AI with some added features for that vertical. But then you can do the same thing for all the other verticals, just add the features, you know?

21:51And so maybe your product team needs to do one by one. That's feasible. Like that's totally imaginable. And maybe your go-to-market needs to be one by one. But the model, I don't think so. I guess to your point on having a very high trust team, how did you think about the team that you assemble? So now this is really easy because we've raised an unbelievable amount of money from great people and we've got things to show that even risk averse people who don't think from first principles can understand that this makes sense or who need that initial like seed of trust. but I did find it very hard to recruit when we got started, to be honest, because when Dario Amode goes out and starts a company, this is trivial.

22:33Hi, I made GBP3. And when there are others like this, it's easy to establish that trust in the outcome. So the strategy we adopted at the start was to hire actually kind of people that like you do once you're referencing um people who might be uh we have one guy who was just like depressed at amazon um working on like alexei and he just hated it and he was like so great like he just knew everything every single paper i i was he was like he is rag um like i would i would go like okay like you know let's talk about this and he would just know everything and let's talk about this and he would know everything and let's talk in engineering in college and then just got into him.

23:24He's one of the most brilliant people I get to work with. And so we hired him and he did a bunch of stuff. He invented a new sharding dimension for a total training. And this is just like a random, you have to really pay attention to identify these people. But then there is an amount of drive and loyalty that you just don't get if you poach the obvious guy, right? and so that's the type of person we have and we have a decent number of them now we've gotten really good at identifying them I would like to have roughly four times as many as we could but that's it again I think with the series of announcements that's going out in the batch that this podcast is going live that again will get easier and better but I love our culture it's just everyone cares about the mission deeply.

24:19What I said earlier about why we do what we do, the safety-bounded AGI recursion, it just takes a lot of brainpower and understanding of the world to comprehend that this is the right thing to do, and or a lot of trust to trust an organization with doing that in the first place. And everyone cares deeply about this, but we don't do it for years like your marketing sign or whatever. It's just not who we are. At the same time, everyone is deeply productive. One of our core engineers who writes the inference engineer, I guess one of the two people writing the inference engine and some of the kernels.

25:01When we need join, I was like, why do you want to join? What do you want to do? This was before we raised this giant stack of cash that's getting announced now. It was like the tiny amount still compared to other labs, or I guess compared to any lab by a large margin. And he was just like, well, you know, he saw this as an opportunity to, he wanted to be one of the best, or he said he wanted to try to be the best kernel engineer outside of NVIDIA. And he didn't say this in an arrogant way. He just said, like, this is a ton of work. I've done this for the last few years. And I just need to be in an environment where I'm sufficiently challenged to do this.

25:32And he's been grinding every day. and just having like that level of ambition, but not with like the typical San Francisco, you know, but it's like quiet with humility, drive, you just come into the office every day. You're not like working until 3 a.m. because you prove that you look like this is our culture. You do it sometimes because you're just so obsessed and you try to be healthy. You do work all the time because you just care so much, but we're not buying the IP by poaching someone from a lab who tells us how they train GPT. Never done this. We'll not do it. We just do our thing. We have our plan.

26:09And we have brilliant people who I'm delighted have trusted us and spent their energy and their best years on our company. What does AGI look like? I think it just talked to it and it does everything. And it asks you questions. That's important. And do you think the existence of that, I guess one could argue it can increase GDP, but it may also decrease a lot of sort of human-driven activities. How do you think about the eventual implications or impact of AGI? This is going to take longer. I'm going to try to compress it, but it's actually quite complicated. One of the biggest problems with this question is that everyone tries to simplify it by picking one side of the argument.