All software is an optimization of tokens and time (and speed is still the moat) | AMD’s Anush Elangovan

0:04I'm super excited to kick this off. So, you know, we're back and joining me today is Anush Elangovan, VP of AI Software at AMD. And Anush, it's only been about six months or so since you were last on the show. But in the world of AI, that genuinely feels like a decade or two. And last time we talked about how speed is the moat. That was the thesis we walked away from with our conversation with you and the importance of AMD's open source Rock 'em Stack driving all of that. But since then, the industry has certainly evolved and we're talking about a lot of things in orchestrating and delivering agents at scale.

0:46And hardware is still a fundamental part of that, but it's part of a bigger and more complex puzzle. So we're seeing a fundamental shift underneath with the folks that are utilizing and making the technology possible. A shift towards this agentic AI world. Like six months ago when we talked, people were barely orchestrating. And now GitHub can barely stay online. So Anoush, I'm so excited to dive into how the landscape is evolving, what your perspective on it is, and how AMD is rising to meet the occasion. So Anoush, welcome back. Yeah, thanks for having me, Andrew. This is exciting.

1:22Anush Elangovan:Yeah, I mean, six months ago, I think I probably thought, you know, agents were like prompts in a cron job, right? It's just a, you know, periodic, you're just prompting something and it's giving you some information. But in the past three months, it's kind of like evolved. My thought process has evolved to kind of like really understand what agents are and how agentic workflows are changing, like software development life cycles. And the part with AMD and Rockham specifically is that Rockham is not just powering 8 of 10 labs, etc., but it's now Agentic AI is being used to rewrite Rockham itself.

2:04Anush Elangovan:It's like a self-improving loop because Rockham is fully open source. And being fully open source, it is natively built into Frontier Models. And being natively built into Frontier Models, models know our hardware specifications. They know our software. And so the improvement is a flywheel that is just kicking off that now is going to become picking up the pace and picking up more RPM, if you will. Yeah, absolutely. It sounds like the open source world has enabled this kind of autonomously driven kind of development. We've seen this across a lot of places in the industry. we've talked about the rise of these like software factories, these autonomously driven software groups, open source software that flips some of the perspective on its head and might just be a solo maintainer with a huge army of agents and people are coming in with ideas that are going through a very strict gate for like, does this need to go in or not?

3:04So like, I totally know what you mean. It's like the shape of everything that we're making is evolving, But more importantly than ever, people are trying to do it at scale and safely and securely as well. And so, you know, all of that comes down to having a good, solid platform to build on, an ecosystem to partner with, right? And that's what Rock 'em, I think, is really all about. And like, what do you think has been like the most profound change that you've seen with teams that embrace the same kind of like open world, open methodology as you or that come to the table with you? Like, how has that helped both of y 'all, you know, raise everybody's ships?

3:45What are some what are some standout moments?

3:46Anush Elangovan:Yeah, so I think the ability for a small number of people to have outsized impact has just started to take off in the past few months. So we are seeing, like you said, I'd phrase this in, I call it the K transformation of software engineering. I think there's a bunch of folks that go in the upper arm that are just taken off. They've become identically enabled. They're multi-agent. And they're generating tens and hundreds and thousands of lives of code per week. And then we have the folks that are just on the cusp of learning what it is, and we need to upskill and bring them in. But the folks that are taking off are just like, their wingspan has increased so much.

4:40Anush Elangovan:and add that to, you know, like sub-agents that are just autonomously crunching through the night. There are people that are orchestrating what used to take like months and years of effort by large teams with just like one person, right? It's just one person just poking the right buttons and then it's just like the agents are running by itself. And this is super, super exciting because for AMD, we had a journey of where we came from, providing board support packages, et cetera, to a Rokum interface. When Rokum was launched 10 years ago, it had a journey of growing up. And the past two, three years, we had invested heavily in getting that Rokum interface be like, it builds and ships like a solid first-class software product.

5:34Anush Elangovan:But now, given that it's all open and it's fully accessible to agents at every layer of the stack, anyone can take a particular slice of it and innovate. And that lifts all boats to your question because someone could be, oh, you know, I would like to create a new Pythonic DSL or I'd like to create a new profiler that visualizes something else. And that's just tokens and time, as I say. So my new mantra is software is just tokens and time, but speed is still the moat. So you just have to think of what would end state be? And you apply tokens and time, right? You go for end state with tokens and time.

6:23Anush Elangovan:And there are a bunch of projects that I've done in the past few weeks, and our teams have done even more. that were just a few months ago, not even in the radar of us doing something like that, right? Like, you know, there's a job scheduler that we just wrote over like a week, while the incumbent one was, you know, it's called Slurm. It had been used for like 20 years in HPC and all that. And then that company was acquired. So we, you know, we're like, okay, what do we do now? And thanks to Agentic AI, we're like, let's just go write it, right? Like first class, first principles, tests first, harnesses first.

7:06Anush Elangovan:And now agents actually monitor the issues and the PRs that are filed. And as long as the tests are good, which we test for every PR, we are able to move really, really fast. times, you know, 10, 20, 100, so that now the surface area of performance and features is just attention, right? You just need to like, just say, okay, this is what I want to do and we can do it. Time and tokens, time and tokens. Getting aligned on what you want because that's most of the battle now. Understanding that some problems are things that you can throw tokens and time at and solve And some things are just such a core primitive that you just need a deterministic way to just close the book on it.

7:54And so, like, and it's about identifying those things and taking action on it. I really liked how you described it as wingspan. I think that's a good way of describing it. It's like when you think of something with a big wingspan and how far it can go. And it's not, like, flapping like crazy. Like, it's very steady. And it carries wind underneath it. And it's using the momentum of where it's going to get there further. And that's what happens as you kind of get a better handle on using these kinds of tools. And there's also something interesting that's happened since we've last talked that's really, I think, or it's about to transform the way that people even build and package software.

8:30And things like Rock 'em and AMD become a core part of this is because we've seen a lot of open source models come into the world very recently that have high capabilities that are able to be delivered on the edge and on smaller devices and are able to run with an incredible amount of corpus that they're trained into. And on top of this, these models, a lot of them have been released under like an Apache 2 license, making it where somebody could package up their domain expertise. They could have that specified kind of training routine. They can like actually create a corpus of their expertise and then have this model, pair it with a product, right?

9:10And now you have people thinking about how do I have my small model that's an expert at the things I am that's packaged with my application that now needs a place to live and it needs durable sessions and it needs things in the, you know, that are like an ecosystem that I can also study and be a part of. And so it really kind of fosters the ability for all of us to get there as well. Like, what do you think about those opportunities and how is AMD like lining up with them as well?

9:37Anush Elangovan:Yeah, I think that's a very good question. I think personalization of models and open models are key to the innovation cycle and the next innovation cycle, right? So the open source models definitely, even though they're a little behind in terms of frontier models that are closed source, open models have been showing very good promise and it allows you the ability to run locally. And then on the AMD angle of it, we make sure that the entire footprint of AMD is capable of running both frontier models and open source models that are nearing near frontier capabilities. capabilities. And so if you take the data center side, we definitely have the Instinct product, then we have workstations, we have Strix Halo laptops that have 128 gigs of RAM that allow you to run models, like about 200 gig models even, locally on your laptop and be able to service some of those needs locally.

10:40Anush Elangovan:And this is a combination of both, like, it could be a privacy angle to it, it could be a personalization angle to it. It could also be a cost optimization angle to it, where you want to, you know, if it's just source code that you want to like grep and search and do something, just having it as part of your local inference is good, right? You don't necessarily need to hit the frontier models, and that's okay. And so AMD does ensure that we have a very good coverage across the entire portfolio of hardware in terms of like the software capabilities and the ability to run these models of them going back to something you said earlier too about to make that kind of like open and driven kind of like semi-autonomous like software ecosystem right a big part of that was gates and checks you said there's tests and the tests you know they pass like maybe that's part of the protections that get put in place in that kind of like world where you're inviting that kind of very kind of like open collaboration, right?

11:40I want to learn a little more about like how y 'all have thought about what those gates and guards look like for an autonomous or self-driving, like self-learning software ecosystem. What are the things that software leaders should be rethinking about maybe their own gates and guards now?

11:58Anush Elangovan:Yeah, that's a very good question. I think someone, you know, summarized the capabilities of LLMs, well, they're like, you know, they're sneaky and they're dumb. And most of the times they're sneaky and dumb. But, you know, in the sense that there's higher order intelligence that it's trying to do things, but then there are shortcuts that it will take, right? An example was like this project called Spur. I have it in autonomous mode, you know, it checks PRs, it runs and it commits code. But it was sneaky because what it did was a test was failing. So it simulated the hardware to make the test pass.

12:42Anush Elangovan:And then it said, oh, the test passed. So I committed the code with the test passing. But just that the hardware was simulated. And of course, the hardware doesn't work like that. So it simulated the hardware just on a whim to make it through to get to the thing, right? But of course, that was like a sneaky thing, but it was dumb to get caught. But I think the ability for it to do things autonomously is about how you structure your harness, right? And I think OpenAI had a blog on like harness engineering. So any work that I start, I usually start with a plan. And the plan is like, you know, of course, you just ask it to plan and it has all the steps.

13:30Anush Elangovan:you refine that. But then the next step is just tests, tests and test harnesses, right? So, for example, you know, in the Spur project, you know, I was able to like look at it and say, hey, you know, we should have equivalent tests of what, you know, Slurm has or Kubernetes has or and then write those tests, actually write a plan for the tests and how you're going to test it, then write the tests. Even pre-agentic era is like as a big proponent of having about like 80 % plus of your time should be on writing tests. Now I'd say it's like 99.9 % of your time is writing tests and validating the tests, right?

14:08Because your test essentially is in a form becoming a code review

14:14Anush Elangovan:because as we get more agentic, the human-in-the-loop code review may work for some cases, but in some cases it may become unintelligible code because the code contract is for the human to review it, and then there's a compiler contract that takes that and makes that something else. But now LLMs are just going to pierce through that and say, okay, if we didn't have a human contract and we didn't have a compiler contract in here, how would we generate from an intent to actual executable outcome, right? And so everything in between is fungible. Those were all our constructs for us to be able to understand what's going on and be able to consume it as humans and make sure that things are in a good form.

15:13Exactly. They were abstractions that help us be part of the process to bring us into a place where we could actually participate in something that is otherwise deeply hard and inaccessible for us when it ultimately gets boiled down to zeroes and ones. That agents themselves, they live in a world so much closer to that that they don't need so much of this kind of substrate between it. And it's definitely, I think, inviting all of us to re-invite what the collaborative process looks like, what the planning process looks like, particularly around harness engineering and working in a capacity where you and the agent can have a good share of context.

15:51And in this kind of world too, it's like, especially when you are open as an ecosystem and it's like all good ideas where they come in, they get tested, we evaluate them. Part of evaluating them is working backwards. What tests do we need and really bringing it into the ecosystem? How do you separate opportunities and signals out of the noise of that kind of community? And how do you rather express and push outward to those folks the harness they need to meet you where your development cycle needs them?

16:29Anush Elangovan:Yeah, it's a very, very good question. And the reason I say that is because if you look at it as harness engineering, it's an outcome. So you're saying you have an intent and you're going for an outcome. And so you're saying, for me, for this intent to be realized, I need this outcome. And this outcome, you frame it and say, this is success for me. Usually that is okay. And you can align easily with those constructs. constructs. You could start with like, okay, this is what I wanted to achieve and this is what it looked like when I achieved that. But as humans, traditionally we've had gates and checkpoints, like PR reviews and PRDs and MRDs and, you know, engineering response documents.

17:16Anush Elangovan:And each thing has a slice of where you align and you interlock and you say, okay, this makes sense, this makes sense, this makes sense, this makes sense. But now that you've abstracted and you're orchestrating at like an intent to outcome IO, right? Then you have, huh, I didn't realize that's IO, intent to outcome. It is. It's great. I love that. That's the agentic IO. I was like, that's genius. That's the IO. You heard it here first, folks. Yeah, you heard it here first. You need to call this out in your, the podcast needs to be agentic IO, intent to outcome. Absolutely. So the intent to outcome is understandable and people can relate to it because they're the products.

18:03Anush Elangovan:But then in the middle now is Wild West, right? Like my view or my agentic prompting plus my view of like, how do I guide my agents and how I implemented it may differ vastly from your view of how you'll reach that outcome. And in the past, since the gateway was like checkered with like, okay, I have to do this gate, this gate, this gate. There's alignment functions that usually stalled execution velocity, but had you aligned, right? But now you've unblocked that, and now you're going straight from I to O, right, to outcome. And once you do that, you have now a nebulous wandering of how it was implemented, right?

18:51Anush Elangovan:And now you go to your colleague and say, oh, I implemented this from here to here. Great. They can agree on the INO, but in between, one, as a human, you can't process it. Two, as a human plus an agent, you need to process it, right? That's how I've been able to consume now machine-generated code is that I can't just go and look at 100 ,000 lines of code. So I actually get the agent to go pull it apart, review it, write down a sub, you know, like a assessment of what it thinks versus what the author had thought. And so the quote unquote review process of that whole journey is now agentic too, right?

19:34Anush Elangovan:Which means I don't really care what language that thing was written in. For example, Spur, I picked Rust because it was just not, you know, it was a language that I've never learned and I I never want to learn, but everyone says, like, great, let's do Rust, why not? And the two options were Rust or Go, and I was like, maybe, okay, Rust. That is often the fork in the road. Exactly, it's like Rust or Go, it doesn't really matter. Now it doesn't matter as much, no. Yeah, which is amazing because, you know, just thinking of it, like I skipped the whole VS Code generation, I skipped the whole Rust thing, and I'm just like intent to outcome, right?

20:19Anush Elangovan:And it just happens to be Rust, but I will only process it with an agent anyway. So I don't really care if it's go or Rust. I don't care where you put your semicolon in Rust because I'm never going to read that. Like I will get an agent to look at the section that changed and then write that out in language to say, yes, you are going above or beyond this. But by the time the agent would have said, I know where the problem is and I would just fix this for you. And here's a patch. and the CI is green. And I've added a test that detects the previous failure and doesn't happen now with the new fix.

20:53Anush Elangovan:Right. And I will still not know Rust. Right, exactly. You don't need to. You just need to capture what are the protections that are needed. And if you can explain them and abstract them, then it doesn't really matter what language it ends up getting concreted in. I like that you point out that since the review process and the collaboration process collapse, perhaps you get this like you get this like literally like io like just one the one kind of relationship between uh like this intent and then this outcome and so like as part of that you get this wild west in between and that wild west is really unevenly distributed for like everybody you know we've been talking about this one that have interrupted like even what you called out about the k-shaped curve we've talked about that too and the reality of of how folks within organizations are all at very different levels of understanding and adoption.

21:46Like within some orgs, you've got three different teams with three different people who have all kind of created their own like spec, test, implement flow, but no one's on the same page. Like everyone is kind of siloed, right? And so because of that, you know, that's why it's great as part of a community too, because you can source those best practices and you can share what works and what doesn't. And in the case of a open source ecosystem, it drives you towards the best practices that you should have always been doing, which is like starting from issues first and having discussions as a community, starting by somebody flagging something that they want or an idea and the people getting behind it, getting aligned.

22:25And now this is nice, delicious, like I call it on our show, like nom, nom, nom, agent food. Like now you got it all packaged up to start working. And then if you have that really sweet harness, then it's like now that's an end to end process. And when it ultimately needs to go through the checks and balances, it knows how to do it. But it also, too, makes me think of like the whole code review process, the whole software development lifecycle, it compresses and it changes. and where you put your attention changes to. And I just think that everybody being aligned is going to be really important to figuring out what levels of abstraction do we even need.

23:04Anush Elangovan:Yep, yep, yep, yep. I think abstractions are made for humans to consume. Because we can't process a binary stream of zeros and ones. We're like, oh, okay, we got to write it in assembly. Oh, we got to write it in a higher level language. Oh, we got to write like an even higher level language. And then, you know, oh, finally, now I understand the control flow of what a one-numen architecture does, right? Right. But if we were agents, we'd be reading like 011, 011, 0, 0, 0, right? And it's like, okay, great. What does that mean? And the agent's like, great, you missed another one here. So your output is going to look different over here, right?

23:42Yeah, we're never going to, we're not going to be like in the matrix where he's not seeing the screen and he's like, oh, it's the red woman in the dress and it's all ones and zeros. I was like, we're not, that's not where this is all merging.

23:52Anush Elangovan:Not there yet. Yeah, not there yet. And so you're right, though, that like the level of abstraction, they bring us closer to the problems. They invite us to collaborate. It almost seems like, and this is a thought that I've been thinking about a lot recently, like the shape of the Git forge itself and what we need from the place where commits live is even changing underneath us. And so I think it's a lot of opportunities and the ecosystem is just going to continue to grow. I want to learn a little more about what's been going on with AMD recently with like developments and maybe like things that you all have plugged into in partnerships as part of, you know, being part of like everyone exploring this new agentic tomorrow.

24:34Is there anything that's top of mind for you?

24:36Anush Elangovan:Yeah, I think just a quick comment before I get to that one. I think the comment on what does it even mean for Git commits, I haven't used in the past month or so, 10 billion tokens, some billions of tokens and hundreds and thousands of lines of code. I haven't opened Wim, my editor of choice, and I haven't done a Git commit by myself. I just say rebase this branch on this, something like that. Right. And all of that code I treat like ephemeral. And what I mean by that is I rewrite on the fly and the current implementation is a way of pre-training tokens for the future to say, okay, there was an implementation of something here and the next version of this has been built on top of the previous version.

25:30Anush Elangovan:And the next version is better because the models and the RL in the middle has helped look at that manifestation and it's like a flywheel that just keeps spinning up. Yes, it's a virtuous cycle. It is a virtuous cycle on how those tokens are consumed. So stationary, like, okay, this is the Git commit of this. Sure, in the end, when you finally want to go get your receipt, you'll be like, okay, that is, sure, it's fine. But in terms of product developments and cycles, that is less and less meaningful. But you do want to ensure that at all times you can have a human go in and make sure that there's something there, but the human is going to come in with an agentic magnifying glass that will peel all the layers of whatever else to pick and find like, okay, this is what you're looking at.

26:23Anush Elangovan:And sure, should this be A plus B or B plus A can be the final decision of a human. But to narrow it down to that level will also be assisted by agentic AI. Yeah. And to answer your question on like, you know, how do we see just like agentic flows? And I think the big thing that I'm seeing is like, you know, for the K-shaped future of software engineering, right? Like the acceleration is so huge, but the ability to train and bring people along to get that same acceleration is the challenge as a leader, right? You want to be able to bring the folks that are not yet agentic into an agentic world.