Infrastructure Scaling and Compound AI Systems with Jared Quincy Davis - #740

27:24But you have some kind of best of N construction. These are very different. In the best of N case, I might want a million GPUs for 400 milliseconds to do these inference calls. I don't care if they're interconnected. They can be part of different or fit of N domains. I don't really care. in the vertically scale case, I want a highly interconnected system for 10 contiguous minutes. You know, and everything should be part of the same domain. It's one contiguous block of compute. These are very different cases. They also might have different implications depending on the task for the performance I can expect.

28:02But these are very different cases from a system's perspective, right? You know, and so one of the things that's cool for us to saying okay to what extent like these things should probably be priced differently they should be scheduled differently you know if there's too much of one than the other then becomes actually a little bit better it's like hey there's a lot of contiguous compute blocks already allocated that i can't disrupt i can fit you know kind of less contiguity requiring workloads in between, you know, et cetera. So you can kind of push, nudge, you can kind of nudge things. You can kind of nudge to use this broader pool of resources really efficiently and say, hey, I've got a lot of stranded workloads because I've got a lot of contiguous work going on, a lot of contiguous systems.

28:52So I've got a lot of stranded stuff. So if you have a stranded workload, you know, be my guest, come fill this. You know, actually the costs are relatively, are going to be relatively low because it would otherwise be stranded and unutilized and then vice versa. So it gets really, really neat. And it's a part of the thing that we're trying to do with Foundry, but also this research on compound systems with our broader set of collaborators is trying to, you know, make sure that as we get into this world that's highly authentic with these very, with this diverse almost ecosystem of systems occupying their own ecological niches, that we can actually, that we have the infrastructure that can actually support this.

29:25That can, you know, in the right way, in an elegant way. So hopefully that makes some sense. No, it does. It calls to mind for me the world of, call it like this transition over a prior era from supercomputing to distributed systems, right? There was a time in which in order to run any large computationally intensive workload, you needed computer systems with shared memory. And then eventually we got to things like Beowulf clusters that had some degree of shared infrastructure. And then over time, more and more of those workloads that required this like highly integrated system got chipped away.

30:09And now, you know, you've got people running a lot of the things that used to require these super computing systems on Kubernetes clusters and highly distributed independent systems. And a lot of that was changes in algorithm. A lot of that was the development of tooling. and just hearing you talk about this, you can kind of see the same thing playing out on the AI side where because of the lack of, you know, the infrastructure work or the algorithmic work and the frameworks, for some workloads, you might just want like NVIDIA's biggest box. But more and more of that's going to be using inference parallel or parallelized types of implications in a very distributed setting.

31:03100%. Yeah, that's a really good thing you're raising here. Yeah, I think maybe two ideas. One would be, I think there's a pendulum, right? And it's almost like a competition, you know, there's almost like a deeper way to explain this. But essentially, I think there's a lot of things that people talk about, which is, hey, will more of the work be inference or will more of it be training? And I find a lot of these questions, you know, I don't think they're the right questions. For example, I think that these things will always balance each other out. So for example, I think one of the things that we all know is if you're doing a lot of inference, then you might actually want to train your model so that they're cheaper to inference, meaning distill them, meaning over-train a small model, kind of pushing the chinchilla scaling law somewhere they shouldn't go or they weren't initially intended to go.

31:54So you have, you know, over-trained a small model so it will be cheap to inference and the lifetime costs are lower, et cetera. So there's kind of a natural way in which these things will balance out, you know. And so I think that what this kind of hints at is that point I've been kind of hammering, which is around co-design. I think, I don't think that people fully appreciate the extent to which the systems that we have have dictated the types of research that's, you know, that's been ascendant over the history of deep learning. I think, frankly, I think deep learning overall, part of the reasons deep learning is the ascendant ML theoretic paradigm is because it scales so well.

32:32It's kind of a bitter lesson type notion. It scales so well, it assimilates compute so effectively. And so as Moore's law progressed, deep learning is one of the methods that could take advantage of that. You know, same thing with the methods within deep learning that have been really ascendant, just a super dense transformer attention computation versus kind of a neater LSTM type of thing that's not as efficient at using the fact that we have these really, really dense parallel systems that are initially designed for graphics in some sense, but are really, really good at these dense maestrian operations, et cetera.

33:04Like, I think that the people don't fully appreciate this. And so one of the things that I guess is exciting to me about, you know, the next little while is I think we are going to see, you know, to the extent that people all converge on one approach, you know, vertical time scaling, whatever it may be, I think we're going to see that actually opens up, you know, kind of creates, as long as there's a system that can actually manifest this, like what we're doing with the scheduling, if there's a lot of contiguous blocks of work, that actually frees up a lot of gaps. And if any type of workload that can fill those gaps, you know, kind of is occupying an ecological initial to itself right and isn't facing competition in the same way and you know that can be cheaper and so i think these things will balance out and the pendulum will continue to swing um as it has for for quite a long time i think you invoke kind of the map reduce and you know the history of these things and i think that's very much going to play out i think we see the same thing with async kind of work versus real-time work you know i think you'll see that as methods are more and more real-time there'll be a benefit computational benefit to having things that can run in batch mode async in the background.

34:11And, you know, I think these things will, I think they'll be very, very, very, very diverse and interesting landscape going forward. And so I think if you're a systems NML person, there's never been a better time to be alive. And I think for people who are one on one side or the other, I'd encourage them to try to get more familiar with the other side of that. Yeah. So, you know, and kind of thinking about all that we've discussed with regard to compound AI systems, like practically speaking, if you are, you know, machine learning engineer or researcher, and you're trying to actually do something, like how do you take these ideas and put them into practice?

34:49Yeah, for sure. I think that for now, the one primitive or the one idea I'd encourage people to think of is just ensembles or best event, you know, as a reduction, as the community, you know, gets more and more intelligent. I think that in the case where you want to push the reliability or quality frontier, which I think that's where most people are. I think most people aren't trying to optimize costs yet. In my bias view, you know, they're not yet that stage. They're trying to get it to work. You know, they're trying to, you know, trying to get to work, trying to make the reliability good. They're also trying to cross that demo to real-time chasm, where you start being judged not by the outlier good examples, but by the outlier bad examples, they're trying to make sure that everything works.

35:31In that case, they want GPT-6 early, They want GPT-7 early. I think one of the really simple ideas that people can try to employ is just calling the frontier models multiple times and trying to combine the answer, at least on those verifiable tasks. That's a really simple thing that works pretty well. And actually, in this Network of Networks paper, we're able to characterize when this will work well and by how much it will help in terms of three simple numbers. the numbers being some base difficulty or just a base likelihood that you get the answer correct with one call and then also some you know basically soundness and completeness number which is a characteristic of a verifier which is if an answer is right how often is a verifier model able to certify it as right and in a case where an answer is wrong what's the fidelity of the verifier in terms of saying hey this is wrong and rejecting it if you have those three numbers a base correctness probability, a verifier soundness or certification probability for correct answers, and a verifier rejection probability, then a very simple ensemble or best event structure with generators and a judge will push the frontier pretty reliably.

36:44Obviously, it will keep through costs as well, but these models are actually overall pretty cost effective, at least at tasks that aren't many, many, many steps or multitask. And so I think that's the thing I would encourage people to try and we actually have this framework called ember that we created trying to make these primitives really really simple to construct and express and efficient to run um that you know we're trying to make this easier and easier for people to construct these things and yeah i think that base ensemble you know is already a great place to start and also has other nice benefits like you can use the disagreement between ensemble members to get some sense of okay is this thing hallucinating or not.

37:22You know, you can kind of, and that's already a pretty good place to start, to, you know, try to employ these ideas in their own work to get things that maybe weren't working to work. Yeah, so that's what I'd be, that's what I'd encourage. And, you know, but also I'd encourage people to, you know, follow this area of research to, you know, start being creative themselves. I think we're in the early era. It's kind of like 2010 deep learning, you know, where a million flowers were blooming and there were live ideas and stuff was just starting to work. and people are really excited. I feel like it's 2012 again, but in this new kind of networks of network setting, not neural networks.

37:58And so I encourage people to say, hey, it's 2010 again. That was a fun time, the 2010s. Yeah, get excited. Do research. If you're an academic, you can push the frontier now. You don't need to be at a frontier lab. I think that's what I encourage people to think about. With regards to Ember, Or what is the, like, describe that as a framework to work with. What are the ergonomics of it, the developer experience? It kind of came out of our own research on these architectures, on these for-time scaling architectures. And we wanted to kind of be like what Torch was for neural networks for networks of networks.

38:35So we wanted to be fast to run, fast to write. The contributors, a few of them were the original contributors behind Ray and Spark. And so Ember is from that same family. And so we're reusing that Spark banner of Faster and Faster Right. And so it should be, you know, things that would take a long time or be, you know, have all kinds of little pitfalls like constructing these big graphs, these big almost like neural network graphs, but out of inference calls. Want to make that really simple, concise. And then also want to be efficient in the sense where, just like with neural networks, there's vectorization, you know, some parallelization, et cetera.

39:07You can do that here, where maybe you have, if you have a large graph of calls, you want to do some type of topological sort and parallel dispatch. rather than just doing a for loop, right? And so it'll run faster and you won't be super slow. So we want it to be concise and also pretty efficient in terms of run these things. And so that's what it is. It just has a little bit of like a Jax slash PyTorch ergonomics. But then there's also like a kind of Unix-y, a string-based compact syntax that someone built as well. But primarily it's kind of like Jax or PyTorch, but instead of neural network modules and nn.linear and nn.dropout, things like that, it's these network and network modules.

39:42and non.ensemble, non.judge, things like that. So it's just a bunch of primitives, you know, tested, validated, you know, trying to make it easier for people to do this work. And we've been using it with collaborators spanning a pretty broad range of universities and also companies. And we've been, you know, there's a lot of really cool work going on that I think will be coming out over the coming weeks and months from the community. You know, some projects I'm pretty energized by that I don't want to steal their thunder by leaking any results. But yeah, I think that's the goal of that, is to make it really easy to search this space.

40:16Just like Torch made it really easy to search that neural network space in the 2010s. And kind of looking through the code on GitHub or sample code on GitHub, the basic ensemble seems fairly straightforward. or I think where some of the complexity might come in in terms of how do I apply this to my use cases rather than like the voting and the judge elements. Can you talk a little bit about how one should think about building those out? Yeah, I think the most basic variant is already, already works pretty well. So I would encourage people not to, you know, overthink it for the first variant. The most basic thing is just, you know, These models are already pretty smart, and so you don't have to have a really fancy prompt to start to get some advantages.

41:06You can just have a number of generators, let's just say, as a default. I think the default in the system is like 10, you know, 9 or 10. I think it's actually typically an odd number depending on the aggregation function. And then you have a judge, and the judge is, you know, some bigger, more powerful model. The generators, you know, can be that same model or some cheaper model. And basically just ask the question to all of the generators, tell them to generate candidate answers and then you just feed those candidate answers to the judge and say hey here's some advice from your advisors you know your council of advisors you list it given all of this produce a final response to this original question that very simple construction is often pretty helpful people can test it a couple times in their setting and see if it helps but it's often quite helpful Yeah, I'm thinking about this in the context of a use case that I've worked on, which is surprise, surprise, podcast transcription.

42:06Right. And, you know, what I've found was that, like, particularly for a technical, you know, podcast like this, there are a lot of, you know, names and places and things like that, that, you know, your off the shelf transcribers would get wrong. it. So the approach that I ended up taking that worked really well was to scrape a lot of information about a guest, like the stuff that they've worked on online, and then go like essentially paragraph by paragraph and ask a model to, you know, given all this information we have, like what in here is probably a transcription error and not right. And it works really well.

42:49But in light of this conversation, I'm thinking that another approach would be, you know, just do it 10 times and, you know, have something like try to rationalize across those 10 times because it was often the case that, you know, it's just a randomization, a random nature or probabilistic nature of this where sometimes it would be right and sometimes it would be wrong. and the additional machinery that I built, like it increased the likelihood that it was right, but it wasn't because I was necessarily introducing new information. It was just like biasing it towards that new information, if that makes sense.

43:30I know exactly what you mean. Yeah, 100%. Yeah, two ideas come strongly to mind here. You know, one is in this our moral M calls all you need paper, you know, we're able to kind of characterize what you're describing a little bit, but also find out when this doesn't work when it does work. And it's actually very nice intuition. Basically, there's what we change between the networks of networks paper and the RMRLM calls all you need paper is the nature of the aggregation function, right? So they're both ensembles, but in one case, you're doing best of N. In the other case, you're doing most common of N.

44:06And these have very interesting theoretical properties. Most common of N is like, you know, democracy, right? It works really, really well. It's like a quorum type of. It's a quorum. Exactly. We found that that simple idea does work really well, but only in cases where it's an easy question. And the default model would have gotten the answer right, but there's some probability that it hallucinates. In that case, the variance reduction of the ensemble quashes that hallucination probability. And so you kind of converge on 100 % accuracy. And my intuition is that that probably is not the best approach in my case because, you know, oftentimes it's like some obscure acronym or something that it's, you know, it would get probably, you know, 20 % of the time right, but not 80 % of the time right.

45:02Exactly right. And so in the other case where it's a hard question, the variance reduction actually hurts you, right? It removes the outlier probability that you get the answer right. And there's actually some funny organizational design implications, like, you know, for, you know, breakout ideas, you probably don't want consensus or, you know, consensus or committee, you know, but I'm extrapolating a little bit. But, you know, I think it's kind of fun to think about. that's a property of this type of voting-based or quorum-based system, you know, or consensus-based system, rather. In the other case, with the best event, you don't necessarily see this, right?

45:39It has its own properties. And that is actually more monotonic in terms of the more ensemble members you have, the more calls you have, the more that kind of leads to a performance increase, regardless of whether you're in the hard regime or the easy regime, which we can talk about. it has to do with that verification versus generation complexity asymmetry notion and the soundness completeness notions which are characterized in that paper but it's kind of like it's kind of funny that number one there are differences in this you know um you know the behaviors of these sort of types of systems and then also you can think about how that applies to to your case right i think it's yeah it's it's a it's a yeah it's a fun it's a fun set of things but yeah i think in your case that's a really neat example this kind of you know come you know trying to, yeah, trying to, the transcription error resolution thing.

46:26Yeah, that's neat. Yeah, one thing that occurs to me in this context is, you know, say we're going for, you know, best event and do we end up with a case where now my, you know, single judge is a single point of failure, like that thing hallucinates and it destroys my whole, you know, it doesn't matter how many generators I have because I'm really dependent on the judge. Yeah, that's totally right. I think that the, so there's a couple ways that you can do this. So the construction we study in the Networks of Networks paper is actually a specific type of judge, right? So this actually is a funny question.

47:03Like what should the judge, what should the judge's prompt be? How should the judge work? The construction that we study for theoretical reasons is what we call a verifier-based judge. It implies though that your problem can be verified. Like it's a verifiable task. It does, but actually even if it can't be verified, it's an interesting construction. So the way it works is, let's say you have 10 candidate responses. You can show them one by one to a judge, and the judge is just trying to produce, is this correct or is this incorrect? Thumbs up, thumbs down. That's all the judge can do. That's all the judge can do.

47:36And so now that starts to get interesting because, and then if the judge says everything's wrong, then you return a random answer, or you return the first thing that the judge says, yeah, this looks good. Now, the funny thing about that, really simple protocol is that it doesn't necessarily make things worse versus like your your average case of the you know initial of like the generator likelihood like unless your judge is like more likely to certify a wrong answer as correct than a correct answer as correct which is kind of extreme so but you see you can look at those probably it's kind of funny i kind of like this because i like cryptography a lot i like theoretical computer science a lot.

48:19It's kind of funny that those kind of P versus NP ideas of how easy it is to verify something versus those cryptography ideas from probabilistically checkable proofs and PCPs come in here, where very rarely is that going to be the case. You'll have some model that's so ill-conditioned that it's more likely to certify a wrong answer is right than a right answer is right. But that's something you can study probabilistically and empirically on some subset of questions, a hold out, et cetera, and kind of try to get a sense of that, right? But as long as that's not like really skewed in a bad way, then, you know, you're actually pretty likely to, you know, like see a gain from this, with high probability, see a gain from this verifier-based judge construction, right?

49:04And so, you know, I think that kind of highlights that there is this design space, you know, a normal judge might be bad, like, you know, might be overwhelmed by context, you know, in some way, like there might be issues that you might, or might just be more expensive. is verifier-based construction can be, yeah, it's kind of funny, like the properties of it are really, they are really kind of clean theoretically in a way that you can kind of guarantee and be confident about. Does this verifier-based judge and limiting it only to kind of thumbs up, thumb down mean that you could use much cheaper models for verification?

49:34Or is there still a requirement that you use kind of a more powerful model as your judge? Yeah, you can definitely use much cheaper models, you know, for verification, as long as, you know, that kind of soundness, you know, completeness story is correct, you know, and that's right. And so it doesn't have to be a powerful model. That's 100 % right. It's kind of a function of the, you know, likelihood that you certify a correct answer versus the likelihood that you, you know, certify an incorrect answer type of thing, right? So that's, you're right. But then also another nice benefit is the thumbs up, thumbs down.

50:04Also is nice because, you know, input tokens are very cheaper than output tokens. You know, and so there's kind of something nice about if you just, if the output is just - Kind of a token compression in your judge layer. Exactly. If the output's just thumbs up, thumbs down, you know, that's actually pretty, you know, and then you're just returning the answer that was already generated by generators, you know, that kind of plays some role into this kind of cost equation as well, which is kind of fun, which is kind of interesting to start to think about. But you didn't talk about kind of parallelizing that judge layer.

50:33Like I was envisioning, you know, you can, you know, turtles all the way down, like, you know, highly distributed networks of networks of networks with each layer being very parallel. Is that something you've looked at? 100 % yeah exactly right so you know to your exact point you could say maybe the judge itself should be a network right you know or maybe this whole little primitive of an ensemble plus judge should be a node within another network with a judge you know it's kind of trees right and so it starts to get pretty pretty rich and still you start to have deep networks of networks instead of deep neural networks you know denons you know you know etc and so it starts to get in you can start of envisage, and then you start to think about, oh, there's loss in the middle dynamics, right?

51:14We know that these models have somewhat limited context, right? And so you can't feed an infinite context length, or even if you do, quality diminishes. So maybe you do want to break things up and have kind of smaller domains where fewer generators are going to a single judge, and then you're aggregating. It's like representatives, right? This whole architecture layer emerges, or maybe there's some notion of skip connections that starts to emerge where you're like, oh, well, you know, something way down below got quashed by this judge layer, so I need to kind of feed it up, Right. You kind of get back to this neural network, deep neural network concepts of vanishing and skip connections.

51:46And these things come back in cool ways, which I think is really exciting to start to think about. And it reminds me of the term that I was searching for before. But like neural architecture design is an idea that comes into mind here, like where it's network of network design, where it's a highly automated process that. But, well, now I'm thinking like, yeah, I've seen evolutionary approaches to this and, you know, other approaches of searching the space. It can get pretty interesting, but I don't get the sense that we're quite there yet. Not quite there yet. Yeah, not quite there yet. It's still the very early days.

52:21You know, a lot of the work that's been so far is like those early days and like perception research and MLP research. we had like these, you know, like kind of wide, but relatively, you know, but shallow networks. And they, you know, had relatively few parameters, you know. And then you're like, over time, we scaled them up and things got richer and richer. And we started to run into vanishing, exploding gradients and started to think about ways to combat that. And, you know, it kind of started getting richer and richer. I think we're in those very, very, very early days still. And partly because I think the infrastructure doesn't support doing richer research.

52:53Like, you know, the latency is going to kill you if you try to make, you know, a thousand calls to GPT-4 with like with for loops, nested for loops or something. It's going to get pretty ugly pretty quickly, you know, and also it's going to get pretty expensive pretty quickly, you know. But then as this dispersion emerges more and more, as distillation works better and better, as we have cheaper and cheaper systems, you know, as you have the infrastructure like Ember to make this faster and faster to instantiate and run these types of, you know, ideas. I think there'll be more and more research and over the next months and years, I think it'll get start to get wild.

53:23We'll have, you know, multi-billion parameter networks, networks, you know, with very intricate structure. And you kind of alluded to this earlier, but you can almost see a world in which, you know, you've got some problem. You kind of define your problem. you have your data and then you, as opposed to training a model, you evolve a network of networks and then you distill that network of networks down to a simpler either model or network of networks. And so I'm wondering like, yeah, I guess we're, you know, we're way, you know, we're even further out ahead, but I'm wondering, do you track any research that looks at like, you know, network distillation?

54:12Are we anywhere there yet, or is that still to come? Yeah, I think that we see early forms of this already in some of the reasoning research. Yeah, so actually, I want to make maybe two points about ways in which compound systems are already kind of around us, but that we don't really recognize as a compound system and haven't taken those examples to their limit. One is specular decoding, and the other is what you just mentioned of kind of distillation of networks and networks, right? Would you add MOEs to that mix as well? In some sense. Yeah, in some sense, that's definitely true. Although maybe in a slightly different way.

54:49Yeah, so let me actually, maybe let me start with speculative decoding because it's just so common, but, you know, and it's everywhere and et cetera. So speculative decoding, you have kind of a funny little compound system where there's a small drafter model and then a big verifier model. And the idea is you can get greater efficiency. that's kind of like it pushes the prior frontier in terms of latency versus quality in a way that's provable, that you're not losing quality, but you're definitely getting an expected improvement in latency, you know, in speed and tokens per second throughput. This is, you know, it's kind of obvious, but the way it's doing it is you have this, the drafting model is running ahead and then the verifying model is kind of in parallel, you know, rejecting or accepting multiple tokens, right?

55:30And this is kind of, it's a funny case of, you know, at least a very simple hybrid system with a small model and a bigger model being combined to push the frontier, you know, the kind of proto-frontier for the overall system in terms of cost, latency, et cetera. And it doesn't actually increase cost by as much as people might expect because of this fact of a GPU, which is kind of funny, which is you want to oversubscribe it, you want to keep it utilized, the cost is mostly CapEx, et cetera. And so having that smaller model doing things in parallel that then can be verified by the bigger model doesn't actually increase your cost the way people would maybe imagine, right?

56:03And so that's one very simple example of a compound system that's kind of all around us, but we don't think of that way. And another is what you're talking about, which is more of this distilling a network. So I think we're kind of already doing this when we have many, many replicas in kind of a standard RL pipeline. You spin up many, many replicas. You ask them to do questions. They all produce candidate responses. Maybe one of them's correct. And that's the trace. That's the rollout that you end up, you know, you're training on, you know, overweighting in your training. Right. That's kind of a, you know, a funny case of using compound systems, you know, but within the kind of framework of RL in a more classic way.

56:45Rather than using that training data to train a single model, you can imagine that system itself being something you just do at inference time. Right. And there's obviously the whole spectrum. And so. And by the way, this type of thing's been done multiple times where in a lot of this chain of thought research, one of the basic things people have been doing is saying, let me go ahead and do multiple steps multiple step rollouts of the current state of the art model get it to generate 10 consequential improvements based on some question, just initial reasoning and then let me take that entire thinking trace and use that as my training data to train the next generation of the model to then try to produce that whole 10 step reasoning trace in one step and then let me try to bootstrap off of that do that again and do that again.

57:34There's, you know, and obviously there's all kinds of trouble in that potentially, um, it's still, you know, perspicuity thing, but I think you can kind of see, oh, you can't see the contours. You can see how that would work and how that would, how that would extend that type of, that type of thing. So I think that's been, that's been already in the works, um, implicitly at least for a while. Um, yeah, a hundred percent. And yeah, it's a, yeah, it's pretty, it's pretty, pretty interesting. Yeah, I think something that I don't think, you know, everyone maybe knows or has intuition for is, we mentioned earlier how shocking it is that distillation works so well and just how well distillation works.

58:16I think that even at the hardware side, I don't think people appreciate how well ASICs work, you know, application-specific integrated circuits, how well these, when you have a task in mind, if you can design the chip for it end-to-end, how well this type of thing works. And I think that's why the kind of, you know, compound systems or system of models or menagerie of models type of vision. I think the GPU in some sense is a proof point that like, you know, application specific system built in a co-design way with an end use in mind can really push the, you know, bend the curve, you know, on cost.

58:52Yeah, it's interesting that, like, you know, there's one lens in which a GPU is a general purpose building block, but then there's another lens that you can look at it as an application-specific building block for graphics, and now, yeah. Yeah, it very much is, right? And I think it was kind of obvious that the CPU was not the right form factor, you know, for a lot of use cases, you know, in a fundamental way. And that's why you need accelerated computing, and that's why you need the GPU in a fundamental way. I think people haven't taken that to its limit, you know, or even recognized, at least, obviously there's some natural, like, you wouldn't necessarily need infinite variation and infinite specialization, but there's at least different poles.

59:38And we've seen this in the last few years of deep learning, where the TPU is very much explicitly described as a co-designed system, where there's, you know, elements of, you know, what level of precision do you need, right? you know the precision format the you know kind of design and in its in the sense which it's well suited to parallelism but then also the the models themselves and the learning the learning the actual learning algorithms and how tolerant they are to noise from lower precision etc like all these things co-designed into one system yeah i don't i don't think you know people have kind of really appreciated this and so i think when it comes to that question of will the future be compound systems or will it be a single model you know i think i i basically point people to at least a cpu versus a gpu to say at least there'll be a couple different poles right there'll at least be that kind of you know perhaps small model highly distilled you know with big models at least that type of pairing if not something even even richer you know so yeah it's a i think it goes back to this kind of theme that i'm continuing to push which is you know one basic research is back and there's a lot of work to do.

1:00:44But number two, it's never been a more interesting time to do co-design and to have both the systems and ML or engineering and research perspective on these things. And bringing this all full circle, like given this viewpoint about the way that ML systems are going to evolve and kind of, you know, scaling out of distributed inference makes perfect sense that you'd start a company at the compute layer doing inference. Yeah, for sure. I think the way we, you know, that I thought about it at least is it was pretty clear to me that compute is really a root node problem for Hamel. You know, as I was very much kind of an algorithms deep learning person, you know, but it's pretty clear that part of the reason that deep learning is the synonym L theoretic paradigm in the first place is because, you know, the way it assimilates compute.

1:01:32And so in some sense, you know, deep learning is a root node problem for all of these downstream applications across health and education and you know materials and all these things that i think are really important interesting you know robotics you name it but then the problems that are kind of upstream of deep learning are largely the systems problems and it's pretty clear to me that the cloud as we currently know it the algorithms economic model was not well suited to deep learning and it was really going to narrow our creativity limit the types of things that we could do um and then it was already hurting companies where they you know would have to raise tons and tons of money to buy a long-term contract.

1:02:05The cloud just wasn't working the way it was supposed to. The whole promise of the cloud, of elasticity on demand, infinite compute on demand, on tap, not needing to capacity plan, you know, that whole notion was not being fulfilled in the AI cloud context. You know, all of the desiderata of the original cloud, none of them were being fulfilled really in the AI cloud context. You know, almost all the complexity of infrastructure is being force it onto the users themselves, as opposed to the cloud handling it for you, so you can just focus on making your beer taste better. All that stuff, none of it was fulfilled in the AI cloud context.

1:02:39And you need to really fundamentally rethink the way cloud works, reinvisage a lot of these systems, abysmoles, et cetera, in order to fulfill the original promise of the cloud in the AI cloud context. And so I think it was clear to us that we were not on the path of that happening, that things were not moving with the right velocity, either in the right direction or with the right speed overall across these multiple angles. And so it's like, okay, we need to push this. Otherwise, you know, we're going to definitely reach some limitations at the research level. And so we started pushing on that.

1:03:11And I think I was fortunate to have both a systems perspective, you know, and at least enough understanding of, you know, the cloud and how scheduling works and all these types of things. But then also the algorithms to see, you know, where there are gaps where people on one side or the other weren't thinking about how you need to design the full system. We've got a lot of work to do. at Foundry to fully realize our vision. But I think we've made a lot of progress so far, helped a lot of companies. We've been able to, for certain types of workloads, cut the cost by 12 to 20X, particularly for workloads that are amenable to running in a preemptible fashion or being checkpointed or running in a heterogeneous way or running in a batch mode where they can run, they just need six hours within the next 12 hours and they don't care which six hours.

1:03:54And so for those types of workloads, We've been able to create a lot of, you know, surplus. Yeah, when I look at Foundry and think about it in the context of other companies that are going after similar ideas, like I think of like modal and fireworks, it's like where I would characterize Foundry as like really pushing innovation and like the economic interface of, of inference whereas you know these other companies are more like modal is pushing like the serverless idea fireworks is like more traditional like you were just going to abstract away the call and you use it in a similar way but there are some interesting ideas in what you're doing like you set the price for your workload and you're essentially you know this is how much it's worth to you and like run it when you can like that's kind of an interesting concept and there are a bunch of other like marketplace oriented ideas built into the way you're approaching things.

1:05:01That's right. Yeah. I think one way I'd characterize that is, you know, Jensen and others have talked about how there's a couple of different regimes that you might fall in. Like one regime is the very, you really care about low latency and the others you really care about high throughput, right? Yeah. It's a little bit different. Let's just say in the life of a researcher, let's not even go to the user of ML. In the life of a researcher, I have some kind of verification training and debugging workloads I'm doing throughout the day as a researcher. And in that context, if I have a five minute, 10 minute, 20 minute, one hour queuing time on the cluster, I'm very unhappy.

1:05:32I want it to be snappy. I don't need many nodes. I want like one node quickly, you know, and actually these are short lived workloads. I'm going to pay you a lot more than the typical rate of a GPU, like to get it faster, for example. In the other case, maybe I have a workload that I set off at 8 p.m. You know, as a typical researcher, and I come back the next morning at 10 a.m., I want it done. In that 14-hour span, I don't really care when you finish the workload. It's a six-hour workload. You know, finish it whenever. However, maybe as a one caveat, maybe I want it to start quickly so I can start to see the curve going down.

1:06:05You know, it is confirmed, you know, that there's no error. But then at that point, like as soon as I see the curve starting, you know, I'm going to close my laptop and walk away. Right? And I just want it done by the time I get back. That's what I'm done by the time I come back. And if you tell me that you could do that 10 times cheaper, right, then if I need it done, you know, just in one shot, I'm like, yeah, sure. That's totally great. You know, if other people can do their work and there's 10 times more effective capacity on the cluster because of that, that sounds awesome. Yeah, why not?

1:06:33Right. Versus, and then for that thing during the day, if because you're being efficient and making some things run batch, I can get my stuff snappily, you know, even if it's a little bit more expensive, you know, than the kind of batch rate. I'm fine with that, right? And so I think that there's this heterogeneity. Same thing with these agentic cases. They're going to be things like operator, where I want it to be really fast. Like, it's way too slow. I want it to, like, you know, be as quick as possible, go and perform a task in real time, you know, etc. And I have other things that I'll launch async that can run in the background, you know, like deep research or like the kind of async coding updates, et cetera, as long as it's high fidelity, right?

1:07:11And so there are these different modes. And sometimes just like in the search case with Google, you actually have some workloads like MapReduce and indexing that are background things that you're doing explicitly for the purpose of making it fast when you need to serve. So you can have a distilled model or a really fast model or just a retriever, you know, actually at inference time because you did all this work up front, pre-computing things, right? And so there are these different regimes. And I think you need an infrastructure, you need a cloud that can handle those infrastructure efficiently, you know, pack them all.

1:07:44It's kind of like playing Tetris, right? You know, it'd be much easier to play Tetris. It'd be a crazy cheat code if you could delete blocks that were kind of in a weird shape and making things hard and have them come back later. If you could take a block and split it into sub-blocks or make a block into sand when you need to, right? You know, if you could make a block that's falling fall slower or just like pause it while it's falling, you know, because it's inconvenient, if you could do these things, obviously, you know, that'd be a massive cheat code in Tetris and will allow you to pack the space more effectively.

1:08:12I think something similar in the kind of the cloud context, so we want to have an economic model that would allow people to say, hey, if your workload could be deleted and come back later, then there's a massive economic benefit we can give you for that. That is, you know, based on the effect that you're having on the overall system. You know, I think that's a, that's kind of what we want to be able to express at the industry level, which obviously requires a ton of things. There's some very heavy, virtualization and networking and scheduling elements to this, but then also UX, business model, et cetera, elements to this.

1:08:44But being able to express that is like a really, really powerful thing for the ecosystem as a whole. We have this parking lot analogy I can give that I think even more intuitively and kind of convey some of this, but I think you get the intuition overall. As a sidebar, if someone vibe codes that Tetris game, I really want to play it. It sounds really cool. Yeah, if we have any listeners out there, That is an open invitation, a challenge rather. These types of questions, like these systems utilization questions, were pretty esoteric at one point. Back when some of my collaborators or some of the senior PhD students in my lab were doing work on using spot and reserved instances and on-demand intelligently for a workload, you were saving some costs, but these were like 20K workloads.

1:09:31And this is like the olden days. Now, you know, these companies are spending more on compute than on people in many, many cases from the biggest companies like Google and Microsoft to the smallest startups to companies in the middle like OpenAI, you know, that are very late stage privates, etc. You know, it's the primary economic budget in their P &L. And, you know, and also the dollar amounts, you know, as a result are much, much larger. We're talking, you know, tens of billions of dollars, hundreds of billions of dollars on infrastructure spend. These workloads are certainly more expensive. And so these questions that were previously esoteric about compute utilization, you know, are now central questions, right?

1:10:10And so I think these are pretty fundamental to the future of the field. And so that's partly why we said, okay, we need to build this the right way. You do this via kind of a translation layer on top of another cloud. Phenomenal question. So we do both, right? So we have some infrastructure that we own and operate, but relatively small. But then now we can also partner with other clouds. We have a partnership we're going to be announcing relatively soon, you know, with one of our partners that shows how we do this. And we'll talk about, you know, the way in which we're partnering. Basically, it's a win-win thing.

1:10:42The parking lot analogy makes us clear, but if we can basically pack more work, we're creating surplus. We're creating more revenue for the owner of the compute. We can cut costs for the person whose job is being packed in there more efficiently. You know, and then we can also, in that surplus, we can also capture upside. So it's a really nice structure. And so we want our layer to be run across as much acuity compute as possible, ideally on every unit of compute, every single GPU, but then also literally anything that can do useful computation at a sufficient scale over time. And then different workloads need security or contiguity to different degrees, and that's all managed by the schedule.

1:11:23And that's partly why this layer needs to be smart. The kind of orchestration and scheduling layer needs to be smart. But yeah, exactly right. And so obviously owning our own compute, but there's a limit to how much we can scale that, you know, particularly via venture. And so we ideally would run as a layer across all compute and partner with others to basically create more surplus in this way. And I think if we can do that, not only are we creating a lot of economic value, but I think also that will lead to, you know, just the ML space being so much more dynamic, so much richer as well. Well, Jared, this has been a great convo.

1:11:53I really appreciate you taking the time to talk through all the many things that you're working on. Very cool stuff. Yeah. Thank you, Sam. Yeah, really fun conversation. Yeah, really fun. Your mind, you're pretty creative and generative. So that was a fun conversation. Awesome. Thanks so much.

1:12:29Thank you.