Everything you need to run Mission Critical Inference (ft. DeepSeek v3 + SGLang)

28:50In other words, is the quality the same or somehow something has changed underneath the hood and the model isn't actually producing the same quality? How is the latency? And especially for certain use cases, what is the time to first token? And is that sustained? What is the P95 of that? What is the P99 of that? How well does it handle throughput? When you start getting a massive burst of traffic, does it still sustain those P95s and P99s? How do I make sure that the security of the data being sent into the model is guaranteed? How do I make sure compliance is guaranteed for HIPAA use cases? And how do I make sure that the data remains within a certain geo for compliance reasons or for latency reasons?

29:38And so those are the concerns that the customers are coming to us with. Less so about, hey, here's my model. Make sure you run it with your TLM or make sure you run it with SGLang. Can you maybe give us an overview of all the different frameworks that people might use? So you have SGLAN, TRT-LLM, VLLM. Those are kind of like maybe the open source research ones. And then some of the other commercial companies are building some of their own stuff. But what's the state of the art today? Maybe like the top three most popular. And then we can talk about why SGLAN came to be and what makes it different and some of the performance boosts that you get.

30:15Okay. Yeah. I think for the common use case, maybe not the DeepSea Cafe 3, for the common use case, I think SGLAN's performance is better than FLM, and its usability is better than TENSR-TLM. So when users care about the performance and the usability, I think they will choose SGLAN. And for the DeepSeq Phi 3 case, because we do a lot of optimization in SGLAN, something like DeepSeq Phi 2, they proposed attention parent named MLA, multi-latent attention. And I think SGLAN is the only framework to support that. Maybe Lite-ALM and TRT-ALM also support, but PLM doesn't support. And also in SGLAN version 0.4, we also support the DP attention for DeepSeq.

31:08And in the latest SGLAN release, we also support the blockwise FP8 kernel. And that kernel was adopted and copied by PLM later. So I think we have done a lot of optimization for DeepSeq. That's why SD-Line is the recommended engine by the DeepSeq team. And maybe one thing to point out, and I think this is important, is that the framework that you choose is part of the equation for running mission-critical inference workloads, but it's only a part of it. So maybe I can draw this out just based on the experience, based on what I've seen in the market as to what it takes to run mission critical inference workloads in production.

31:53I think it takes three things. And each of them individually is necessary but not sufficient. One is performance at the model level. So in this case, how fast are you running this one model running on a single GPU, let's say. The framework that you use there can matter. The techniques that you use there can matter. The MLA technique, for example, that Ine mentioned, or the CUDA kernels that are being used. But there's also techniques being used at a higher level, things like speculative decoding with draft models or with Medusa heads. And these are implemented in the different frameworks, or you can even implement it yourself, but they're not necessarily tied to a single framework.

32:37But using speculative decoding gives you massive upside when it comes to being able to handle high throughput. But that's not enough. Invariably, that one model running on a single GPU, let's say, is going to get too much traffic that it cannot handle. And at that point, you need to horizontally scale it. That's not an ML problem. That's not a PyTorch problem. That's an infrastructure problem. How quickly do you go from a single replica of that model to five to 10 to 100? And so that's the second pillar that is necessary for running these machine critical inference workloads. And what does it take to do that?

33:16It takes, some people are like, oh, you just need Kubernetes. And Kubernetes has an autoscaler and that just works. That doesn't work for these kinds of machine critical inference workloads. And you end up catching yourself wanting to bit by bit rebuild those infrastructure pieces from scratch. This has been our experience. And then going even a layer beyond that, Kubernetes runs in a single cluster. It's a single cluster. It's a single region tied to a single region. And when it comes to inference workloads and needing GPUs, more and more we're seeing this, that you cannot meet the demand inside of a single region, a single cloud's single region.

34:00In other words, a single model might want to horizontally scale up to 200 replicas, each of which is, let's say, two H100s or four H100s or even a full node. You run into limits of the capacity inside of that one region. And what we had to build to get around that was the ability to have a single model have replicas across different regions. So there are models on Base 10 today that have 50 replicas in GCP East and 80 replicas in AWS West and Oracle in London, etc. And that was a big investment that we had to make. The final one is wrapping the power of the first two pillars in a very good developer experience.

34:44To be able to afford certain workflows like the ones that I mentioned around, you know, multi-step, you know, multi-model inference workloads. Because more and more we're seeing that the market is moving towards those, that the needs are generally in these sort of more complex workflows. So these are the three pillars that it takes to run mission-critical inference workloads. And the choice of the framework, the serving framework, is really a part of the first pillar. And that's something that I'm seeing in the market that people who are somewhat new to it, they're like, well, VLM equals equals production.

35:21That's what it takes to run inference workloads. And in practice, that is not true. And I wanted to call that out. I agree with Amir because I think it's open source libraries such as VLM, SGLang, LATLM, or TENSRTM. They only provide a library. They don't provide a product solution. Yeah. Can we maybe talk about some of the SGLang unique things? I read through the paper. It sounds like some of the main use cases, like when you have very large batches, which makes sense for your use case. And also kind of longer context. You know, what was the decision buying creating the framework? which I think is like around one year old.

35:58I think the paper came out December 2023, something like that. So it's still fairly new compared to some of the other ones. And then maybe what were some things that you had to change, you know, as you built it or any fun stories? Yeah, yeah, yeah. I think last year, oh, not last year, sorry. At 2023, maybe August, at that time, Viamin and Ying want to create the SGLN maybe for the front end, something like LLM program. They want to solve that problem. And at 2024, January, they support something like Redix cache. It's a prefix caching technology. I think SG-Line is the first framework that supports prefix cache.

36:39And at February, they also support a concentrated decoding and support some jump forward. So at that time, it's a no for the language generator, not the inference backend. And at 2024, July or June or July, we want to make SGLAM a fully functionality ALM inference engine is just equivalent with the VLM or with the TensorFlow ALM. So at that time, we published a blog compared with other frameworks. And its performance is amazing. At that time, I think its performance is maybe three times, its throughput is three times than VLM. So after that, VLM also do some refactor to make it faster. And at September and December, we continue to release new versions for SG-Line.

37:36Yeah, we support some DeepSeq optimization such as MLA optimization, DPR tension optimization. And we also support the serial overhead, CPU schedule. Also, we support something like SGLN router for the cache aware load balance. Yeah, we deliver so many features. We just build and ship. And I think why Liamin and Ying want to create a new framework rather than use the existing solutions such as VLM or TensorFlow RT. because at that time, you know, at that time for the VRM, I think it's easy to use, but its performance, maybe it's not good. Some design, I think it's not okay. Yeah, maybe the code is a little messy.

38:21And if you want to extend some new feature on top of that, it's a little hard. And TensorFlow RTM, I think it's blazing fast. Its performance is so good, but it's not easy to do some secondary development. If you want to add some new feature, it's a little hard. So just think about, oh, how can we create a new framework? It can achieve the good performance. Also, it's easy to develop, to maintain. So that's why they create the SGLAN project. Let's run through maybe the three main techniques behind SGLAN. So the first one is a radix attention, which focuses on KVCache. when you think about a model that is, you know, as large as DeepSeq v3, especially, like having better KVCache reutilization is great.

39:08Can you just talk a bit about that performance impact? Yeah, Redix cache, I think it's the technology of the prefix caching. And it is a special case for something like block size is one, you know, for VM or for other frameworks, they use something block size 32. And SGLAN use the block size one. I think if you use the block size one, you can make the cash hit rate higher than other frameworks. I think that's the many benefits. And for your case specifically, how does that change when you have like a base 10 type use case where you do not have a share endpoint versus like, you know, is this less helpful for GPU clouds to do one model for like many people that have like very different use cases versus like when you have just one endpoint for one customer?

39:58I'm sure to have a system prompt that a lot of models share and things like that. Anything you want to mention there? Yeah, we've seen this be massively helpful for the reason that you mentioned. There is a certain sort of finite number of prompts or at least prompt prefixes that are being used per customer. And what we've seen is that prefix caching and the fin techniques to make that better has been massively helpful. However, we still had to build on top of that. The example there is that you have a model with dozens of replicas, each of which has its own state of KVCache. A new request comes in.

40:42And what we used to do back in the day was that that request would be randomly assigned to one of these replicas. But the better way to do it is that knowing the state of KVCache in these different replicas, trying to decide which one it should go to. One of the parameters that you need to consider, there are other parameters to consider around the size of the queue at each of the replicas and the location of each replicas, depending on how geo-aware you want to be. But adding that additional consideration around KV cache-aware load balancing was something that we saw improve latency quite a bit for our customers.

41:23And then the second part, which was maybe the harder one to understand as a practitioner, which is this idea of like turning some of the decoding process into a finite state machine instead of a more open-ended. When you're using, especially for structure outputs, can you maybe explain what that means? And I would love to learn too. So maybe this is an opportunity for everybody to better understand how you think about going from a normal kind of like token by token decoding to having a more, I wouldn't say precompiled, but like pre-understanding of what the paths are going to be. I think SGLAN supports concentrated decoding, and it also supports jump forward.

42:02And we use something like outline or the X grammar to do something like change the comfort of the schema from JSON to the FSM, the state machine. And we can use the state machine to control the output, something like the output maybe to the JSON mode or something like. It should be obey some rule. So in that case, because the output should obey some rule, so you can skip some token, something like you should decode four times, but you should obey that rule. So you can get that token in advance. You can just use one preview to replace the full decoding, for example. So that's why you can jump forward.

42:48I guess the question is like, why doesn't everybody do that? When I was reading, I was like, this just sounds better, especially both for accuracy. You know, you're kind of constraining for structure output as well. You can do faster decoding. Are there downsides to it as well? I think maintain jump forward is a little hard. Yeah. At later, we support something like CPU overlap. I think in the overlap model, we even make it compatible with the jump forwarding. Because just if you want to maintain the jump forward with other features, it will be more complicated. So I think we only use the fault setting.

43:25We disable it by default. But if you want to enable it, you can just use some arguments to enable that. But it's a little hard to maintain, especially compatible with other optimization features. Just as a side note, you mentioned Xgrammar, which I never heard about. And I looked up the GitHub repo. It's actually from MLC, which we talked to TQ, I think, a while ago. Any comparisons between Xgrammar and Outlines? Is there a trend in this world or is it mostly settled science? To be honest, I prefer ex-grandma, you know. Okay, yeah, tell us. MLC AI is founded by Tianqi. Both Tianqi and his ex-grandma's other Yixing Dong, they were the students graduated from Shanghai Jiao Tong University.

44:13And the creator of SG Lang, Lian Ming and Ying, They also graduated from Shanghai's Jiao Tong University. Oh my God. Is it the Berkeley of China? Yeah, you're right. And I think Xgrammar's performance is better than the Outlines. And also in the TensorFlow RTLM, the latest release, TensorFlow RTLM also integrate Xgrammar as the backend for the constructed coding. Okay, this is new to us. We had Remy from Outlines speak at my past conference, but I wasn't even aware of Xgrammar being a thing. But yeah, I mean, structured output is something that a lot of people care about. We had OpenAI talked about their structured output implementation, and there's a lot of interest in making sure that there are no trade-offs.

45:01I think there's a little bit of FUD around how maybe the models are dumber when you use structured output instead of the sort of base, sort of next token generation, but I don't think it's significant that much. Yeah. We can talk about the last one, which I don't know if it's as relevant for Base 10, which is the third technique of SG Language is API speculative execution, which seems to be only for API-only models. Oh, yeah. I think it's the front-end feature. It's not the back-end. Yeah. Something like you have some control flow for the LM task. Yeah. Such as you have the one request to get a result and you just continue to another call.

45:41And for this case, you can use the SGLAN front-end language to describe the control flow. It will make it easier to control that pattern. Okay, awesome. Tracing this human path, I'm pretty sure I know the answer, but is there a reason big projects like Grok, like XAI, also use SGLAN? Yeah, yeah, yeah, right. Is it just the same people? Yeah, yeah, yeah, right. Right. Lian Min and Yin are the XAI member of the technical staff. I mean, it makes sense. I wonder if it, you know, what's the impetus for SG Lang to kind of break containment? It seems like VLM obviously has the, it's one year older. It has more community pool.

46:24I wonder how this will shake out. I don't really know. But, you know, you said it's a library. You said VLM's library of SG Lang is much, much more comprehensive. I mean, do people care? Maybe it's like when you're serving models at scale, then you start really prioritizing the sort of performance that SG-Lang offers. I think if you care about the performance, maybe TensorFlow RTM is the best solution for now, especially for the latency sensitive scenery, TensorFlow RTM doing well. But if you also want to implement some feature by yourself or do some optimization by yourself, you want to customize the framework.

47:06I think SGLAN is a good option. And VLM, I think VLM's community support is very nice because it was used by so many users and it has so many GitHub stars. And, you know, SGLAN, when I participated in the SGLAN team at July, it has only 2 ,000 stars. And right now it has more than seven stars. Yeah, I think it also grows so fast. Anything that people should look forward to that's on the roadmap for Xilang that people should be aware about? Yeah, yeah, yeah. And we post the roadmap in the issue and we ping that issue. Also, we have bi-weekly meeting to think with the community about our progress, our plan, which feature do we want to implement in this quarter, something like that.

47:55And we also co-host some meetups, something like the first meetup we co-host with the MLCLM and FlashInfer. And we also participated in some hackathons, something like CamoAI hackathons. We do the presentation about SG-Line. I just saw it now. It sounds like actually that there's, you know, we mentioned Eagle and mentioned Medusa. I think Amir mentioned Medusa, but Eagle is also part of that cabal of speculative coding techniques. It looks like now you support it. Yeah, we already support it. And I think in the open source implementation, something like VAM, SGLAR, and other frameworks, I think it's the SOTA performance.

48:35And currently, even use the TensRTM, it only supports Ego1, not Ego2. And one thing to note about speculative decoding, different versions of it, is that the framework supporting it is one thing, but you will have to do the job of the training of the draft models or the additional heads. And a lot of the benefits will come from how good you are at the training aspect in terms of the data that you use to train the draft model to essentially distill the target model or mimic its behavior so that you can have a very high rate of acceptance. it's the kind of throughput improvement that you get are ultimately dependent on how good of a job you do at training the draft model in the case of the draft target model mechanism.

49:31So that's another thing that's like, hey, does the framework support it? Or you can just turn on speculative decoding with a flag. That's not the case. There's more that goes to it. One more side thing on training i also noticed that with opni offering fine-tuning for o1 and all these things i think people are also very interested in sort of rl trainers is what they what you have here it looks like you're supporting hugging face tl and open rl hf do you think that this will become something that a lot of people are demanding like the sort of general feel of rl for lms relatively abandons i think up till like the end of last year basically yeah i i think so i don't know it's like it's One of those things where maybe people have to wait for a base model that has some layer looping or some other sort of friendly architecture for reasoning instead of just pure RL on LLMs.

50:26Because I think so far people have not really exploited RLHF as much in the wild. I mean, correct me if I'm wrong. Yeah, so I can give you some examples of when we've seen at work. Again, this is generally done by our customers before they come to us for inference. But there's examples like in the healthcare world, fine-tuning models for understanding medical jargon, like for Whisper, for instance, a version of Whisper that I can actually understand medical jargon. So that's not an LLM use case. In the LLM use case, staying in the healthcare space, the models that can do medical document extractions and do a very good job compared to even state of the arts because of the data that the company had gathered through human in the loop.

51:17Are those going to go away? The need for those are going to go away because there's a model that can do reasoning and do a very good job at it. I don't know. My intuition says, yes. Will it be cost effective? That's the question that I have. In the short term, no. In the long term, maybe. But I haven't seen the need for the more traditional fine-tuning actually go down. In fact, we see that quite a bit right now in the market. The question for us is, do we want to address that market knowing that the entire market might go away one day? And my general answer to that is, let's solve today's problems.

51:57Even if they're not around in two years, you will learn a lot along the way by onboarding customers that have today's problems. And you'd learn from them about tomorrow's problems and you will build ahead for them. Hang on. Why do you think fine tuning might go away? Because like you said, there's going to be models with complex reasoning capabilities that can actually figure it out in a few shot kind of way without needing a large data set to fine tune the model with. That's what some people are saying. I really have trouble believing that that'll be the case. Much more believe that it's just easier to change your prompts rather than actually do full fine tunes or even parameter efficient fine tunes.

52:45For sure. Is there anything else that we haven't touched on that you wish the people ask you more about because it is something that is very interesting from your point of view of what you're seeing among your community. Yeah. When we released the DeepSeek Feast 3 support, we have some community user, something like a cursor. Do you know cursor? I think it's very popular. Of course. Of course. I use them every day. When I type code. inside of my IDE, my terminal, it actually opens cursor instead of VS Code. I feel very bad for VS Code. And when we released the DeepSync Fee Story support, some employees from the Curso team also very interested in our implementation and reach out and ask some questions from us.

53:31So I think as SGLAN grows faster and the features optimization, we iterate so fast. And I think there will be more users from different companies, from different teams to use it. Honestly, I would go back to what I emphasized earlier, which was that I wish more people asked about what it takes to run mission-critical inference workloads. Because I see this in the market sometimes that they're like, well, I can just use VLLM and that puts my model behind an API and that is production. But really, it takes three pillars that all need to be there. One is performance at the model level. That is where frameworks that we talked about today really help you with, but you still have to guide them.

54:24Like when it comes to speculative decoding, yes, they support it, but who's going to train or fine tune the draft model or the Medusa heads, or who's going to ensure that the reliability of the VLM server that you see it in production, like there are crashes, how do you recover from those without affecting production traffic? But by itself, that's not enough because invariably that one model running on a set of hardware is going to get too much traffic that it cannot handle. And at that point, you need to horizontally scale it. And that's not an ML problem. That's not a PyTorch problem. That is an infrastructure problem to ensure that you can horizontally scale up your model extremely fast to meet your P90, P99 latency requirements.

55:15And to ensure that you're not running out of capacity in a single region that that model lives, you end up having to scale that model across different regions and across different clouds even to ensure that that model is not being starved of resources in the one place that it lives. So that's an area of investment that we started investing in some time ago and really paid off this past year. And the third pillar is enablement of workflows. Workflows such as the sort of AI phone call example that I told you about, the ones that require multi-step, multi-model inference, but in a very low latency way.

55:58That's the third pillar that really allows the developers to be able to use the power of the first two pillars and then be able to combine them when especially you need multiple models for your workflows and doing so in a reliable way, repeatable way, and a low latency way. And those are the three pillars, honestly, that we have been investing a lot on, some of which we started investing in three years ago, and it really started paying off a year ago. So it takes quite a bit of build to get it to the point where you're truly running folks, customers' mission-critical inference workloads. What do I mean by mission-critical inference workloads?

56:41Inference where if inference is slow or down, the main product of our customer is slow or down. So they really care about it. They have strict requirements around latency, around being able to support large throughput, about being able to do so in a way that, you know, other customers' usage doesn't affect the SLAs that they are getting and dealing with noisy neighbor problems. An inference done in a compliance way, whether it's HIPAA or certain SOC requirements. And also inference done in a geo-aware kind of way, both for compliance reasons and also for latency reasons, where you forward the traffic has an impact on latency in situations where 50 milliseconds really matter, 100 milliseconds really matter.

57:29And we're seeing more and more of those use cases. Well, one way I would recommend doing that is kind of a manifesto type of thing. I'm sure you know Heroku's 12-factor app. I've seen that, yes, yes. That's a good idea, actually. Yeah, maybe even put it on a separate property than Base 10 and just go like, here's what we think, you know, mission critical a application should be and you know have some thought leadership there and flesh it out and you know see if the market takes it on as a mission and obviously you will be best prepared to to serve that market as well i've also seen this done very well with enterprise ready.io i think it used to be done by i think it's called gravitational or replicated one of those from uh replicated yeah yeah these kinds of things when you have a list of requirements when like you're like look everybody needs this okay like write them up and then you know put a little bit of marketing on it split it out from the main company brand that tends to work very well yeah good idea cool well thanks thanks so much for your time i think this is a really good dive into both base 10 and sg lang and a little bit of deep stick v3 which people are very interested in i'm trying to talk to them as well because obviously they're uh they're a fascinating lab and but you know i think you guys you guys are doing a lot to make it accessible for everyone so thank you so much And as a, just to give based on some street cred, you know, they were one of the first sponsors for Latent Space events.

58:50And Amir brought a hundred croissants at our Latent Space hackathon in 2023. So yeah, I just want to bring that up. I always tell, I saw Phil and at AWS reInvent and I told them that was one of the first events that we really did. And one of the turning points of this industry, as far as community goes, in my mind, you know, everybody, everybody, everybody was there. The croissants? No, no, not the croissant. The event itself. Yeah, entire companies launched. Yeah, you were, I mean, you know, like Natter from Brev was there and did the, with Joseph from Roboflow, they did like the prompt battle thing.

59:23Like Harrison was a judge and it's like Jerry from Lomindex was there. Like kind of like everybody that is kind of now breaking out. If you look at the graph that Jensen put on the screen at CES with some of the companies they work with, a lot of them were at that event. So yeah, thanks for staying involved with us, Amir. and I'm sure we'll do more together. And thank you guys. Many more years to come, for sure. Thank you for taking the time today. Good to see you both. I'll see you, Sean.