Multimodal AI Models on Apple Silicon with MLX with Prince Canuma - #744

32:41Usually it's just MLX name of the project dot generate, and that should be enough to generate an initial response. If that does not exist, then I go look at the transformers implementation, which they usually have a transformers implementation. I look at the code and I just convert one by one. MLX is very similar to PyTorch in terms of the API design. it's more inspired by NumPy and JAX, but overall it's pretty close. If you put them side by side, you'll be able to see where the commonalities are. So you just look at the Python class. Let's say it's defining the vision part of the model. You just take that code and look at the MLX syntax and you should be able to convert it pretty easily.

33:37There are a lot of foot guns because MLX does things differently from PyTorch. You have to have experience with that, but it's pretty easy because we already have a lot of examples. So if you see a particular component on, let's say, Command-R and then they release Command-A. If you go to Command-A and you see that it has a similar class that already exists in MLX, you can just copy that in and you should be okay. I'm talking in terms of like beginners. Advanced people, I would just say spend time in the MLX documentation, spend time looking at the code that already exists, spend time looking at the PRs of adding support for new models, especially the family of models that you want to add.

34:22And then you'll be able to see the pattern and you'll be able to see what you need to do. In very rare cases, you will need to pretty much write something from scratch. and those are the hardest ones. So with MLX Audio, for example, I had to write a lot of layers from scratch that just don't exist in MLX right now. Then I sent some PRs and some people helped me send some PRs to MLX to add those layers in, but those are very rare cases. It only happened to me because audio, it wasn't really as front and center for MLX initially, but now it's kind of taking up that space and they are more aware that audio is kind of taking, growing as well as video.

35:12So usually I just look at the config file, identify whether it exists or not. Second step, I just implement it if it doesn't exist. And after that, I just make a new release and the models are on MLX. After that, after the release is done, I usually upload all of the model weights to MLX in particular setups. So the first one is quantized to 3 bits, quantized to 4 bits, quantized to 5 bits, 6, 8, and BFloat16, which is usually half precision from the original models. Some models are float32. I just do half of that. but we still keep the same quality and speed compared to the original. And people can just choose.

36:01So if you have a smaller machine, let's say an M1, M1 Pro, you can go for the 3-bit, 4-bit models, or even 5, depending on your RAM configuration. And then if you have more performance, you have a kind of middle range MacBook to high range, you can go for the 8-bit, 6.8, and BFloat 16. Are you just like programmatically producing the various quantized versions of the models and uploading them? Or are you doing like testing to determine like where that performance quality sweet spot is? And okay, for this one, you know, four bit works, but you know, two bit doesn't work, you know, for this one, we can get it down to two bit.

36:48But like, or is it like you just do it and it's up to the user to determine what works for them and where they want to make that tradeoff? Wow, Sam, the questions are really good.

37:03So initially I used to do this by hand. I would do one by one manually and test them. but nowadays I have to port an average of like two to three models a day sometimes and that is just and sometimes even more depends like in the morning could be open AI and afternoon could be anthropic and evening could be mistrial so I never really know who's releasing so what I do I I test out the 4-bit and 3-bit. Those are the two that I see the results. And then the rest I don't see the results because I expect the performance to be good. I expect the performance of those ones to be good out of the box. But the 3-bit, some models just are too small or too fragile or sensitive to quantization that 3-bit will not give you any good outputs.

38:00So I watch out for those. 4-bit is usually standard. It just works. Might not give you the best performance when it comes to, for example, vision language models. But it works pretty well for text and audio. Because even when you think about audio, especially the more recent audio models, there are language models. So the tokens they are outputting are, you could imagine them as just normal text tokens. So they are a bit more resilient to quantization. But vision models, they are quite sensitive. So three bits, you have to see how big the model is. If the model, let's say, is one billion parameter, probably doesn't make sense to make a three-bit version of that model because it's too small.

38:48You're compressing the model too much. Yes. So I usually just test out those two and then the rest I upload and I wait for community feedback. Right now on Hugging Face, I have, let me just quote this correctly. On Hugging Face, I have over 10 ,000 notifications. These are people that want you to port their model to MLX? Sorry, correction. Not 10 ,000. 1 ,700. Sorry, not 10 ,000, but 1 ,700. um they these are people giving me feedback on particular models asking for ports of new models and usually these also end up on github um and i just i just have lots of notifications i think 10 000 is because i also have email and in overall i was counting this this week i have over 10 000 notifications across all platforms.

39:48So, yeah. I'm imagining you building like some LLM based thing to like go through all these notifications. I talk about that a lot. I actually released a video recently talking about cloud code because my expectation, I had this dream I actually posted. I want a system that could be a great

40:14intern for me because I want someone to check out the PRs. I want someone to see the GitHub issues across all my projects and just notify me if there's anything. I already have a body of work so the model can just see whether the new PR has the standard that I want or just give me an initial breakdown of what is there because it's like too many things to manage. But we're not there yet. It's still a long way to go before we get to an agent that you can reliably say, here are the keys to my life. Go wild. Plus, I really, especially with open source, I want people to know that I care, know that I am there.

40:57I'm reading every single notification. I just usually keep them as unread because it's easier to go back to some of these notifications. On Hugging Face, I don't check because it's too much. Like there's a lot of things going on. GitHub, for sure, it's me. It's always me. I'm there all the time. You've talked about quantization so far. You know, pruning you mentioned as well. Like, is that an extraordinary thing that you do, a regular thing that you do? Like when do you incorporate pruning into the process of porting a model over? Okay, so I didn't mention this. So with quantization, there are a couple of new techniques, especially the ones developed by Anhui and the MLX team.

41:44One is called DWQ and AWQ. AWQ, everyone already knows, which is activation-aware quantization. Basically, at inference time, you figure out which neurons fire the best and you quantize the rest. And then DWQ is more of a quantize the model down and then use a higher precision model, either an 8-bit, for example, if you quantize the model to 4-bit, or you use the full precision model and you do this sort of distillation. So basically, you're taking the model, quantizing it, and recovering some of that performance through distillation. I am quite familiar with this. at RC, we worked on the SteelKit.

42:30It was one of the things that we were pretty much the best at doing. And we created this toolkit where people can do it as well. That technique is pretty good. Right now, I just don't do it because it requires a lot of compute. I'm waiting on the M3 Ultra that the community got me to be able to do that. So once the M3 Ultra gets here, I'll be doing more of these specialized fine-tuning. of quantization models or quantized models. Additionally, with pruning, I think it also falls into this bucket. It requires some compute to recover the performance of the original model, which means you have to take some data set and train the model again a little bit or have a LoRa to be able to recover that performance.

43:17Because what you're doing at the end of the day is cutting the model dry. You're just going in and say, hey, you are out, you're out, you're out, you're out. even if you have some sort of metric that tells you which neurons to prune which is something that i do i created already the algorithm to do this you still need some training usually you would need an a100 cluster of gpus to do this and i i'm just waiting to get a more powerful machine to be able to test out some of these new techniques that can improve the performance of quantized models. But I'll give a shout out to actually DeepMind. They have been creating a lot of quantized aware models that they train already beforehand towards being optimized to 4-bit or even lower quantizations.

44:09So if more companies did that, I think that would actually save me the time and save a lot of people from the community at the time. So thus far, we've primarily been talking about models that other people create and you help port over to MLX. But in the intro, and you've mentioned MLX Audio, I believe, there's also MLX Embeddings, MLX VLM. Like these are models that you've created. So MLX audio and embeddings and MLX VLM are mostly packages for doing inference, fine tuning of these models and other optimizations there as well. Now, we are actually working on a model. Before we get to that model then, so these are all similar to when you mentioned audio, like MLX is the foundation, but in order to support audio models, there's a core set of components that are required.

45:12And that is MLX audio. You've like built out this kind of audio adaptation layer or, you know, you tell me what you call it, but that's the way I'm thinking of it. And think of it as a package or Python library, and even Swift, we also have a Swift package that allows you to run inference of all the models that we port. So just like MLX LM, when we see a new model in audio, we port it there, and we provide you the way to do inference. And this can sit on top of your applications as your local inference engine. We have servers, so you can stash a server that runs in your MacBook and serve it to your network.

45:53So you'll be able to generate audio or process audio on the fly in your network or use the Swift implementation where you can build an application with MLX audio components built within. So you'll be able to generate audio, do audio understanding and more. So the way to think about this is I've got a model that's a file that's a bunch of weights that's structured in a particular way. MLX knows how to take those weights and like do the math that require that's required to constitute a model but then you still have this problem of efficiently invoking or using MLX to do an actual inference and that's what these libraries are you know for audio and embeddings and VLM.

46:44Exactly. So it's quite interesting. I ended up creating more libraries that I wanted to. MLXVLM now is basically an Omni package. So we do inference of more than vision language models. We now do inference of models that support audio as well. So I wish all of this could be like one package, but unfortunately, just for the sake of my sanity, I needed to separate them and be able to think about them individually. But maybe MLXVLM comes to supersede all these other models and becomes like a single package for MLX inference? I think this is going that way, especially with the event of what I call media models.

47:33So models nowadays not only support text or vision as input, they support audio. That is something we support at the moment. But soon we will support, also we support video. We could consider that as well. But soon we will support even more types of modalities. And eventually I see all of that kind of falling into MLX VLM. Then I would have the problem of renaming it to something. Either MLX O, because Omni is taken, so I have to think about a new name. But eventually I think that is the direction the industry is going overall. I'll talk more about that later, but right now, MLX Audio is focused only on audio models.

48:17So if you want to do text-to-speech, speech-to-text, and speech-to-speech, you can use MLX Audio. A great example of this, I don't know if you heard of Unmute from Qtai. We have a similar pipeline that we released to open source weeks prior to their launch, which basically what our pipeline does, it's called a modular speech-to-speech pipeline. It allows you to take any language model or vision language model and turn that into a speech-to-speech conversation. So you can turn that model into a chat GPT with voice, for example. So you'll be able to speak to it and have it speak back to you. And you can use whatever language model or vision language model you are most interested in.

49:06interesting and so that like how is that working i got the impression that in order to have a you know let's say useful experience for speech to speech like you have to very tightly optimize that loop um and so like being able to like to to say that you have like an abstracted one that you can just plug in any model, like, that sounds hard. No, it actually works. So the beauty of it is the entire pipeline runs on MLX. So all of the models. So the speech-to-text model is MLX model. So right now we support Whisper and Parakit, which are the top automatic speech recognition models. Parakit's the NVIDIA model.

49:52Yeah, and we also support Gemma 3N through MLX VLM. So geometry N also supports audio, vision, and text as input. So you could also count that as one model there. Then we have the language model component. You can take any language model or vision language model supported by MLX and use it there. So you can use a model quantized to 3-bit, 4-bit, whatever quantization you want, and whatever model we support. And then we have the text-to-speech part, which is also on MLX, and it was how MLX Audio was created. So you can choose Cockro, which is an 82 million parameter model. It's a tiny, tiny, tiny model.

50:37You can use all of the existing models we have, including Sesame and etc. And our upcoming model that is going to bridge that gap. because right now, all of the existing models don't do streaming of responses. And if you were to stream the audio back, it will be super choppy. We are solving that and our model is able to do pretty much real-time on M3 Max. Full precision. So this is full precision. If you quantize it down, we should be able to do real-time or close to real-time even on an M1 Pro and potentially M1. And this is what the new model you're working on? Yes, it's called Marvis. So it should be out.

51:24Yes, my really awesome real-time intelligence system. As opposed to just another. As opposed to Jarvis. So this is where we are super excited. And I will share more details about this later today. Okay. Okay. Later today in our conversation or later today on Twitter? In our conversation as well as on Twitter. Let's do it. Share more details. Tell us about Marvis. With Marvis, we wanted to solve the problem that we also had with the modular speech pipeline. It's pretty fast, but if you want quality and you want to be able to stream, let's say you have a really large text that was outputted by your language model.

52:11So with previous models, you'd have to divide characters every hundred, probably characters or every hundred words and be able to create audio for those individual parts or paragraphs. And that is not good because the audio will be super choppy. One. Second, you'll have interruptions that are unnecessary so the audio will not feel natural. you will see, for example, you might have to have a break mid-sentence because if you generate any longer than that, it's going to take a long time for the user to hear the feedback. We found out about a particular decoder that is able to do streaming. So we have around 80 milliseconds of response time for the first audio.

53:02And I was like, okay, this is great. But there's no model within the category that we are aiming for, which is just a few hundred million parameters or even just a few million parameters. So we set out to build this model. And we have like I'm very happy to to announce that we we succeeded in that. We have our first model. It is about 250 million parameters and it can run real time on Apple Silicon. and you will be able to pretty much hear the voice as it comes with no breaks or even like lag detected. Talk to me about like what the model is. Is it a port of some other model? Is it a post-trained version of an open model?

53:47Is it a ground up model? So we built this model from scratch. We are training it from scratch. It is based on an existing model, which is called Sesame. We kind of were inspired by the data they shared, which is not a lot. We only have the inference code and some blog articles to go from, but we managed to reverse engineer most of it to be able to have a great, great model. And we trained it from scratch using a few NVIDIA cards. Initially, it was a cluster, so I burned$700 initially training the first version. It turns out we needed to optimize a few things. We optimized those further, and now we are training using a single GPU card.

54:39And we just finished training of the first version. Now we are training an even smaller version of the previous one so that people with even less capable machines will be able to run Marvis close to real-time. So it's basically, the architecture is basically a language model. It's a language model that gets your text in and outputs audio tokens. And those audio tokens are decoded into audio at the high level. But we will have the data, we will have the article, the code, everything will be open sourced. um you know what are some of the things that you're seeing folks do with uh kind of these speech-to-speech models primarily like audio assistant types of experiences or have you seen anything that really blew your mind to answer that question i need to go back a little bit in time uh late last year i was making a demo for data science summit i think the shirt i'm wearing right now.

55:45I've been speaking there for pretty much the past three years. And I was thinking about an audio solution because I had one problem. I created a computer use system using VLM. So I had an MLX VLM model, analyze my screen and be able to click stuff and achieve a certain tasks based on my input. But every single time I had to start a task, I had to walk to the computer. And that became annoying. And I said, why do I have a computer agent, but I have to sit in front of the computer and have to watch it do everything I told it to do? I don't want that. If it's an agent, it's supposed to be kind of in the background doing this, and I should be able to command it in any way, shape or form.

56:37And on top of that, I think the major reason why I created MLX Audio is because my father is actually blind. And he became blind in the past five years. And he loves to read. And I told him, hey, I will help you read. I think in 2020, I promised him that I'll help him read and navigate the world again. And it's part of the reason why I created MLXVLM to help him see. So he has an iPhone. He'll be able to like navigate the world and be able to detect what's in front of him. I actually made some glasses for him, but those are just not Ray-Ban quality. They would be super bulky for him. On a hackathon, I made some glasses that would help him see the world, but it didn't work out.

57:21I just saw there's a group that's doing like an open platform glasses. Have you seen that? But, well, if I get my hands on those, my dad will be able to see the world again. So I promised him I would help him see the world. And ever since, I've just been focused on vision and natural language. And now with audio, the task was he loves to read books, but he can't really read physical books anymore. And not every book is audible. So I told him, OK, I'm going to generate the audio of the books that you want. So he would give me some books that he wants. I would generate the audio and send it to him.

57:57He could listen to those books using MLX audio. And that is like part of the reason. But then going back to the computer use case, I thought, well, what if I could just sit on the sofa the way I'm seated right now? And I could just say, hey, Marvis, go ahead and order me this or find me this or send an email to Sam saying, hey, this was amazing. that should trigger the agent the agent should go on and do the task and just give me feedback as it's going now i open chrome oh now i'm actually writing the email i wrote this to sam what do you think like i want that that level of interactivity so i set up to build mlx audio to solve that problem and i made a demo i think on the first month that i released mlx audio doing exactly that.

58:51So we now have computer agents that run locally that can be able to speak, which is great. So with Marvis, this is going to go to the next level, which is actual real-time voice agents that can have a full-blown natural conversation with you. And it should be close to a conversation you would have with someone that's next to you. And it can also clone voices so you can use the the voice of a loved one, even if you want to. So that could be really cool. For my dad, he hasn't seen me in close to 10 years, so I could make his voice agents sound exactly like me. And I already have that running locally on my machine.

59:34Oh, wow. Wow. Very cool. I know you did a video recently on Claude Code. I think you mentioned it in our conversation also. So that would be a great use of a speech-to-speech agent. So you can like code your way through a Cloud Code project as opposed to like using, you know, Whisperflow or something like that to paste into the thing. Absolutely. It could be even built in Cloud Code. I just don't have the bandwidth. Like I have so many ideas, but I just don't have the bandwidth to do. If I were to give into my cravings to create, that would create a million things. That is one use case right there that you can build in.

1:00:20Cloud code could be able to speak all the thoughts and everything. Because I don't know about you, but personally, when I am coding with a model, I'm actually more interested in its logic than the answer. I find myself looking at the thought process, opening that tag like thought for three minutes, click. Let me see what it thought. Ah, this is wrong. This thought is wrong. then I can already deduce whether the answer is correct or not. So if it could just speak through, now I'm thinking about this or just, I don't know, find a way to find the most interesting things or summarize the thought process and just speak out.

1:01:01I think that would make collaboration much, much easier. Well, especially if you can like reduce the latency of like, you know being able to like just have very casually interrupt it and say well i don't know about that maybe think about it like this and then have it uh you know continue like you can do all that with keyboard but it's not fluid like you're working with the intern that's sitting next to you or you're pair programming with this with this machine you should try our modular speech-to-speech pipeline it does exactly that like one of the things i wanted is the ability to interrupt it so I could interrupt it at any time and just say whatever I wanted to do and it will do.

1:01:44Now the question is I want to be able to empower developers and create the best SDK so that people can just integrate voice into their products and pretty much all of their local solutions be it open source or enterprise if they want to. And this is what I'm working on and Marv is a step towards that. So is that the next big thing in your world, Marvis? Or do you have big plans beyond that about how it gets integrated into other things? I have big plans about how it integrates with everything else. But I'm the type of person that is focused on what I can offer and then going step by step. I already met, like I have probably the next three to five years of Marvis in my mind.

1:02:41But I think the most important thing right now is getting it out of the door, getting people to use it. And then we'll be able to do really, really cool things. We already, I don't want to say too much, but we already have really great examples of what Marvis could do. You should be able to, for example, manipulate Marvis to have an actual agent that speaks the way you want it to. whether you want a Jarvis more relaxed way or you wanted a more active way or I could just take all of your audience say hey sound like Sam in terms of cadence prosody everything just have a an agent that sounds exactly like you I think that is important for me in particular I think I would love that I haven't seen my family in a long time and I think having agents with some of some of my loved ones voices would be pretty cool or they them having me which would be nice then i would annoy them everywhere that's funny awesome so we will uh keep our eyes peeled for uh for marvis updates and um get a link from you as to where we can go to to look for that of course it'll be on x as well.

1:03:56Very cool. And tell me what else, like, you know, beyond the things that you're working on personally, like, what are you most excited about in the world of AI? I think the most interesting thing right now is the direction that we are taking. Initially, it was all AGI, AGI, AGI. But now people are realizing that, yes, that is great. It would be a great world. But I want a system that is capable enough or competent enough to have an actual intellectual discussion with me and tell me when I'm wrong in a very reliable way. I can trust it. Like, I want a system that I can trust. And I feel like we are moving towards systems that we somewhat can trust.

1:04:46There's so much backlash to, you're absolutely right. His came up in an interview I just did. People are realizing that actually we want something that's going to challenge us and like help, you know, work with us to like produce better things or, you know, think better about the thing that we're asking about. Exactly. I have a tweet where I say, I said 22 hours ago, I'm waiting for the day it will say you are wrong. Here's why. Delivers facts with sources. I want that. You're absolutely wrong. I think I want that. I don't want a system that is just like, oh, you're right. Or, oh, it's my bad.

1:05:32My bad what? You're actually right majority of the time. You just have this tiny bug here. And I want you to just fix that one thing. So that is one direction that I'm loving to see companies take on. So they are trying to build new architectures and new training strategies to be able to address that. Second thing is media models. I think over time, the same way last year when I created MLX VLM, there was literally very few vision language models, but nowadays vision language models and including audio models can do function calling. So you can do function calling with your voice, which enables experiences like Marvis.

1:06:14You'll be able to do way more with new types of media. So I'm seeing a lot of interest in audio space. I'm seeing a lot of interest in the video and lip syncing space, as well as like generation of images. I think going towards a world where newer models can do all of those modalities in one, and it can do in a very reliable way and efficient in the sense that you could get a 1 billion parameter model or 100 million parameter model that could do a few modalities and you can embed these directly into your products and make your life simpler. To kind of illustrate this, before Gemma 3N, the modular speech-to-speech pipeline, you needed ASR and you needed a language model.

1:07:11Now with Gemma 3N, you could replace the first component and the second component into one. And actually third component, which is multimodality with vision. So you now have one model that does audio understanding, does text and image. Now imagine if you could also do other types of processing all in one or even output audio as well. So now you can simplify your pipeline and your entire system can just run much faster because you're not loading all of these individual components and try to mesh them together. So I see media models becoming something pretty great. And I intend to pioneer that, at least when it comes to MLX, I really want to have people do this in the best way.

1:08:00To add to this, I actually have a challenge with someone Julian challenged me in Germany last month to create a package that's able to generate images and video in under 20 seconds with a few gigabytes. And I'm taking on that challenge. So look up to that as well. And the goal there is something along the lines of Sora or Veo? Yeah. Okay. But it runs completely offline. Completely locally? Yes, and locally. So I really want to do that. Yeah, I want to do that. I think the world is going to benefit a lot from it. And who knows? Perhaps the whole equipment here for the audio, because it's for the studio, might be replaced.

1:08:51In some videos, I might just take a copy of me and make a generated video and say, hey, guys, at least I thought it through. I made the script and you guys can watch something pretty new. That's awesome. Awesome. Yeah. Well, Prince, it was great chatting with you. Thanks so much for jumping on and talking a little bit about what you've been up to and how the MLX ecosystem is evolving. Yeah, absolutely. Thank you very much for having me. Thank you.

1:09:30Goodbye. Goodbye.