Unlocking the Future: Meta's MusicGen, AudioGen, and EnCodec4 Go Open Source

0:00Meta has announced that they are open sourcing and launching three new AI audio plays and those are called Music Gen audio gen and of course are updating their codec. So on the podcast today, we're going to be diving into what these three new AI models do and why it's so important that these are open source. So let's jump into it. The first one that they unveiled is called AudioCraft, which essentially is a framework designed to produce high quality, realistic audio and music. And they do this, of course, just with short text prompts. We've seen similar tools out of Google, but what I love that Meta is doing here that I always give them credit for is the fact that they're open sourcing this project.

0:41Anyone can use it. And of course, you know, we'll see a cool tool like this out of Google. And the assumption is, you know, you can get it if you have some sort of subscription to Google in the future. That's kind of where Google goes with these tools. And so this is really cool seeing this out of Meta where this is open source and they're allowing people to take it. And they're kind of releasing the model weights and everything that goes around. So this isn't Meta's first step into the whole sound generation. The company actually had previously made a couple steps in this field by open sourcing an AI music creator music gen back in June.

1:13But this time, Meta, you know, says that its newest progress significantly enhances the quality of AI created sounds. So whether that's a dog bark, the blaring of a car horn, or the sound of footsteps echoing on wooden floor. So in a detailed blog post, Meta kind of outlined how Audiocraft's frameworks has conceptualized and how it was created to really streamline the application of generative models for audio. And they talked about the comparison between this and their earlier efforts in the field. There was Refusion, Dance Fusion, and then OpenAI obviously had Jukebox. so now this open source audio craft is a really comprehensive package which i believe offers a different kind of suite of sounds and music generators along with a compression algorithm and you know simple which really helps to simplify the creation and the whole encoding process without you needing to hop between different code bases which is a really big lifesaver so these three models which are audio craft um and or i guess audio craft is kind of the platform so the three models on the platform are music gen audio gen and in codec um and we've seen music gen before so this time meta has announced um or i guess it's kind of launched its training code which is allowing users to actually train models with their own music data sets so this is kind of big right before they just said hey here's this tool you can use it we've trained it um but now we actually get the model, which is amazing, right?

2:46If someone has a big data set of maybe their own music or other music that they have rights to, they can train, they can use that to train, which is really, really cool. Also, inevitably, you can assume that, you know, someone's going to go and take the entire, you know, the entire library of Johnny Cash and throw it in there and train it to be a really good Johnny Cash music creator. But that's already something that's on the internet is all the bootleg AI generated music, which to be honest, I've listened to and thought was pretty good. So, you know, take that with a grain of salt, but it's going to be interesting.

3:18This is open source. It's open on the internet. Anyone can use it now. And yeah, some people will complain that, oh, it's getting used on copyrighted music. Um, but I think at the end of the day, this is a really great tool. Uh, I think copyright is obviously going to apply if you're, you know, putting this thing on YouTube. I wonder if YouTube's going to put a content strike against to you for AI generating music in the style of an artist. I mean, I think that's fair if they do. But at the end of the day, I think this is a really big win for developers and for any kind of music enthusiasts. There's so many different use cases for this.

3:48So I think this is something great to have open source and in the public. Something really interesting that Meta said this time around is that they've actually specifically clarified that the pre-trained version of MusicGen was trained using, quote, Meta-owned and specifically licensed music. So they said that there is over 20 ,000 hours of audio, which is amounting to around 400 ,000 recordings. And they were, you know, they also had text descriptions and metadata in there, right? That's that's kind of the genius of this is that you upload the recordings. And then you have, of course, you know, if it's a song, you have the lyrics to the song.

4:24And that's how and also metadata on the style of the song and a bunch of data about it. And so that's how this thing is able to you know be trained to to replicate stuff but i think it's really important right they they had the licenses to this music no one can complain that you know they scraped the internet and grabbed you know johnny cash and every other it's kind of different here too because when you see someone like open ai not saying that this is right or wrong either but when they scrape the entire internet you know they're grabbing the responses and text of of you know broad users that a lot of them weren't professionals maybe they just had hobby blogs or they're just talking about you know x y and z or comments on reddit and stuff and uh i think it's a little bit better an easier argument for them to make that no no you know this isn't like copyrighted material we're just grabbing what some people have said and training our data off of this for meta if they were going to try to like train an ai model um off of music every single song is obviously copyrighted and so it's kind of just like a whole new ballpark that i think is a lot trickier so they definitely need to make sure that they're using you know something that they have licenses to so the data was sourced from meta's own meta music initiative sound collection so shutterstock's music library and pond 5 were also included in that but you know a really prominent stock media library um that they and a collection that they have there so i think that you know it's obviously given them a ton of data meta also took steps to remove vocals from the training data in order to prevent the model from reproducing artists' voices, right?

5:56So that's, again, something I think people will appreciate. However, the terms of use of MusicGen really discourage out-of-the-scope usage beyond research, so MetaStop's sort of outright, you know, prohibiting any commercial applications, but that's really not something that they're encouraging at this time. Maybe that will change in the future, but at the moment they're really, you know, saying they just want researchers to use this and maybe that's just for liability reasons. The second model which is in the AudioCraft platform is AudioGen. So this kind of focuses on the generation of environmental sounds and sound effects rather than music and you know melodies.

6:35So this kind of shares traits with modern image generators like AI's Dolly 2 where or you know Google's Imagine and Stable Diffusion. so really audiogen operates on a diffusion based model so it's designed to learn how to kind of progressively erase noise from starting data which could also be audio or images and get it kind of close to the target prompt and it does this step by step so kind of set up with a text description of an acoustic scene you know if you if you gave it that audiogen has a capability to generate environmental sounds with realistic recording conditions and also complex scene content so that's what meta has said and uh as yet there hasn't been an opportunity to put audiogen to through its paces i've been able to try this out yet um ahead of its release but a white paper published alongside audiogen reveals that it can also generate speech from prompts in addition to music which is really interesting um which is kind of you know it can do this mirroring the diversity of its training data essentially so the white paper for this also says that audio craft could potentially be misused uh to deep fake a person's voice so um that also gives us some hints about its capabilities but you can kind of add to this audio crafts music generation capabilities and you know if you if you put the two together right um audio craft and the ability you know essentially on the platform with music gen and audio craft you could really create full-on songs i'm assuming so just like music gen metadata metadata also doesn't really impose stringent restrictions on how audio craft and its trading code can be used so you know whether that's for the better or worse the final model of the audio craft platform is going to be in codec which represents an upgrade over a predecessing meta model so this is designed to generate music with fewer artifacts and meta's assertion that encodec um is more if is essentially more efficiently models audio sequences um and that it can it's good at capturing different layers of information in the trading data's audio waveforms to essentially assist in crafting unique audio so encodec is described as sort of a lossy neural codec trained to compress any form of audio and reproduce the original signal with high fidelity and the you know the different streams capture varying levels of information from the audio waveform so this kind of enables the audio to be reconstructed with high fidelity from all streams really really interesting what they have going on so i think whether it comes to gauging the potential impact of audio craft meta naturally chooses you know the high to essentially just highlight the positive possibilities they envision a tool that can inspire musicians and assist individuals in iterating their compositions in new and exciting ways.

9:32But we've also seen, you know, with image and text generators, every new kind of technology brings with it its own kind of share pitfalls, legal complexities, and all of the controversies. So it'll be very interesting as tools like this get bigger and bigger and are bigger in the industry. It'll be really interesting to see how that brings, you know, those controversies into the light with audio specifically. So So undeterred by those challenges that I think we may see in the future, Meta plans to essentially continue probing in ways to enhance the controllability and performance of generative audio models, while also working to kind of cut back on the limitations that we're seeing from them.

10:17So on the topic of biases, I think it's worth noting that MusicGen currently, when dealing with descriptions and language other than English it does not have a lot and you know music styles and cultures outside of kind of the western fear a sphere I don't think it supports them at the time so it'll be interesting to see if if they bring those in but Meta in their blog post also noted quote though the development of more advanced through the development of more advanced controls we hope that such models can become useful to both music amateurs and professionals. So I think this is going to be a really interesting area to follow in the future.

10:56Obviously, this is a really powerful tool. And I think we're going to see some pretty solid adoption of this in the future. First off, probably from companies implementing this into platforms and then from users that will be creating all sorts of music and interesting audio effects based off of these new AI tools.