#188 Edward Balassanian: The Evolution of AI Music

0:00I think the music coming out of Amy is producer grade and it's generative music so I think we're already there. I think the more important question isn't, are we going to be able to distinguish AI music from human music? I think the more important question is, are we using AI as a tool to accelerate and amplify human creativity or not? And I think the answer is resoundingly yes, I think we are using it to amplify human creativity. AI might be the most important new computer technology ever. It's storming every industry and literally billions of dollars are being invested. So buckle up. The problem is that AI needs a lot of speed and processing power.

0:39So how do you compete without costs spiraling out of control? It's time to upgrade to the next generation of the cloud, Oracle Cloud Infrastructure, or OCI. OCI is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds, offers one consistent price instead of variable regional pricing, and of course, nobody does data better than Oracle. So now you can train your AI models at twice the speed and less than half the cost of other clouds. If you want to do more and spend less, like Uber, 8x8, and Databricks Mosaic, take a free test drive of OCI at oracle.com slash I on AI.

1:32That's E-Y-E-O-N-A-I, all run together. Oracle.com slash I on AI. That's Oracle.com slash I on AI. Hi, my name is Craig Smith, and this is I on AI. In this episode, I speak with Edward Balasanian, founder and CEO of AIMI, A-I-M-I, a generative music company that aims to transform music into a service. There are a lot of these generative music startups on the market today, but AIMI is different. Edward shares his vision for enabling an ecosystem of applications through AIMI's platforms, which combines various AI models in a proprietary scripting language to create high-quality, precision music.

2:24We discuss the challenges of AI adoption in the music industry, Amy's approach to building their platform, and the future of AI-generated music. Edward also demonstrates Amy's music creation process and explains how the company plans to empower creators and enterprises alike. I hope you find the conversation as fascinating as I did. Why don't you introduce yourself and give your background, your education background, if it's relevant in your work history. Sure. And then we'll get into AI music. Terrific. Well, my name is Edward Balasanian. I'm the CEO and founder of AIME. AIME is a generative music company.

3:14The name actually stands for the AI Music Initiative. We founded about four years ago, and our goal then and now has been to provide generative music as a service. In other words, to transform music into something that can be thought of as a service rather than just an artifact. My background is engineering, computer science degree, worked at Microsoft for five years and have been starting companies since then. I've always been sort of a platform guy. I love building platforms that enable an ecosystem of applications to succeed and thrive. And that's exactly what we're trying to do at AIME, is to turn generative music into a service that can be incorporated into any application, product, or service.

4:01Yeah, that's fascinating. I've spoken to a few people on the podcast about music generation. uh it's the market for me because uh i tend to look at at uh basic research more than uh specific startups or applications it i see a lot of activity and maybe you can help disambiguate that a little bit i know there's uh like google magenta ophonic like AudioCraft by Meta, Stability AI has Stability Audio. You know, I think Soundful's doing something, Hook Sounds, AI Studio. What's going on in that sector? Is everyone doing kind of the same thing with slight variations because it's a new market and there will eventually be consolidation?

5:07Or is everyone doing something very different? Well, I think music is one of the last mediums to really be fully engulfed or consumed by AI. And the reason for that is multifold. First of all, music uniquely represents legal challenges that the other creative mediums like photo, text, and even video don't have. There's an international body of incumbents that vigorously protect the status quo that will vigorously enforce copyright. And they also vigorously defend the business model of pay-per-listen. So it's difficult to create solutions in companies that succeed in the tech space where you have to line your business model around pay-per-listen.

6:00But I think the other The reason you're seeing so much momentum now is as machine learning has evolved and matured, music is a very dense medium. Not only is it very dense compared to something like text, but it's multidimensional as well. You know, it's something where you've got layers of instruments and then you've got a temporal aspect to it as well. And both of those need to be coherent. You need stylistic cohesion. You need musical cohesion. And you also need the mathematical cohesion, too. It needs to be correct from a music theory perspective. So it's been late in terms of AI adoption in the music space.

6:47Having said that, what you are seeing now is a lot of training of monolithic models on large bodies of music. And some of those models are trained on licensed content where the content was licensed specifically for training the model, and a lot of them have not been. And I think you'll see a lot of training in secret and a hope that we can kind of get away with it, if you will, building in mechanisms so that you can't identify the source inputs to these models. that uh that's i think where we're at now where a lot of these models are essentially trained on finished music our perspective is very different on that we just don't think that that kind of model training is going to be effective for truly creative uh for for true creation and creative expression in the space of music because music is made from the bottom up it's not made from the top down.

7:51And the best music making is done from the bottom up. When producers make music, they essentially mix and arrange audio artifacts and they sequence these in time to tell a story. That's a big part of music making. Just like if you were a chef and you wanted to make a soup, you can't just go up and taste a bowl of soup and try to recreate the recipe. You need the recipe. You need to know what the ingredients are. So yeah, our approach is very different, but I think you're seeing a lot of sort of sameness around the training of these monolithic neural nets right now. And this is using transformer-based models?

8:34It seems like a natural for transformer-based models since they're predicting the next token music is note-based. Yeah, I mean, what you're seeing is essentially these large language models that were designed for text being applied to music by essentially turning music into images that then get turned into text. So you're able to essentially transform the audio, which is very dense, into a representation as text. And then you can use a large language model to essentially generate music that way. One of the challenges you run into with this is you lose a lot of the nuance in music. And you can really quickly hear AI-generated music because a lot of the nuance that goes into music making that you get from building it from the bottom up is lost when you try to replicate the finished signal by just listening to it.

9:34Yeah, that's interesting. Well, presumably you would generate individual tracks and then stock them, right? You wouldn't generate an orchestra. You would generate a violin and generate a cello and generate, you know. That's the way our system works. So we actually are combining audio artifacts, sonic artifacts that are the compositional elements that go into creating music. So we're mixing and arranging small audio artifacts that we are applying effects to, that we're sequencing in time. It's very different than listening to a finished signal and trying to replicate that sound, which is how the large language models are being trained.

10:27They're being trained on a song and then trying to replicate it. So they would listen to a symphony and try to replicate that sound. But as you just said, you're going to lose the nuance of the five violins versus the four cellos versus the piano versus the harp. That nuance is super important. And you lose it when you try to just listen and imitate the music that you're hearing. Yeah. I had the founder of Soundful on the podcast probably a year ago, and they had sampled individual notes on individual instruments. They built up this massive library of individual notes of individual instruments and trained a model on that.

11:18And then the model combines those individual notes and then the individual instruments into a composition. Are you guys doing something similar? Kind of. I mean, the goal is really important to understand here. Our goal is really to be a creator tool. And when I say a creative tool, we want professionals all the way to amateurs to be able to make music. And that means giving the professionals the kind of creative input they need into the composition of the music. One of the challenges you've got with large language models right now is if you prompt it, it's difficult to reason about the output based on the inputs that went into it.

12:05If you go to ChatGPT and you put in the same prompt three times, you're going to get three different answers. and it's difficult to have the kind of creative input into that that you want. And what we wanted to do is really provide a system that would give you that kind of creative input without watering, without making it too complicated for those who just want to say, I need a hip hop song that's two minutes and 35 seconds and I want it to have the last four seconds tail off in an outro. So we want to give you the kind of input that you need based on your skill level but not relegate you to watered down music just because you're a novice or a pro.

12:44Yeah. And are you in your training data? Is it individual notes or individual tracks? I mean, I haven't. I was reading recently about what is the unit of audio? And it's, I think it's called a sample. Is that right? Yeah. Yeah, we use a lot of different kinds of audio artifacts. Samples are some of them. And we actually believe that in a lot of cases, you want a sample to reflect human creation, like someone singing. We call those hero samples. You know, sounds that need and should be human because the human being created them with emotion and flaws, you know, imperfections. That's part of what makes music beautiful.

13:42So we do use a lot of audio samples. We also have generative instruments that can play along with other samples. And these are programmatic instruments. So there's a lot of different kinds of audio artifacts that we're using at runtime. And we have a programming language called AmyScript, which is essentially a programming language for generative music. That programming language is mixing, mastering, and arranging music in real time, like a producer would. And our models generate that script rather than kind of generate the music. They generate, say that again, they generate that. Yeah, so think of AmyScript as the way you describe a recipe, right?

14:24So instead of trying to train our model to make soup, our model creates recipes. And then we have a programming language that executes that recipe. So we don't have to train our models to create finished music. Instead, we train our models to generate these recipes and these recipes generate the music. It also means that we're able to execute and generate music orders of magnitude more efficiently in terms of CPU utilization and time. because we're executing highly optimized TypeScript code. That's the basis for AIM script. I see. And when you said that you convert sound into text and then the language model generates a string of text that then is decoded into sound, is that right?

15:21That's not what we do. That's kind of typically how large language models are being used to train monolithic neural nets. So you essentially, large language models are really good at text. I mean, that's what they're designed for, right? That's why they're called large language models. So you essentially, the exercise is to transform audio into text. And you can do that by turning it into these visual representations and then turning the visual representations into textual representations, and then training the model on the text. So what are you guys doing that's different from that? Well, we have seven, I think, seven different models, actually, that do all kinds of different pieces of the construction of music.

16:11We have models that understand audio artifacts, know what key they're in, know what instrument they're in, know whether it's a female or a male vocal, know what tempo it's at. We have other models that separate sounds so that we know that there's two instruments, what those two instruments are. So there's a lot of collaborative models that work together to provide the framework that our scripting language can then use to generate the music. We don't fundamentally believe that music AI is a one model problem. So we employ and deploy models as needed to support the infrastructure that we've created.

16:50Yeah. And on the samples, the voice samples, do you have people in a studio making those samples? Are you pulling the samples from catalogs? Or how do you get the voice samples? We've commissioned a lot of artists to create vocal samples for us. we've also purchased a lot of vocal samples and we also generatively create vocal samples so again this just goes back to us not trying to apply one model to the universe of music problems that we're trying to solve instead we're very opportunistic about the way we leverage technology including models for the system so you know for example using lyric generators and then using those lyrics to generate vocals works it doesn't sound as good as humans, but in some cases that's okay.

17:47So it's fine for us to have audio artifacts that include human-generated artifacts as well as machine-generated artifacts. Yeah. And when you said you're providing this as a service, what does that mean? That's a great question. So going back to something I said earlier, the music industry has really sort of relegated music to something that you consume and you pay for the consumption of it. So if you listen to a song, you pay to listen. If you play music in an auditorium, you have to pay for the number of potential ears that would be in that auditorium. So our business model is very different.

18:30We charge for music as a service, meaning just like if you use OpenAI, they're not trying to take a percentage of your book sales if you use Chai Tupiti to write a book. And in the same vein, we don't try to take a percentage of your audio sales or music sales if you use our service to generate music. We want you to be able to use the music however you want. If you want to score a video with it, if you want to upload your own sample and have Amy make a song for you around your sample that you put on Spotify, and you become the next biggest hit great we love that but you pay for the service the generation of the music not the artifact of the service yeah i see and uh okay what i mean i've been talking to another guy who's in stealth right now but he's building uh an ai uh music uh platform uh you You know, not to generate music, but to distribute music, which is an interesting idea.

19:38How big, I should say, who's your client base right now and how big do you think this market could be? Well, I actually just talked about this at South by Southwest. So, you know, we spent four years building this underlying platform that is the backbone of annual music services. And we built a consumer app last year that we released. We built it mostly to show off the power of the system. We're not in the direct-to-consumer business, but you can download the app and listen to free music on it. And the music's great. Now we're releasing Amy Music Services, which is the service that powers it, so anybody can incorporate generative music as a service.

20:22And one of the things I talked about was this market has been slow to form, and it's not because of a lack of opportunity. I mean, music cuts across so many different industries, gaming, social media, content publishing, video production. It's everywhere, automotive. The problem is that the business models have been stymied. I mean, you've been forced into this play-per-listen model, and it doesn't work for the TikToks or the Metas or the Canvas or the gaming companies of the world that need music integrated into their product. So our vision of this is that music as a service is a massive opportunity.

21:03And by not trying to compete with published content, we're not trying to replace Taylor Swift. Taylor Swift and Drake and The Weeknd are always going to have a place. People want that human connection. But sometimes you just want audio for your video that you're going to upload on TikTok. And you don't want it to be Taylor Swift. You want it to be something uniquely created for your video. And that's where I think generative music can really play a role and where music as a service really can fit in. Yeah. And as a service, you mean that creators can log. log, they subscribe to the service, and then they can generate X number of hours.

21:47I would imagine they're tiers. Is that what you mean by service, or is the service that... That's exactly what I mean. Okay. It's not that you plug in Amy's API, and you can... That too. We have both. Yeah, and on the API end, you can integrate different music that's on offer into your project. Or is the API just pulling the music creation interface into your system? Yeah, so the music services is offered in three different versions. One is the pro version, which is designed for creators. It's a website, log in, you have a conversation with Amy, and Amy will make you music. You can upload your own sample to it.

22:39It'll build music around your sample. And you can see the construction of the music. You can actually interact with that construction, or the composition, I should say, which allows you to have a lot of creative input. So this is designed for amateurs all the way to pros who want to make music and they don't want to spend hours or weeks inside of a digital audio workstation. Separately, the same service is available by API to enterprises. So the TikToks of the world can integrate this directly into their platform and seamlessly include generative music for their creators. So if you're uploading a video, you can score your video very precisely with music that's exactly the right length and that follows the arc that you want.

23:20And then last, we also have what we call live streams. And these are just internet audio streams that you can integrate directly into your product when you want zero configuration. You don't get the input into these that you would otherwise have with our API or our Pro Tool. But for a lot of our customers, which actually are our initial customers right now, these live streams give them a way to just integrate in continuous streams of music that's not repetitive. It's high quality and you can just leave it on. You can play it, you know, 24 hours, seven days a week and not get tired of it. I wanted to jump in and give a shout out to our sponsor this week.

24:00When it comes to ensuring your company has top-notch security practices, things can get complicated fast. Vanta automates compliance for SOC 2, ISO 27001, HIPAA, and more, saving you time and money. With Vanta, you can unify your security program management with a built-in risk register and reporting, and proactively manage security reviews with AI-powered security questionnaires. Over 7 ,000 global companies like Atlassian, FlowHealth, and Quora use Vanta to build trust and prove security in real time. Listeners get$1 ,000 off Vanta at vanta.com slash ionai. That's I-O-N-A-I, E-Y-E-O-N-A-I, all run together.

24:55And that's Vanta, V-A-N-T-A dot com slash ionai, E-Y-E-O-N-A-I. Give them a try and get$1 ,000 off Vanta. your security program management with built-in risk register and reporting. Yeah. What's a use case there? Who would be a typical customer for the live stream? So that is offered mostly right now where our customers are B2B2B. So we provide it to businesses that in turn provide it to retail, hospitality, fitness, restaurants. So for example, one of our customers has integrated these live streams into their server product which slides into a rack at your hotel and it distributes music throughout the hotel so it's transparent to the hotel that it's amy's music and for them it's very uh the the service that we offer to the company that is our customer is empowering to them because right now they have to buy catalogs pay a lot for them and And when they distribute music to European customers, the European customers have to actually pay additionally, based on the size of their venue, to the performing rights organizations.

26:16None of that exists with Amy. There's a one-time per month fee for the service. You can put it in a closet or you can put it in an auditorium. We don't care. The service is what you're paying for, not the consumption of the music. Yeah, that's fascinating. And my son's in the music business, and he's been talking to a startup that is buying catalogs from not major catalogs from small musicians. I'm not, I don't know the language, but I'm wondering, is AI music going to change that market because these smaller catalogs?

27:21Yeah, I don't know. How do you think it's going to change the market? Look, we've seen this with kind of every creative medium. Like when iPhones first came out, all of a sudden everyone was a photographer. and anybody could take a picture. And all of a sudden, the quantity of pictures was overwhelming. And that didn't change the fact that good pictures are good and bad pictures are bad. It's the same thing with music. We're building a tool that allows people to make high quality, high production and high precision music. By high precision, we mean it's what you want. It's not just something that you know, and that spit out for you.

28:00So we see... an acceleration of good music being created with tools like this. But there's also the fact that a lot of music spam is going to start coming out. I mean, you already saw it with one AI company that was spamming Spotify. And as a result, Spotify had to pull down their content because there was just so much of it. And the unfortunate thing is these tools are being used to create music that's below good. So it's fun to make the music, but no one's listening. so you end up with a lot of music spam and music is expensive like to store it to stream and it's expensive so that's going to be a challenge i think you're going to see dsbs really crack down on music like that but again we're we're not in that business we're in the business of creating high quality high precision music as a service would you be able to uh share on on the uh on the screen the platform and just walk us through a simple music creation okay so this is the pro version of amy music services and again this is designed for creators and i know the word creator is really overused now but i like applying it here because we don't want to alienate people who are not musicians who are not professional producers from being able to express themselves musically at the same time we don't want to alienate uh producers by giving them tools that are too simplistic for them to use so let's take a quick look here on the right what we call the plan this is essentially an articulation of the plan that amy has the recipe for the song and i can interact with it from here i can say let's let's pick a reggaeton song for example and it'll show me a plan.

29:45I can also ask it to do it on this side. This is more of a conversational interface on the left. So I'm going to actually use this here to say, let's add a build up between the intro and the verse. So we'll go ahead and add a build up in there. And I can interact with this. You can see here, I can see the whole plan. I can see all the different elements in each of the different sections. We're adding the ability to get into each of these and choose the instrument. you can also choose whether there's vocals or whether they're male or female what language they're in so we can give you a lot of introspection of it so you can see here there's a buildup so we have in the intro bass harmony rhythm i'm going to actually get rid of the rhythm in the intro and i'm going to get rid of the percussion in the buildup so we have kind of a natural progression from intro to verse and then you can see here uh the verse adds in melody harmony in counterpoint i'm going to get rid of the counterpoint here i'm going to add it in over here and again i don't have to do this if i don't know anything about music i don't have to muck with any of this stuff i can just let any kind of do its thing now i'm going to change all this to g minor so let's change the key from d minor to g minor in all sections So Amy will go through and now redo the key for each of these different sections.

31:13I can also adjust the number of seconds that this is. I can add new sections here manually. You can see here it switched to G minor. BPM match, there's some kind of interesting tensions between how long you want the song to be what the bpm is and how many beats per section you have and those all have to be juggled when you're saying you want the song to be a certain amount of time right now we had a photographer at south by southwest with us who needed music for a 24 second video clip that he had and he wanted the last four seconds the outro to be only four seconds so he just went in there and click four so that was four beats and the outro became four beats so you get that kind of precision over this so let's go ahead and hit produce here basically i did an indie pop song i did a i did a hip-hop song i did a same one twice i did a techno song as well and it's really easy to do this you can see here these are some of the genres that we support we're adding edm We're adding pop.

32:23Super simple for us to add new genres. We actually sit down with producers who are very steeped in the genre and we translate their expert techniques into expert algorithms. We then source audio artifacts that are consistent with that genre. And that's it. Within a few weeks, we've got a new genre that we can generate music in. And when we have these audio artifacts, we're actually able to expand the universe of sounds just by using our own generative AI to replicate those sounds. So we can go from a small number of audio artifacts to a large number very quickly and support a massive range of music for any one of these genres.

33:11Are serious sort of producers, pop music or hip hop producers using Amy to do background tracks or that sort of thing? well this just got released at south by southwest so we are in the process of getting this out into the hands of we've worked with close to 200 plus artists over the past couple years they are effectively our audit committee if you will and one of the things that was really important to us is not to release amy until the sound was good to great and they were the litmus test until they gave us the thumbs up we weren't ready to pronounce this as being ready for for consumption so So this is now in beta.

34:00We're providing this to our artists. As I said, they can also upload their own samples. And a lot of our artists are focused more on the creative elements in samples than they are in trying to make arrangements, which is very tedious and oftentimes very prescriptive. If you want to make a song for Spotify, there's a specific formula that you need to follow. And that's not the fun part. The fun part is coming up with the cool melody or the cool bass line and that amazing drum beat. Those are the fun parts, and you can add that audio to this and place it anywhere in here you want, and Amy will build a song around it for you.

34:42Are there – and this will – music generated by Amy will be uploaded to Spotify and the various music platforms. Is that right? Yeah, that's correct. So we're going to allow people to use this music for whatever they want. Now, there are a couple of constraints and restrictions. You cannot copyright the output of Amy. You also can't use the output of Amy to train in the machine learning models. So the output of this content is either for an individual to use in their own publishing, for example, on Spotify, or for incorporation into a video. If you're an enterprise, you can use this audio on your platform.