Making human music in an AI world

0:03Welcome to the Vergecast, the flagship podcast of music apps you play on your phone with your mouth. I'm your friend David Pierce and I am at Union Station in Washington, D.C. It's like 4.54 in the morning and I'm on my way to New York to go do some planning meetings with a bunch of other Verge folks. I've been at the Verge for like two and a half years now and there are a bunch of co-workers that I've just never met in person. It's a weird post-pandemic universe we live in, but I'm going to go meet some of them in person. This is the third and last episode in our mini-series about the future of music.

0:36We've talked about Trackstar and TikTok. We've talked about autotune. And today I'm talking with a Stanford professor named Gu Wang. Gu is fascinating. He's been in the music world for a long time, but he's also an academic. He's also a former entrepreneur who built a company called Smuel that you might have heard of. He's just been in the music world thinking about this stuff for a really long time. So I wanted to talk to him about AI and about virtual reality and about what it means to make music now in a world where increasingly technology mediates all parts of this process. We had a really fun conversation.

1:14It's not at all what I expected, but I really enjoyed it, and I suspect you will too. All that is coming up in just a sec, but literally this train is about to leave without me. This is the first cast. We'll be right back. So, your AI agents, they make the team that uses them more productive, right? But if they aren't connected to other agents, or your data, or your existing workflows, how productive can they really make your teams? Any business can add AI agents. IBM connects your agents across your company to change how you do business. Let's create smarter business, IBM.

1:56Welcome back. Let's get into my conversation with Gua Wang. I wanted to talk to Gah because he sits at a really unusual place in the music slash technology universe. A bunch of years ago, he co-founded a company called Smule, which is still around. You might have heard of it. They make a bunch of really popular music-based apps. One is a karaoke app called, I think, Smule that's still really popular. There's also one called Magic Piano, which is kind of like Guitar Hero or Beat Saber, but it's for playing piano. Gah also wrote a whole programming language called Chuck, which lets you write code that outputs as music.

2:33It's very cool and kind of wild to watch someone work with. He's now been at Stanford for a long time, also teaching in the Center for Computer Research in Music and Acoustics, which is a very cool thing that I just learned exists. He's also written a book about design. He's been teaching about music and technology for years. And in his free time, he's also the conductor of Stanford's Laptop Orchestra, which is exactly what it sounds like. It's a bunch of students on stage using their laptops and other gadgets to make music, frankly, that you've never heard before. Here is just a glimpse of what that sounds like.

3:15that sound if you can imagine it is being made by guh himself who is standing on stage holding a glove connected to his laptop and it's changing the sound as he moves his hand in space it's wild you should watch it their concerts are very cool i wanted to talk to guh about the future of music because I figured he would just see it all from an unusual perspective. He makes music. He teaches about music. He's a programmer and an engineer, both for fun and by training. He's started companies before. He's seen kind of all sides of this. And I figured he'd have cool ideas about how we make music, how we can interact with computers to make music, and probably would have some like neat new AI apps that he was into that we might talk about.

3:59Our conversation went really differently from that. And it went really differently in a way that I really love, actually. What we ended up talking about more than anything was what it means to be creative in a time when everything is being optimized and simplified and made more convenient and more efficient and more engaging absolutely everywhere you look. This is much more philosophical than a lot of the stuff we talk about in this show, but I really like the way he talks about what it means to be human and be yourself among all of this technology. And I hope you enjoy it too. So let's just dive in.

4:35To start with, I asked to try and frame the way he sees the world a little bit. One thing I've heard him say a few times in interviews and stuff is that he makes computer music. But it's 2024, right? Pretty much all music is made with and on computers. So what does Gu's version of computer music look like now? And how did he get so obsessed with it in the first place? Since I was a small child, I've had a fascination with computers, probably through video games. Actually, I still remember the first time I saw a video game. I think I was seven or eight in Beijing. You know, I had not seen a computer up to that point.

5:13I was thinking it was a big deal when we got a nine-inch color television. This is in the early 80s in China. And the first time I went, my mom, I think, took me to an arcade, a video game arcade. I don't know if listeners still know what those are. They've seen them in movies at the very least, you know? Yes. Those are places where you go and you put like coins into and you can play games. And there's these giant machines, arcade machines. And that's the first time I saw like a video game. And something about the pixels, I think, really kind of drew me in. not unlike a moth to a flame, I guess.

5:51It's not entirely like a bad analogy, but I'm like, ooh. And I think ever since then, I've been fascinated with just computers. But I think over time, I think I've really come to love computers and computer science as a discipline, but perhaps in a strange way. And I'll try to explain that. So all my degrees are in computer science. My My Bachelor of Science is in CS. My PhD is in computer science. And in some ways, I'm kind of a strange computer scientist. But I think I say strange because I tend to build things with computers that nobody quite asks for. Interesting. And that solves no problems that seem to exist.

6:33For example, Ocarina, right? This is something that was designed back in 2008 for the iPhone. and, you know, let me see if I can play you a little here so I can hear this.

6:51So I'm holding my phone as I might have sandwiched and I'm blowing into the microphone located at the bottom of the phone and I'm using multi-touch to control the pitch and vibrato is controlled by the tilt of the phone. So I'm barely even seeing the screen But, you know, it's kind of a physical thing. So I'm going to try to play a little ditty with this.

7:29For example, so that's me blowing literally into my... Well, thank you very much. That's me literally blowing into my iPhone and into Ocarina. And this is an app that I would say took a lot of, you know, it took software engineering, software design, interaction design, signal processing. All the sound is generated live on the phone. Using, you know, the sound parts written in the programming language Chuck, which was my dissertation at Princeton. Oh. Which I'm still developing. Right. Like 20, actually 20 some years later. I'm still working on and myself and a large team of people in a community of people working on Chuck.

8:10Just explain what Chuck is just real quick for people who don't know. So Chuck is one of these tools that I've been talking about. So Chuck is a programming language for music synthesis. And you're writing code to generate sound, but also to write code to kind of either to algorithmically figure out or generate like kind of the music that's coming next. Or it's actually could be mapped from human interaction. In this case, in Ocarina, it's human interaction, human computer interaction. And I am using my physical interactions to actually control the sound that's happening. And so, and a lot of times it's a mixture of the two where it's, there's some amount of automation, but also like it's really trying to also figure out what is an interesting way to kind of put human interaction into the loop of the thing that we're building.

9:00Again, nobody asked for. that solves no problems that exist. And I think in a way that's kind of been the running through line, if you will, for my students and I and in my research group, is that we think about engineering, we think about technology, and we think about what kind of problems can we solve with this, right? And while, yes, problem solving is something that we as humans need to do, and I'm interested in doing it myself, I also feel like as an engineer we can build things that isn't always motivated by kind of a practical utility. Tell me how you go through that as kind of a creative process.

9:40I think I spend a lot of my time talking to engineers and developers and stuff who say, you know, some of the stuff you just said, and then they like small leap to, and now I make B2B database management software. And so I made yet another AI chatbot, right? And I think there is this easy way to work backwards from like, okay, what is the problem I can solve that will help people? And I think using technology in service of going the other way and saying like, what can I just make is just such an interesting and different kind of creative process. I'm curious how you have sort of, especially you have to teach this stuff to students.

10:24Have you refined the process of how to do this stuff for people? Yeah. I mean, I choose this. I wrote a whole book about it, Artful Design Technology in Search of the Sublime. And it's really this idea of looking at tool building as engineers, but also as critical engineers, you know, critical tool builders. And the critical part really has to do with the question, why are you even doing this thing, right? Why did I design Ocarina? And it wasn't because, like, I went out and figured out what do people need? You know, it's like, hey, David, what do you need? And you're like, well, I need to blow into my phone.

11:00I need a flute app. Yeah. I need a flute app. And also in the same app, I need to be able to listen to other people around the world blown to their phones. Right. That's been missing. That's a deficit. No, that did not happen. And so it wasn't really kind of this traditional idea, at least traditional now when we think of engineering, we think of this as like, okay, that's a, what is the need? What's the problem? State the problem. And so we can find a solution. This is not need-based design, at least not in that sense. So why was this designed? This is what some people actually might call, for perhaps lack of a better word, value space design.

11:36And value space design is saying, well, you don't design out of necessarily a clear and present practical need, but out of something you just, you really deeply as a person believe in. And so for something like Ocarina and perhaps for a lot of the other tools that my students and I build, perhaps one of the values, the core values that we are trying to speak to in these design is just simply that music making is good. That's the belief. Music making does a person good. And it's not about getting to a product necessarily, but about actually the process of playing and sometimes and often learning an instrument.

12:15Not unlike learning to play like a well-made, perhaps challenging video game. Because there's a great satisfaction in that if the game is well-made, it does not matter if it's difficult. In fact, you might appreciate the difficulty. Because then once you feel like you're learning the system, you really can then express yourself using the system. And then the challenge and then able to overcome the challenges eventually of the game becomes hugely gratifying. Okay. Has that started to feel like a, I don't know, dying phenomenon to you? The idea that In this world of incredible convenience and efficiency where everything is available to you, you know, with a push of a button or like a plug-in in an app, that doing something just for sort of the joy and pleasure and slog of doing it is worthwhile.

13:11Like, that almost feels like a sort of beautiful, deeply anachronistic way of looking at the world right now. Well, unfortunately, it is. It feels anachronistic. It feels like it's out of time. It's not something that, you know, in this convenience-driven, optimization-driven, competition-driven fabric of a society we live in, it feels deeply anachronistic to build things that are playful, that are interesting for their own sake. but at the same time I feel like it's what makes us us and I don't even mean I don't mean just building things I mean anything I think that you feel like makes you you whatever that may be I like to at least offer the possibility that it actually is something that is that you you want to do and you and you care to do in fact you would probably forego a lot of practical needs in order to do.

14:11It's like a passionate hobby. And the passionate hobby is usually not about getting to the result. It's not about optimization. And so I think this gets to this idea that, yeah, while this idea of building things for its own sake or doing things for its own sake feels like it's out of fashion, but it's also not. In a sense, I think we actually still do these things. They're just usually not in engineering contexts. For example, when people cook for themselves because they enjoy cooking. There's a deep joy to go into the pantry, work up, find the raw ingredients and make a mess of things as you concoct a dish.

14:49And that dish, doesn't that taste different because you know, because you made it, right? It does. It has to, especially if you're someone that enjoys cooking. And I think tools are no different. And I think the danger of being in, I think in a world or a society where the things, the tools we make are only about optimization, only about convenience, only about reducing the cost of labor, is that we actually, in those tools, those tools can actually alienate us from who we actually are. Because, you know, well, let's go back to music and let's put, you know, let's talk about music and AI, right?

15:29Right. Yeah, today we have AI that can produce really impressive images, video, music. Well, that's a tricky word to use. Even songs are a tricky word to use. It's like, you know, what is a song? But let's, for all intents persons, say AI can generate quote unquote songs. And with just a prompt. It's like this almost age old thing that people thought they always wanted, which is if I have a musical idea in my mind, I just want this thing realized. Wouldn't that be great? You know, and I don't, maybe perhaps I don't have any music education. I've never taken time, but I just want this idea realized in my head.

16:06Wouldn't that be great? And in a way, AI kind of does that somewhat today. But then I think the question lies, what does it mean? And, you know, and I'll draw another analogy here. And this is something that I call, I guess, it's kind of the bubble gum effect. And what do I mean by that, right? So have you ever chewed gum, David? Sure. Yeah. Okay. So when you chew a piece of gum, when you first pop the gum in, chewing gum or bubble gum, right? How do you feel? It's great. It's delicious. Yeah, it pops. It's like the taste is great. How long do you usually chew that piece of gum? I mean, it depends on the gum, but somewhere between like 10 seconds and three minutes.

16:50And then it's over. And when it's over, what do you do with that piece of gum? Throw it away. And do you, after that, how much do you think about that particular piece of gum? Never for one second ever again. Okay. So I think that's the analogy I'm trying to draw here is that, yes, maybe it's so easy that I can go to a generative AI system and say, render me an image of Barney the dinosaur withdrawing cash from an ATM, but only broccoli is coming out. I think actually, generative AI today would do a rather impressive job. Yeah, they seem to be like, whoa, yeah, yeah, that's it. And I may go and share this with my friend and be like, hey, check out, look what I made.

17:37And I think there's actually that social function of that but then more likely than not i'd probably be like great on to the next thing yeah that took all of three minutes five minutes the timer probably takes to chew a piece of gum whatever think about that thing again maybe a little bit more than the piece of gum i discarded and i think that's what i mean by bubble gumification in fact it is kind of almost the in the limit of the the this kind of push of automation unquestionably pushed automation to basically try to reduce the cost of labor down to zero in like every case we possibly can. Right.

18:16Well, there's a there's a there's an argument inside of that, I think, that that you hear a lot with kind of anything creative, which is that the process matters, right? That that with the provenance of something and where it came from in the story of the thing and the person who made it, that stuff is is sort of indelibly connected to the thing in a way that you don't always understand even, but matters in some way that is very hard to quantify and thus is very hard to talk about in some ways. That stuff matters. And I really buy that premise. But I also, what I wonder then about AI is if what AI is going to create is like some entirely other thing that like maybe what we want from AI is not to make songs that sound like Taylor Swift, but something that sounds completely different.

19:07And this is part of why I think your work is so interesting and why I'm curious about how you're thinking about in using AI. Like you've made a career out of making computers make noises that they're not supposed to make. And I think what those ultimately make is something different, right? Like you're not writing code to make Taylor Swift songs. You're making something else. And I feel like if you approach AI as that kind of tool, both to sort of bend it to your will, but also to try to figure out like what sound it makes. I guess my general life philosophy can be summed up, or at least in aspirations, that there should be room for that too.

19:49Okay. And I mean this both in a way, like what I mean by that is I think about this in terms of music. I think about in terms of actually like people and a society. there should be room for that too as long as the thing is not like harmful and dangerous but also it also means that i guess there's you know it's really what i'm trying to say is that it should be a pluralism we should have room to to have capacity to have a pluralism of well of what i would say of values aesthetic social um in fact that would be my notion of what part of what a civil society might have, but also maybe a necessary condition for humans to be able to flourish, is this idea to have a capacity for pluralism.

20:43And so, yeah, in that sense, I think there's room to think about how can we explore, use AI to explore the realm of untapped sounds. But how we do that, first of all, I think is really important because it does go back to what you're saying about the meaning that we ascribe to things that we might call art. All right, we got to take a break and then we have lots more to talk about. We'll be right back.

21:13Amazon has everything for everyone on your list. Like your teenage son, who won't give up the peach fuzz currently posing as a mustache on his upper lip. Get him an electric razor. Amazon has early holiday deals on gifts everyone wants, like the latest kitchen gadgets and toys, and some they don't even know they need, like that razor. Grab him a mirror, too, so he can see how pretty he is without that dirt on his lip.

21:44Alright, we're back. So, to go back to the idea of computer music, one of the things Guh has talked a lot about over the years is the idea of finding new ways to interact with technology and computers. Like that glove I mentioned at the beginning, that's a new way of using a computer, if you'd boil it all the way down. And the way he thinks about it is more explicitly playful and more exploratory. Instead of just trying to figure out how to use an app most effectively to get work done, it's just a totally different approach to all of this stuff. But that can be a hard shift to pull off, I think, to go from trying to learn how to use a technology to figuring out what's possible with it.

22:27But that shift, that transition, is, I think, what Gudd does all day with his students. So I asked him about it. I saw something that you either wrote or said about your music and AI class, where one of your goals was to teach them how to play with AI. We don't talk about AI that way. What does that look like? What does it look like to play with these kinds of systems? Well, play, what does it look like? Well, I ask my students, for example, to build interactive tools with AI that they can deploy into their everyday lives, but also often would involve human interaction. My students have built instruments that basically track the opening and closing of your hand to generate singing sound.

23:13And turning your hand changes the pitch of the sound. opening and closing your hand kind of articulates the sound.

23:28You can't do that with a prompt-based system right now, right? That's not what that's about. But this means you need to go to the process of learning how to use that instrument. And another one, it's playful. I mean, that's playful and it's interactive. Another playful one is one of my students made this thing called Autoriz. Okay. It's a computer vision system that basically has been trained to classify between, like, to recognize a seductive look. And it plays, like, cheesy seductive music when it detects it.

24:11And that's all it does. it's like by the mission of the creator of auto riz like this is such as this is so dumb and yet it's really it's playful yes it's in in the definition of play is actually something that is not about the outcome it's about the process right right because if you're like i'm playing but if you're worried about the outcome that's going to be produced well that's like that's work Actually, if you need to walk away from an activity with a productive outcome, we'd usually call that work. But play is almost exactly the opposite of that. Right. There's a difference I hadn't really thought about there between thinking about these technologies, AI or otherwise, as tools versus thinking about them as instruments.

25:00And I think we talk about AI in particular as a tool, right? It is a hammer and your job is to go find nails. and and that's that's what it's for and we don't necessarily know what all the nails are and that's the exploratory phase but you're looking for nails uh but what you're describing is something very different which is like here is a thing with a set of capabilities right it can it can it can do a bunch of things and where all of the interesting stuff lies is figuring out what to do with those things and how they mix together and new unexpected combinations of those capabilities that turns into something, which is so different from I have a hammer that hits nails and I have to find more nails.

25:43I go find more nails. Yeah. Let's put it this way. Let's say we make a Venn diagram, which means let's draw a circle. And that circle represents the sets of things that the AI is potentially good at doing or would be good at doing. Let's draw another circle. let's call that a set of things that humans do, want to do, and are perhaps can be good at doing, like ping pong or making music or whatnot. In the Venn diagram is that let's draw these two circles, but that's also they're intersecting. So there's a region where it's in basically in space where it's in both circles. It's the stuff that humans are good at doing and AI will be good at doing.

26:18I feel like we're kind of automatically and almost unquestionably stuck in that intersection with AI is that we're constantly looking for things that humans already do. And now we just want to have AI do that same thing. And in fact, the more, the quote unquote better an AI can do it, the more progress surely this must be. You know, some have called this the Turing trap, actually. It's this idea that we have been set down this path where progress in AI is judged by how indistinguishable a system is to humans. Right. And the danger of that is that that becomes kind of the dominant and perhaps almost the only way we can imagine of thinking about progress in AI.