When AI Meets Art

0:01On June 27th, our team headed to New York City. We are at the A16Z office for the first ever AI artist retreat. That was A16Z Consumer Partner Justine Moore. Justine was one of many partners who attended this retreat, which brought together the builders behind some of the most popular AI creative tools in existence. That is, 11 labs, Korea, Vigil, Udeo, Idea Gram, and Civet. Altogether, with 15 top artists. These are the folks who are often doing the coolest things with these sorts of tools. They're kind of pushing the boundaries of what the tools can create. Today, you'll get to hear from many of these AI founders who, together with these artists, are advancing what it means to be creative.

0:48Art is going to get better than ever. The average art output is going to improve, but so is the ceiling. It also is a higher participation rate. Everyone who is interested in creativity can be creative and express themselves. It was just so cool. That was in Nishbacharya, general partner on the consumer team, but that's not all. I've been a founder twice. I've been spinning records as a DJ for 25 years and I'm all about AI and art. So what happens when you put all these investors, leading artists and creative tool founders all into the same room? I mean, the vibes have been immaculate And I think that the thing that's most surprising is how much everyone has in common.

1:27Like the founders are more creative and the creatives and artists are more technical. I think the other thing is just how interdisciplinary it all is. People making video, want to play with generative audio, people making music, want to play with sound effects, it's just incredible to see. One of the coolest things was a lot of the founders had recognized people by their online screen names or new. Oh my gosh, you used my tool to create this incredible song that went super viral. Or you used my product to make this kind of amazing video animation that our whole team was talking about for a week.

2:01These are people who have been interacting with each other often daily online for the past six, twelve, eighteen months, sometimes even two years, but didn't even know what each other looked like in person. Now today, you get a behind -the -scenes look into this event, including the origin stories behind many of these tools, which by the way some have never been shared publicly, and have these tools which have all gone through their own viral moments are navigating this AI wave, and what they see on the horizon. Let's get started. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A -16Z fund.

2:47Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast. For more details including a link to our investments, please see A16Z .com slash Disclosures.

3:04Here we are in 2024. We're at an exciting inflection where your creativity is being unbounded by the tools available. I mean, we're early, but there's more people making more art and more people making more tools to make art than ever before. And if you kind of look at the history of technology and art, every single time there's been a new technology, the amount of art has dramatically increased. People were worried that drum machines would compete with drummers, and instead there's more people making more music with both drummers and drum machines than ever before. So I think there's a sort of equivalent moment here in technology and art where we're at the beginning of everybody who has taste and interest in art being able to make it.

3:45Many have drawn parallels to prior computing waves, but is this any different? Well, what's different is for the first time we're creating these sort of left -brain things. You know, I mean, computers and computing platforms have really been in the business of precision, and now we're creating products that are intentionally imprecise, beautifully imprecise, so it just feels like a whole different flavor for products and product design than we've ever seen before. So let's introduce you to some of the people behind those products. We have companies here covering basically every sort of creative modality, image, video, music, 3D, speech, all those sorts of things.

4:20That includes Conner, I'm a co -founder at UDO. And Omar, I'm the head of design at Love & Labs. Both companies are focused on audio. With UDO focused on music, while Love & Labs is tackling everything from voice to sound effects. Meanwhile, founders like... Mohammed, I'm the co -founder, CEO at Ideogram. Victor and Diego. Who are the co -founders of Gria and these founders are building at the increasingly sophisticated world of 2D imagery and video. Plus 3D. IDogram for example, let's degenerate AI imagery with accurate text embedded, a surprisingly difficult technical feat. Vigil on the other hand is building at the intersection of video and 3D.

5:03Meanwhile, Gria has come up with a suite of AI tools like upscalers and real -time generation. or in the case of CIVET, a new breed of marketplace. My name is Max Phil Holker. I am COO at CIVETI and co -founder. Yes, I'm Justin Merr. I'm the COO and co -founder of CIVET as well. And CPO and CTH. Lots of things. The joys of Astarda. We are a massive community of people making tons and tons of AI creations using community -made models with community -made patches to those models called loras. We give people the ability to either train on a few specific models So a model focused on anime or a model focused on being semi realistic or they can select their own custom model to train on top of With AI moving so quickly, it's clear that we no longer live in a world of just chat GPT and mid -journey numerous companies have springboarded into the site geist and grown at unprecedented rates So we thought it was fitting to take a step back and document this whirlwind of a journey And while many of these founders have been quietly working in research for years, their origin story often started from scratching their own itch, Muhammad for 5 -8 grams.

6:12I guess part of it is that there is this thesis that everybody has an innate desire to create. And as humans we have this inner creative child. The education system sometimes kills this creative child unfortunately. and what's finally possible with technology and AI is to help people express themselves visually and creatively. So that's the interesting part when you think of using image for communication, then you can communicate much more effectively if you have image and text together. From a Hamad it really was this unique combination of text and imagery. For me, image and video is dear to my hearts and like very personal.

7:01But for Connor, it was his connection to music. Music for me I think is, it's very special medium. It's everywhere all times, like it's in the background when you're at a restaurant at a cafe, you're listening to your headphones, when you're going to work in the morning. It really has an emotional resonance with people. And for me, making that abundance, like the kind of promise of generative modeling is that a lot of this can be far more abundant than it ever was before. And for Victor, it was his discovery that programming itself was the creative gateway. When I discover about programming, that was great to me because I realized that through coding you can also be super creative.

7:33But the moment where I discover about early Genai models like DCGAN and later on, StahlGAN, that's when my mind was blown and when I realized about the creative potential that this technology had, and that's when I fell into the rabbit hole, and I feel like, like, Cria to me, it's been kind of the snowball that it has started with me realizing that you can use artificial intelligence in a creative way. but for a mar, it was building his own side projects and it desired to share what he was learning that actually propelled him into his role at 11 laps. It's really funny actually. I, over the last, maybe a couple of years, started diving into AI tools when Chad GBT came out and started making things on the side for fun.

8:15One of those was a children's book that ended up accidentally going viral and that kind of was my journey into AI. Through that and making that book, I started exploring other AI tools and what I really enjoyed was sharing what I was doing and how I made it. So I discovered 11 Labs and made a podcast with 11 Labs where I was like talking to a fictional figure and we were having a back and forth conversation. That also kind of did the numbers on Twitter and then I was like, I love using this tool. I'm gonna make my own AI short movie. Actually, Justin and I are friends and so I showed it to her and I was like, I kind of need free credits.

8:50Because this movie is using of all the credits on 11 Labs And she's like, you should meet the founder, Maddie. We met, we really hit it off, and Maddie, in classic Maddie fashion, was like, very direct, and at the end of the call, it was like, hey, we're actually looking to hire someone to lead design. Are you interested? And then to work on a product that I'd used for over a year. That experience also gave him a taste of just how quickly the space moves. And also a hit of virality. It happened because a friend of mine had their first kid, and I read her at children's book, actually. I was reading, and I was like, this story kind of makes no sense.

9:23So, so I went back home, I'd been using the journey a lot, Chaddgbt2xOld, combined the two to create that book. And then I was like, how do I get this published? An Amazon has this amazing publishing service. You can get a book out within 48 hours. I had a paper back in my hand in 72 hours. So fast. And it's really interesting because writing a book and publishing on Amazon is like, it was almost like iterating on software. I find discovered or a type or whatever, I just updated the PDF and the new book was out and a new publishing line was out. And so, yeah, I put it out there, got a ton of virality from that.

9:57And yeah, that was a really interesting experience. Free AI, we were in this era of consumer where it was just really hard to get people's attention, really hard to get them to download a new app or try a new tool. You had to spend a lot of money on customer acquisition. Now, just with the real excitement around AI, if you make a cool product, But you can get it into the hands of people and get them using and talking about it. This was the case for Victor and Diego at CREA, who eventually met their own viral moment, although it didn't come easy. First of all, it was called Geniverse, coming from a generative universe, best name ever.

10:36And essentially, it was like two things. It was on the one side, an open source library that was kind of integrating all the cool stuff that it was available at that moment. And on the other side, it was a creative tool. And the way how it looked, it was like super experimental. Like we didn't really know how to do UI design or any of that. Like the background really had stars and everything. So it was like a galaxy, like the generative universe, right? And then you could put text, you could put images, and you had a few things that you could tweak, and you could generate images. And you would see like the image evolving in real time.

11:09And the images that you liked, you could keep them. And they were added to this kind of universe. And essentially you ended up with a ton of images in this interactive space. So for us, it was always with the same idea in mind on the one side, controllability and on the other side, intuitiveness. Like how do we make tools that doesn't look daunting? Because AI in the end is like new creative medium. A lot of people are using it for the first time and we want them to have an experience where the AI does what you expect to do. And you don't need to learn about like crazy prominent geno rings and like all these tweaks up to get good results.

11:46And on the other side is controllability because we are dealing with creatives. We are dealing with folks who are not just okay with having a beautiful image. They want that beautiful image. So these are the two core principles that we had since then. And we build kind of a Figma -ish interface for AI and we had every single utility that you had at that point with the civil diffusion in there. We had like thousands of AI models that you could use. We had every single technique, like every control net, everything was in there. But you know, like it was not working. It was like a learning curve that some people were just like not willing to take.

12:20So then we had the first kind of virality moment when we ship this thing that it was almost like an equivalent to a meme generator. I remember that we were seeing like all of these images on Twitter with the spirals, right? So when we were like, what's going on with the spirals? Like we can do it. This is like one day of work. And I remember at that point, Diego was like, We should do something with this, we should do something with this. He's getting so viral and always more like in the mood of like we need to ship like this whatever feature We were working on that moment until at one point we were like, okay, let's fucking do it And we did it like in the sketchiest way possible like in one or two days and we shipped it in Twitter and it got viral It was like the first time That we lived something that I had read about in terms of these what PMF looks like as the first time I was like, oh Jesus Christ, okay But this is how it looks.

13:12As soon as you go to sleep, I can feel the hard beating. And then like, you sleep three hours, you wake up, because you know that there's stuff broken. The email starts to get flooded, Twitter starts to go in there. Suddenly, literally, every day was like crazier than they went before. I was like, oh my god, a thousand people. Oh my god, 10 ,000 people. And they're like, oh my god, like, football, club Barcelona, like, number one soccer club just used us. What? Why is it like how many followers they have? Oh my god, 100 plus million followers on Instagram. Okay, I feel like it was actually hard in the sense of as a founder, you're like, I've put multi amounts of years into like many things.

13:54And then the thing that we literally are like, that's not important, gives you all the success. So it's a moment of reflection. Sometimes like the world throws truth at you. that those years of work were not all for nothing. And I don't think like the years that we've been working on was like a waste. No, it actually is, oh, that's where you learn on the technical level. It works. I mean, because it was so much failure, we learn about, okay, how do you communicate with your co -founder? I think that's something important to note about those times is that we were very, very aware that this was a trend.

14:30And then this was not the end product that we were building. It was like almost like a marketing engine that we were using to get better branding and to get known. They were finding us because of one reason, but they were staying because of another one, which was like this other product that we were working on. Even we knew that. I think that the core learning that we got from this experience is that the AI field changes constantly. Like every month or every two months, there are new breakthroughs, new techniques, new ways of doing things. and the tool that we were building, it was like already starting to get too complex because we were trying to put everything in a single tool.

15:12And I think that what we learn with the experience of the spiral, virality is that there's a lot of value on simplifying super niche and simple use cases. And that was the case again when LCMs were released, right? We saw this technology, and at that moment, we use all the experience that we got from the first virology to engine the second one. And the second one we knew that it was not a trend. It was something extremely valuable. We were like finally being able to get that interaction that we were looking for for almost years, right? Like we can generate images in real time and have full control of the colors, the composition, the shapes, everything.

15:52That was almost like a dream come true. Victor and Diego have now hit virology several times over, but can you engineer that momentum? In some cases, it's all about having a single critical feature not offered elsewhere. Muhammad from I do Grim. So basically it was the version 0 .1 as we called it and this is back then in September of 2023. And it was a model that was working. It wasn't perfect. But if it felt like it's already good enough to give it to users and it was the first model that could put legible text into images. So it kind of went viral because of the unique capability of the model.

16:36Somehow the ability to put text into images felt needed. But in other cases, it's about cleverly enabling the masses. Or in this case, the memesters by drastically reducing the barrier to participate. Here's hung with Fickle story. It went pretty viral, right? What was that like experiencing to put a product in the hands of so many users and also see that kind of spread on its own? Yeah, it was we didn't anticipate that for sure in the very beginning we were thinking most targeting content creators but somehow the mid -makers the mid -sters they catch up on it and That's how it got pretty viral and also that's also thanks to some of the templates We spent so much time discussing like why this is the case There was this template, the Joker Leo Yali coming on the stage, and there's a Joker character that replaced down on the video.

17:30And we've seen millions of different characters just remixing the same moment. And we realized that the main reason was used to use. It's so easy to basically upload one image, and then one click choose that template, and then in just a matter of seconds, you'll have yourself in basically in that same moment. Maybe one other aspect of the virology is, as you said, the meme makers got a hold of it. There's this kind of fun, maybe even silly aspect to it. How have you thought about that? Well, I think that speaks to the entertainment value. For anything we have real entertaining value, it has to work.

18:07It has to work well. That actually requires a lot of rigorous in the research site. So we are pretty serious about being silly. It takes quite a bit of a rigorous research to do that. The second thing is you have to have a tool that provides precise control. And then because people are getting what they want, they can have all kind of a right -haha fun with it. You've mentioned characters and templates a few times. What are some of your favorite examples of those generated on the platform? One is the Docker Commodore State template. That one is basically the moment we realize more actually, people wants to remix, and there is this variety and memes aspect of it.

18:48And the second one is there has been one rocketing advertising song and people are dancing lists and we're also seeing the millions of people remixing that same template. And this is interesting for us because you make such realise that as long as there is this fun elements to it, people actually don't mind this content having a little bit brand message. When you think about applications and I know it's early days but have there been any that have surprised you about the ways that VIGL has been applied every time a founder creates a product. They have applications that they envision. And then the best products are often people are using them in alternate ways that surprised them.

19:27Yeah, that was exactly the case for us. In the very beginning, we were mainly thinking of movie makers, game makers, using this might be a quick animation, a previsualization to follow them. It's exactly pretty useful for that. And we've also seen the early users adopting to that. But then we never anticipated that the Mimsters, so since that will be also providing those templates. So we've been keeping track of the latest trendy dance moves, sports events, etc. And we've also seen content creators hopping onto this. They are actually reaching out to us, say, can you feature our dance or song on your platform and then can we collaborate promoting some of those that's being really interesting.

20:10We ask Connor the same question around what he's learning by seeing how the masses are using UDEO. The model we originally launched was a model which generated 32 -second clips. And so to make a kind of a full track, you would extend that in various directions. You would add an intro, add a maybe a chorus and an outro and stuff. And you would build a song like this and you would start with these junks. And I suppose we've actually come to realize quite quickly that people's experience with music when they ask for a song is actually a lot more focused on that. So they kind of want a song that begins at the beginning, maybe ends with at the end, it doesn't have to be long, it could be like a short, two -minute clip, but it has a verse in a chorus and a verse and there's a structure to it.

20:48And so I suppose we actually underestimated just how kind of important that was and so that's something we were making steps towards rectifying recently. 11 Labs was also no stranger to the surprising and inspiring user behavior. Yeah, I think one of the most surprising ones was people who had lost their voices and then had used a level of labs to bring their voices back to life and then do the thing they love doing. So we had Laurie Cohen, who was a lawyer, she lost her voice one morning and a friend of hers helped her replicate her voice with level of labs and then she was back in the courtroom delivering arguments.

21:21And that to me is just such an incredible moment because you don't expect that and I think our idea was like, hey, we're going to give ideas of voice with our product and our tools. But this gave someone their own voice back. And I think that was such an amazing thing to see. And we saw that again with a climate activist, Bill Wheel, who was delivering his awards speech he suffered from ALS, unfortunately, but again was able to replicate his voice and then deliver that award speech. So I think those kinds of things are just like, you're like, wow, technology being used in a way we didn't see it.

21:51And now we want to lean into that and of course help others. Yeah, maybe in the opposite sense, have there been any applications that you've actually built or designed for? We're like, everyone's gonna use it for this, obviously, where that's actually not in the case. It's interesting when we launched dubbing and automated dubbing. We thought, yeah, this is it. Like everyone's just gonna use automated dubbing, it's gonna be great. And of course with dubbing, one of the most important things is accuracy, right? And so automated dubbing, we realized people still want a lot of creative control on that.

22:18And so we ended up having to build dubbing studio, which allowed people to go really fine -tune that dub and change a lot of the content. And then we also introduced 11 studios, which was basically creative teams that help you dub your content with professionals who are really good at that. And so we realized that actually was what people needed more of and not just automate everything and all the things, right? And then it actually picked up again. And this is something even when I was working at Palantir you learn, which is like the temptation to try to automate everything or to use intelligence for everything, but actually there's so much value in like having someone in the middle and like still having that human touch to take it to that final step and this is something we learned with dubbing.

22:59And as these companies get all this new data, it's not always easy to figure out who they should be catering to. So how do you think about what you build and for who? Your tam is everyone in theory. I think what we acknowledge is that we probably have different types of users, like distinctly different types of users. At the very top being, does someone in a studio who's making an album at the very top level? And then at the end of the scale, it's maybe someone on their phone who wants in a minute, they want just a funny song to send to their friend. And those are two very different experiences.

23:34And kind of somewhat similar to the kind of, I put you can get from just an instrument in general. Like, someone can have a guitar at home that they play just to have fun from time to time, like a totally personal thing, it's not anything necessarily serious. It's just a way to express yourself with it kind of musically. And the same way someone can take that same guitar and a professional can take it into a studio and make it part of something fantastic. We like the technology to basically enable all ends of the oil parts of that spectrum. Several are unspricingly using their flywheel of new users to inform their decisions.

24:08Yeah, we kind of use our user base and the prompts that they enter into the system, to decide how to evaluate the quality of the model and what to prioritize. What's interesting is our users used IDogram to tell us what they want. So they were like, we want image upload, we want comment, we want more servers. So I guess the good news is we already have this flywheel of users coming and using it, some are paid, some are free. And that sets the vision for us. Hong from Vigil has actually used these new learnings to expand who they're building for. How are you thinking about who you now build for?

24:49Are you pivoting or adjusting to incorporate these new use cases? So we are broaden our target audience in this sense. So we are seeing this as eventually we're going towards the section of a new type of AI power content platform. And the content platform is really important to have all these creators. and those are still content creators, the artists, the movie makers, the game makers, the team of game designers. They are the sources for all those new ideas, ideas, all those new templates. And then we're broadened into content consumers. Basically, Vigo is a new way to consume content. Before AI, it was mainly like, if I like the moment I would share it, I would like it.

25:32But there's a deeper engagement you can have with that moment. And I can basically, I left this moment so much that I want to put my own avatar in it. It's almost like in a parallel universe, I want to see how this looks really if that moment myself. So this is a new kind of content consumption. And that's actually one of the most important aspects of a reality actually comes from all those creative ideas. So for us, you saw about empowering those creative community first, making sure they have what They want the best tool, they have early access to newer features, they have almost private channels, they have almost unlimited access.