Giving NPCs the power of AI with Convai's Purnendu Mukherjee | E1839

2 Nov 2023 · 50 min

Ask about this episode

Ask anything about it. ChatGPT or Claude reads this page and answers with the times it was said.

Connect VO and ask about every podcast you hear, including the moments you saved. Add to ChatGPT · Add to Claude

In short

Podcast Summary: This Week in Startups - E1839

Episode Title

Giving NPCs the Power of AI with Convai's Purnendu Mukherjee

Episode Description

In this episode, Jason Calacanis interviews Purnendu Mukherjee, CEO of Convai, about the innovation of AI-powered non-playable characters (NPCs) in gaming and the broader implications of generative AI.

---

Key Points and Discussions

Introduction

  • Hosts & Guests: Jason Calacanis hosts Purnendu Mukherjee.
  • Purpose of Convai: Create AI-powered NPCs for video games, enhancing the interaction quality in gaming experiences.

Convai Overview (6:05)

  • NPC Definition: NPCs (non-playable characters) are characters in games controlled by the computer, often with limited script-based interactions.
  • Goal: To transform NPCs from being repetitive and predictable to dynamic and engaging characters.

Character Creation Process (11:36)

  • Demo of Convai's Tools:
  • Users can create characters with unique backstories, personalities, and voices.
  • Integration into game engines (Unity, Unreal) to enhance character interaction.
  • Characters can learn and adapt based on player interactions.

AI and Gaming Integration (17:18)

  • Game Development Challenges: Creating immersive NPCs is costly and complex.
  • Potential for Realism: AI NPCs can interact in a more human-like manner, potentially altering gameplay dynamics.

Business Model & Monetization (22:17)

  • Convai's pricing structure ranges from free to subscription models.
  • Usage Charges: Similar to models used by chat applications, based on interaction volume.

Personal Journey and Background (25:00)

  • Mukherjee's Experience: Time at Nvidia, where he focused on generative AI and deep learning.

Generative AI Evolution (29:57)

  • Discussions about the generative AI boom over the past few years.
  • The impact of technology advancements on AI capabilities.

AI Regulation & Ethics (36:01)

  • Conversations about the implications of AI regulation.
  • The balance between innovation and responsible AI development.

Future of AI Interactions (40:11)

  • Predictions that AI tools will become more sophisticated and integrated into everyday tasks and video games.

Concerns about AI (43:13)

  • Discussions about the potential dangers of AI, including misuse in warfare and the need for careful regulation.

---

Key Takeaways

  • Transformative Potential: AI-powered NPCs can redefine gaming by offering personalized, engaging interactions.
  • Open Source vs. Closed AI: Debate on the benefits and challenges of open-source development in the context of AI innovations.
  • Ethical Considerations: Importance of establishing guidelines to prevent the misuse of AI technologies.
  • Future Trends: Expectation of AI rapidly evolving, leading to significant shifts in multiple industries beyond gaming.
  • Human-AI Interaction: As AI becomes more integrated into daily life, the dynamics of human relationships with technology will change dramatically.

---

Conclusion This episode sheds light on the intersection of AI technology and gaming, exploring how Convai aims to enhance the player experience through intelligent NPCs. The discussion touches on broader themes of innovation, regulation, and the future of human-AI interactions, making it a compelling listen for those interested in technology and its implications.

---

Resources

  • Check out Convai: [Convai Website](https://www.convai.com)
  • Follow Purnendu Mukherjee: [Twitter](https://twitter.com/purn3ndu) | [LinkedIn](https://www.linkedin.com/in/purn3ndu/)
  • Equinix Startup Program: [Equinix Startups](https://equinixstartups.com)
  • Fitbod: Get 25% off at [Fitbod](https://fitbod.me/TWIST)
  • LinkedIn Marketing: [LinkedIn Ads](https://linkedin.com/thisweekinstartups)

---

*Note: This summary captures essential discussions and insights from the podcast episode and serves as a resource for anyone interested in the evolving landscape of AI in gaming and technology.*

Written by AI. May contain mistakes. Listen to the episode to check what was said.

Hear the part that matters, and keep it.Open this episode in VO. Double tap your headphones to save a moment as you listen.
Get VO free

Transcript

Automatic transcript. May contain errors.

0:00open source can win if there is an incentive system for people to contribute their niche data. Ah, interesting. So if it has a license to use Falcon or a license to use this open source model, and then if I put my data in, I own Getty archives, I say, I'll put my Getty in there. But I want, you know, 3 % of the pool, or whatever it happens to me. So that requires some centralization, which, you know, necessarily. Oh, okay. So, and this, this is blockchain, don't say blockchain. I was gonna say blockchain. Oh, no. Are you telling me that after all this crypto drama, it's coming back? A hundred percent.

0:37The blockchain is coming back. Oh, God, no. This Week in Startups is brought to you by the Equinix Startup Program provides hybrid infrastructure solutions for startups, including up to$100 ,000 in credits and personalized consultations and guidance from the Equinix team. Go to equinixstartups.com to apply today. FitBot. Tired of doing the same workouts at the gym? FitBot will build you personalized workouts that help you progress with every set. Get 25 % off your subscription or try out the app for free when you sign up now at fitbod.me slash twist. And LinkedIn Marketing. To redeem a free$100 LinkedIn ad credit and launch your first campaign, go to linkedin.com slash thisweekinstartups.

1:26all right everybody welcome back to this week in startups everybody knows generative ai is having a massive impact on the creative industries chat gpt amazing to you know brainstorm or workshop some dialogue maybe work on your screenplay mid-journey dolly generating amazing images from text that you can then use for a logo for your company or in a magazine or a website and of course we're seeing all these ai video editing tools to make clips from your podcast or your web show put captions on them make the host of the show speak a different language it is amazing how fast this has happened and one area where ai is making a huge impact we all know is gaming making games is incredibly incredibly expensive and convey is bringing multiple tools together to create a platform where game developers can create AI powered NPCs.

2:24What's an NPC? These are non playing characters. It's used as kind of a joke in online culture. If you hear somebody say he's an NPC, she's an NPC, basically means you're a mid, which is another cultural reference and a way to say somebody is like an average person or unimportant. So NPC non playing character. If you were playing Grand Theft Auto, that might be the shopkeeper might be the bartender, or if you're on tiktok it might be pinky now of course ice cream's so good pinky who slays is not an npc she's pretending to be an npc because npcs are boring and do repetitive things over and over again so the fact that she can mock them and make hundreds of thousands of dollars a year doing tiktok live and people giving her all kinds of virtual gifts shows you how popular the concept of an npc is in popular culture well my guest today is the co-founder and ceo of convey uh and uh it's spelled c-o-n-v-a-i if you're looking for it but it's convey uh as in convey something uh the ceo of the company and co-founder is pernendu mukerji did i get it right mukerji that's right yeah you got it right incredible see this is the thing uh pernendu the harder the name is like indian names sri lankan names greek names i get them and then sometimes like weird english names i get them wrong so anyway welcome to the program you heard my little preamble there uh how did you come up with the idea to make tools to make npcs were in the video game industry before how did you come up with this opportunity yeah absolutely i mean the thing is a multiple things came together which made it so compelling that i was like i have to do this or i'll be regretting um and uh and those multiple threads if i were to like say the most important ones were um i've been thinking about how the human mind works since i was a kid and was pretty fascinated of how uh how our mind works and i've been spending a lot of time with that um and then you know like i did all my thesis work you know like uh on question answering systems, natural language processing.

4:36And after my graduation, I started at NVIDIA as a deep learning engineer and did similar work there, primarily focusing on generative AI, large language models. And that's where I faced the problem myself. You know, where one thing is like you were working on text-based systems, right? But when you want to create a character that has a consistent backstory, that has some expertise, you want to add voice, you know, like have voice to voice conversation and then you want to connect it with an avatar or a non-playing character let's say become the mind of that character right that's a that's a challenging task especially you have to do that in low latency with different voices and different you're basically trying to replicate a mind that can be almost human-like so uh now that the problem that i faced and my passion and interest and what i have been doing i was also a competitive gamer before you know as growing growing up so so like all of those things came together and i was like yeah my life has been gearing up towards this i have to do it so so last year i started convey this was before the chat gpt boom thing but uh uh it started so interestingly the npcs are going to become more like main characters because of your work if you're successful so in a way you're the savior of npcs you could say that yeah making them more human like more uh less like that the npc meme more human than human perhaps so yeah show us uh show us how you make one and then yeah how they act in the game because the kind of joke of npcs is they usually they have like a script that you know as pinky doll was showing us here it's usually a script of like six or seven items and yeah if you go to six or seven items it gets repetitive and funny absolutely yeah i could i could uh walk you through that uh i actually have the whole process maybe i give a brief on that and then we create one?

6:30Yeah, sure. Okay, so I actually have a you share your screen and for people listening will describe what's happening on the screen here. Because this is again, it's cool to make characters in a video game. And these NPCs are typically the non playing characters are typically boring. Yeah, make them a little less boring is the goal. Yeah. Yes. Okay. So what I'll do here is I'll present this particular slide in all the steps that happens. So this is all the step all the way from creating the character primarily on top of LLM to adding voice, to adding avatar, to putting into the game engines. So first thing when you on top of the character, you give them a backstory, they will have personalities, they will have emotions, they can even have memories, you can put guardrails around them in terms of things that they can or cannot say, they can speak different languages, you can add vector databases which augments the knowledge and make them expert on certain topics, you can follow a particular narrative.

7:28And once you've added all of these over text, which we will see right after, that you can add voices on top of them. So like they can listen to you, understand all different kinds of language, speak different languages, and speak them with emotions. You know, like, as you see, my voice is having intonations and whatnot. Once you have this whole voice to voice character, you connect them with the avatars, right? Whatever kind of avatar, you can use them with metahumans, with various different kind of avatar systems that are there. We are avatar agnostic, but we support all kinds. And we also support all kinds of engines, right?

8:01So you can integrate them with Unreal Engine, Unity, anywhere you can create 3D worlds. You can embody these characters with the AI that you just created with the character, right? So lip sync will work, facial expressions, gestures will work. And then finally, once these are like avatarized, you have provided the mind, it means that they can like perceive the environment around you know, carry out different actions, you know, they will move around the world, you know, you can say, hey, go pick up my keep pick up something and they will bring it to you, you know, like, so they can do all of these things, right?

8:32So we solve the whole pipeline end to end. Pause there on that last part, before you go any further. On that last part, you're going to have them go interact with the environment. yeah which means you don't know exactly what their response is going to be because i as the player could tell them hey i want you to go uh you know find me all of the loot and bring it back and they go do that for you okay fine yeah you could say hey i want you to go and insult everybody else playing the game and you know in the style of you could do that too you know jeffrey ross the famous insult comedian um yeah and i want you to use a trump voice or something and go insult everybody.

9:10So now you've essentially created chaos inside the game. So I guess you're, you're, you're unleashing a little bit of craziness. No, that is true. But like, it's up to the game designers that you know, will you allow them to have different voices? Usually, they will probably stick to a singular voice so that it stays consistent with the character. But maybe somebody might might design something like that, you know, to it's gonna hit some language model. And as we've seen, you can kind of convince language models to change their mind about things so yeah things things are going to get a little weird yeah things will but but i think game in a good way hopefully yeah you know in a good way but like the thing is this is a developer tool and the game designers are going to spend their time trying to perfect the character so that it stays in character somebody may want something like that but most cases 99.9 of the cases they would want a consistent character but yes they will have the dynamism where you can tell them, hey, go tell this thing to this friend of mine on the other side of the map.

10:10And they will they will go running inside the world and tell them whatever message you send them. Yeah, awesome. Cloud computing has revolutionized startups over the past decade. We all know that. But reality is a fully cloud based solution is not right for every startup. Sometimes you need a hybrid solution. Like let's say you're working with sensitive data that you just can't put in some random cloud, right? Or maybe you need to connect multiple cloud providers at once. Or maybe you just want a more cost efficient solution, right? Because sometimes these cloud bills can get crazy. If that's the case, you need to check out Equinix.

10:44Equinix Metal will give you direct access to physical servers, but you still get the benefits of the cloud. So there's no need to rack your own servers. Equinix provides on demand infrastructure in over 25 major cities. And here's the best part. They have an amazing startup program. The Equinix startup program offers personalized consultations, guidance from the Equinix team, and wait for it,$100 ,000 in credits. That's not a typo. So here's what I want you to do. Go to equinixstartups.com to apply. And when you apply, James from Equinix will reach out to you directly. That's how serious they take the startup community.

11:24That's E-Q-U-I-N-I-X startups.com. E-Q-U-I-N-I-X startups.com. Equinix startups.com. Get a call from James. Well, I mean, if you asked it, I want you to speak French and I want you to bark like an English bulldog, it could go find in the language model those attributes and start talking French and barking like a bulldog. Yeah, you can. You can. So if you want to try that, we actually have support for multiple languages where if you speak French, you can speak French. So here we are in the interface. And this looks like a piece of, you know, software, you got a bunch of different characters, we pick a character.

12:05Yeah, or we can create a new one. Okay, create a new one. Let's say, maybe I can create you Jason. Okay, so Jason characters voice. Yeah, choose a male voice for now. But we can even add your voice later on. How do you want to say it can you use ai to make character voices like could you say give me a pirate and even if you didn't have pirate already in there we go find pirate yeah we can we can we can add voices uh in this as well um but but uh yeah let's let's go with the default one maybe i'll just say jason hello canis is the host of all in podcast uh and i'll see what what gp3 generates um yeah so um okay put in there hey this is uh all in podcast where you've mastered the art of captivating your audience with your insightful and thoughtful provoking discussions your existence is anchored in the world of tech you just pasted that in or no i just generated that oh you just generated it yeah yeah so based on jason calicanis is a host of all in it generated the paragraph fantastic yeah yeah yeah okay so we'll create that um and once the character is created it basically gives a placeholder avatar, but we can actually have your own metahuman character inside the game engine when we take it there that will look like you feel like you and all that.

13:29So so we have like all of these options to craft it further. So like you were saying, it currently speaks English. So let's just start with it. Hey, how are you? Hey there, I'm doing great. Thanks for asking. How about you? Almost sounds like you almost if I had a very deep voice, it would sound a little like me yeah yeah um what do you do i'm the host of the all in podcast where i discuss tech and investment topics with industry experts i'm also a venture capitalist and entrepreneur so i'm always looking for new opportunities to invest in pretty accurate yeah true sure yeah i think you have this uh knowledge bank let's say you have your own set of files let's say all the all in podcast stuff that's tagged by your name you can upload that and it'll know everything about every episode or something yeah sure exactly and uh so this is the state of mind like how you are feeling as i'm asking this so let's say and this will drive the facial expressions the tone of the voice and all of that um so so let's say i say uh my dog passed away which is not true but i'm just testing it that's terrible i feel so bad so let's see if it changes the the the emotion i'm so sorry to hear that low sing a pet can be really difficult yeah is there anything i can do to help so you see it kind of goes into the grief mode um yeah so you're showing the emotion wheel i forgot what they call that emotional wheel yeah it's called blood chicks wheel blood chicks wheel chicks wheel.

15:04Yeah. Neat. So it shows you the emotional state. Yeah. Uh, yeah, that's interesting. You can also craft the, um, the, uh, you know, like how is your personality? Are you an agreeable person or? Yeah, you can hear it's giving you the it's giving you like a little slider and you can say how openness, meticulous, extroversion, agreeableness, sensitivity, like kind of a Myers Briggs and you can just put me as like a hundred percent extroverted whatever medium emotional whatever and it will just kind of build my psychographic there my personality trait yeah yes yes yes so uh exactly this is the big five personality traits we initially thought of doing myers-briggs but it seemed like this would be more appropriate um yeah to craft this yeah and then there is like you can have long-term memory settings that you can turn on oh you can have you can have guardrails so this is the part it protects from saying certain things so let's say you can do this yeah so it says limit responses stick to character description and knowledge bank only and then all the other other end is talk about other areas so you can just go crazy or you can go somewhere in the middle so that's just how on script the person's going to be so you're basically creating west world here uh when elon or somebody figures out robots you can put this into them and see if they'll stay on script and always stay a samurai or always stay uh yeah a cowboy interesting exactly and then we have this uh narrative design aspect which is basically like you know you can create a narrative one after the other you know like where this thing could be connected to the the next thing and you can craft a story basically where yeah so like maybe i can show you an existing one so let's say you're Making a, what do they call that, a story tree or something?

16:53Yeah, like a dialogue tree or story tree. However, the difference from the traditional ones would be that this is going to be open-ended. So it will still stay within the story. It will try to follow the story, right, or whatever your objective that you have provided it, and yet be open-ended, right? So that's another feature. So that they can play the role that you ask them to, right? So once you have done all of these things, right, you take this character ID, which is the mind of the character that once you've crafted it, now you take it to the game engine, right? So maybe I'll show you a quick one.

17:30So let's say, I'll go to this. So you've got a token, a unique token that represents that character, that'd be very hard to guess. You plug it into the Unreal Engine. Exactly. It's now connected. yeah so you you take it you know like there's a particular character there you put it in the mind of that character that could be a lookalike or you know however you want it and then it's inside the game world you know like and you can you can talk to it this is something that is not shared yet publicly but this is an internal discussion but um this is mainly a scratch video internally but uh this is something we plan to launch in a couple of weeks oh cool uh which which is what i was saying you that these people can actually carry out different actions all right so here we go we got three people in the woods looks like they're in the jungle or something action for example three npcs hanging out sure i can follow you and you're telling it oh follow me there you go okay let's try something so now we are doing something more complex like hey can you go there do something more complex like hey can you move to those trees and back to me again yes i can move to the trees and back to you again oh yeah

18:42so that's one thing the next one we asked this guy to like do a show us a dance maybe i'll share this one basically so one sec yeah this is gonna get crazy uh so we're here yeah and we'll ask the character to bring us the axe. No, no. Tell her to take the axe and kill the other two NPCs. Sure, I'll bring you the axe. I'm joking. Don't tell her to kill the other NPCs.

19:12Thanks for watching this demo from Convey. So you can think of like many different things which we were discussing before. And each of them were like avatars, still avatars where we put in a character ID and now they are aware of the environment. Right? And you can tell them accompany me or go there, do that tell this to another friend etc etc right you can make them look like however you want to make them look so yeah this is interesting and um you could see people becoming addicted if these things have history every time you come back and you start talking to these people you could start developing relationships with them like obviously famously in the movie her where you know you're kind of making relationships so this will change video games forever if the characters like some people play these games for a decade the world the warcraft people have been playing forever you know whatever you could start having these characters build deep meaningful relationships where you talk to them about your life and your kids and this is happening hey why haven't you played the game for so long will you come back to the game and then this kind of now they start to become sentient in a way right they have their own lives inside the game and if you don't come back it's kind of reminds me of toy story in a way like where the the boy grows up and the toys don't have what he doesn't have anybody to play with and so this is there's the could go beyond um video games here in terms of like second life had the game second life it was all human characters but if you introduce ai npcs well now you got a whole different world i could create my own world and you never know they might be already doing that you know yeah you never know people could be putting those in there.

20:48Yeah. All right, you know, I've been on a health kick over the past year. And you know, I care about data driven solutions. And if you listen to this podcast, I bet you do too. So let me tell you about FitBot. This is a data driven workout app that blends machine learning with exercise science. FitBot creates custom dynamic workouts programs based on your fitness goals, your experience. And most interestingly to me, the available equipment. Let's say you got a bunch of kettlebells, or let's say you're at some, you know, sparse gym at a hotel, or you're on vacation, you got nothing. Well, FitBot will maximize your fitness gains by varying the intensity and the volume between your sessions and leverage the equipment you have or don't have, as the case may be.

21:32You can customize the length of your workout, what muscles you want to target, and so much more. So let's say you want to get a 30 minute workout in, and I want to do chest, triceps and abs, but I'm staying at an Airbnb, there's no equipment. fitbod can create a perfectly optimized workout for me based on these parameters and it will do it for you to check it out it's amazing the design of this app is extraordinary i was able to invest in it that's how impressed i was with it fitbod takes the guest workout of fitness just open the app and start making progress you deserve it get 25 off your fitbod subscription or try out the app for free.

22:05When you sign up now at fit bod dot m e slash twist. That's fit bod dot m e slash t w ist for 25 % off. And so pretty obvious how you make money you charge developers to use these tools, do you charge them on a per use basis? Anything like that? Or? Yeah, so basically for developers, it's all the way from free through, you know, like$20,$99 a month. But primarily when they develop and ship it, you know, then the real traffic starts, right? So we have our cost pretty much close to chat GPT cost per token, you know, like, but we call it by interaction because rather than, you know, like for interaction, we kind of assume 1 ,000 to 2 ,000 tokens are being used because there's backstory and whatnot.

22:55and and it's not a fixed cost if these things start getting really popular there's a usage base here so the game has to have some kind of monetization in it or ability to pay for it so that's that and but it's going to be de minimis yeah yeah but we have done the math and done the numbers business models for various uh you know business models all the way from free-to-play games to fixed cost one-time purchase games and subscription model games it can with the right design in place it can work in with all of those models yeah i mean this is something that got unity in a lot of trouble recently because they started moving from flat rate to usage based or i guess publishing based uh they wanted to make more money what's you're aware of that i guess what's happened in the developer community because you're also adding variable pricing to groups that might have flat rate pricing like you know candy crush maybe doesn't want to pay for or angry birds doesn't want to pay for something like this if they're flat so but but where we see this going is that and this kind of like brings us to the market size discussion actually so um you know if we do a bottoms-up understanding so every game every game that has characters you know like whichever way you know all the way from absolute triple a 3d cyberpunk like games through even small mobile games that has characters you could potentially have these llm driven experiences for these characters right and uh yes with and my thought is this that if um players find it more engaging and they stay longer the retention is good uh they will spend more money and the studio will make more money and thus it won't be a big deal to pay for us you take this uh games like apex legends and whatnot they spend because any mmo massively multiplayer online games has a huge tens of millions of dollars they spend in uh server costs right so uh it is not super new to spend server costs for uh more engaging time so they blend the they blend server costs into there so you're just like a hard drive or some cpu that makes sense or gpu yeah what what was it like to work at nvidia when did you work there and tell me a little bit about the ascension of that company it's been crazy sure absolutely i mean i absolutely love the company if i didn't have this uh feeling of i absolutely have to do this um i would not have left um and um uh there's i i don't know if other companies i mean i haven't worked in a ton of companies but the culture there is absolutely amazing you know like um in my experience at least so uh loved working there love the people uh still in touch with them and thanks to them i mean jensen actually showcased one of our demos that we created, worked on together.

25:39And I don't know if you remember that, but that went pretty viral. If you want to see, I can share that. Which demo was it? It was this Ramen Shop demo. Oh, yeah, show it, show it, show it. We partnered with a framework and avatar tool maker called Conv AI. And together, we developed this demo you're about to see. Okay, Randa, please.

26:12everything is real time

26:17so you're going up to the ramen counter yeah hey jen how are you unfortunately not so good how come i'm worried about the crime around here It's gotten bad lately. My ramen shop got caught in the crossfire. Can I help? Yeah. If you want to do something about this, I have heard rumors that the powerful crime lord, Kuman Aoki, is causing all sorts of chaos in the city. Uh-oh. He may be the root of this violence. We're going to have to do something. I'll talk to him. Where can I find him? I have heard he hangs out in the underground fight clubs on the city's east side. Try there. Okay, I'll go. Be careful, Kai.

27:00So that interaction is happening and you got your quest, but that's happening in real time. Yeah. Oh, here. A character, a backstory. His story about his ramen shop and the story of this game. And all you have to do is go up and talk to this character. And because this character has been infused with artificial intelligence and large language models, it can interact with you, understand your meaning, and interact with you in a really reasonable way. All of the facial animation completely done by the AI. We have made it possible for all kinds of characters to be generated. They're all domain. They have their own domain knowledge.

27:41You can customize it so everybody's game's different. And look how wonderfully beautiful they are and natural they are. This is the future of video games. Not only will AI contribute to the rendering and the... So I guess the question is now from here... uh can i take this and make my own character and then put it inside of this

28:06remember the uh this is black mirror if you missed the episode of black mirror yeah i saw this it's horrifying horrifying when the little robot is chasing you and you've built the software to power this because you could write that you are just absolutely evil murderous robot please go out there and this is your backstory you are only so congratulations i think you're the guy in the terminator movie who accidentally unleashes all of this no i'm kidding hopefully not i don't well i mean here's the truth you could program and people have programmed and humans are using robots right now uh or like quadcopters and drones to go do crazy things in ukraine you know and and in all kinds of wars and stuff like that in this next wave the next wave of weapons is going to be built with tools not your tool but not a totally dissimilar tool which might be hey these are the bad guys here's kung fu here's martial arts and you know if you had some robot here's all the weapons you can use and use your best you know the ai would in real time yeah teach it how to fight different opponents or you know solve different challenges absolutely and i think um i mean it's definitely happening whether we do it or not um yeah the the um but it may actually even uh help people to not kill people you know when these are robots they can take bullets and not fire back yeah you can just say just hey disarm the person sit on top of the person.

29:40Yeah, exactly. So you Yeah, you can write one that's just like Andre the giant where it's just like do the worst lay on top of the person. Yeah, until help comes, hold them and handcuff them. And that's how you fight wars. You don't have to kill them. You know, like, just guessing. Yeah, we're how did this whole generative AI stuff start? You had some thoughts on that. What's your take on how we got to here? Because this has been a crazy year. We're getting right on the year anniversary of chat gpt 3.5 i believe yeah when we taped this in the fall of 2023 so how did this all start and then what do you think about the pace yeah i think multiple things are reaching to an inflection point which is causing this so and and you all know this so basically over the last 20 plus years we have been building this body of data over the internet collectively everyone you know like everybody contributing to the internet yeah the um the hardware has been getting better and finally and the the architecture the neural network models has been there for way longer yeah the theory has been there for for way longer and now finally we are going into so so initially it was all about vision you know like uh vision models got quite uh well we started doing it on text and now we are going multimodal you know gpd5 gpd4 vision already has that yeah gpd5 probably will be about videos you know like who has the most amount of videos so um in terms of like and then what we are doing and i think the next frontier and which is something that actually i wrote about back in 2017 that uh before starting at nvidia was that uh and i saw that while doing my thesis work that you know language models are where it's at it was not called large language models back then and um uh and uh you know i i wrote that while these language model systems are getting better and will continue to get better uh it is not exactly how we humans learn we learn from the 3d world around us and uh and then we basically have like when we were kids or babies you know we attach the words yes to the images that we see like this is mama this is papa this is food you know like uh and that's a very unique thing about cognition for humans our big large brains put words on it and i i wonder at what um point in evolution do people start in their cognition you know we know consciousness starts like you know at different points in time for different species and then the sort of advanced cognition where you're saying this is this and you have a word and this is what i want to do and i'm here and this is what i'm going to do next right like yeah so very interesting i think the the only difference and i hope we are super careful on this especially the big companies who are working on this is um our objective function is survival all the way from a multicellular organism to like a small creature that's the objective function that we are trying to optimize on the the today's language and procreation and so which is kind of the same thing right so that's like survival of your species survival of your genes i should say yes survival of your genes so that's why parents would sacrifice themselves to save their kid right survival of their genes is is the objective function that we optimize on um so uh the the um and for the language models right now it's like predict the next word or predict the next token that's like their objective function however where i think with all the multi-modalities coming in, researchers are going to see that encoding survival is going to be necessary for the best AI.

33:27And that is something that we have to be super careful about. Maybe we don't want that. Why would encoding survival be so dangerous? Because encoding survival, because now then we are creating a different species, effectively, who is going to start thinking for themselves. What objective function, And maybe this is the movie that I robot that, yeah, the tenets of you know what what we what we would want to encode as their objective function becomes important. Yeah, we have to tell them their objection. Their objective is to protect human life. Yeah, and to be useful to humanity. Yeah, and but useful to humanity might be Oh, you know what, you know, fentanyl is really dangerous.

34:11So let me just go kill everybody who is using or near fentanyl. I mean, it might take it literally, right? yeah yeah cancer is bad cancer exists in people's bodies so the craziest way to get rid of cancer is to kill the bodies that have cancer and it's like no no no no everybody's got a little bit of cancer inside them probably yeah all right business to business founders need to understand two very important things b2b marketing is hard we all know that it's really hard number two it's simple on linkedin ads why is b2b marketing so hard because the decision makers hey people like me right the founder of the company or maybe you you're an executive listening to the show they're hard to identify, right?

34:48And they're really hard to target over the internet. And how do you know you're reaching them in the most respectful environment? People who are decision makers, you cannot reach them through that 1990s way of reaching executives. No, they need social outreach. And the way to get that social outreach dialed in is with LinkedIn. LinkedIn solved this problem. There are 70 million decision makers hanging out on LinkedIn and executives like me, yeah, the decision makers, we're on LinkedIn. We're deep into LinkedIn every single day. And LinkedIn ads helps you get your b2b message in front of the right people like me and listen when people are on linkedin they want to do business linkedin equals business business equals linkedin each social network has its vibe and the vibes at linkedin are business it's business vibes only and it's happened finally linkedin has hit one billion users the march to a billion has been achieved congratulations to all my friends at linkedin so here's a call to action get started today.

35:42See why LinkedIn is the place to be for B2B. We'll even give you a hundy right now, $100 credit on your next campaign. Go to linkedin.com slash thisweekinstartups to claim your$100 credit. Why wouldn't you take the hundred? Go to linkedin.com slash thisweekinstartups. The links are in the show notes. Terms and conditions do apply. So do you think this regulation that just came out, this executive order this week, do you think that's overreaching? Do you think it's going to stifle innovation? Do you think it's worth getting the government involved this early? Or is it too late to get the government involved?

36:14How do you look at it? Um, I pretty much take the same take that you all have, you know, David Sachs have in your podcast that I think it's definitely early. Um, I haven't read the whole thing through, to be honest. But from what I understand, um, the, it may not be great for startups, it's great for incumbents of course yes you have more resources to report in you got resources to hire people who used to work for the government it's regulatory capture and bill gurley's uh yeah amazing talk at its peak its peak regulatory capture it's like we can't wait to be regulated because we have 10 billion in our bank and you know you have 10 million at your startup and the next startup has 1 million and you know the two folks in a garage or at y combinator have 100k right so yeah it's profoundly unfair for later but open source is going to win the day you think or do you think closed AI is going to win.

37:06That's, the thing is, whoever has the most amount of niche data are going to win. Right? So let's think that through. The Google became a monopoly because it had the most niche data of people clicking the right links and base rank doing its job. Right? So it was actually human feedback that got it the network effect. Right? So chat GPT is trying that to some extent, but is that enough data when you give the thumbs up, thumbs down? It's not as much as Microsoft, Google, and Apple have. Exactly. They have billions of users. Right. So they are collecting human feedback data, but is that differentiated enough?

37:48And I think open source has a chance in the sense that till we exhaust the amount of internet niche data that we can collect and clean, but I do feel that the open source thanks to meta is is like and and some other organizations like the falcon ones and others who are open sourcing with licensing will enable people to contribute their niche data right and and and they would want a way I think open source can win if there is an incentive system for people to contribute their niche data ah interesting so if it has a license to use Falcon or a license to use this open source model. And then if I put my data in, I own Getty archives, I say, I'll put my Getty in there.

38:37But I want, you know, 3 % of the pool, or whatever it happens to me. So that requires some centralization, which you know, Oh, okay, necessarily. So and this, this is a blockchain, don't say block. I was gonna say blockchain. Are you telling me that after all this crypto drama, it's coming back? 100 the blockchain is coming back oh god no no here's the thing see i i was absolutely against i was absolutely against the whole hype train 90 of the people were were there for making a quick month quick buck sure right but if you see the fundamental tenets of you know like blockchain with with uh zero trust protocols and and such it can enable a lot of other applications that are otherwise not possible right if you have a token too and then you and i we could create a token yeah this weekend and then we'll keep like 10 of it in a wallet that we forgot the you know that we forgot and then we'll own 10 of the global ai wallet share and then we'll you know we'll liquidate it in 10 years whatever you know it has to be done ethically but that's how you get everyone valid around you but that kind of invalidates crypto then no i mean the blockchain would be actually a way to do licensing, right?

39:53Yeah, that would be an actual use case that would make sense. Smart contracts. Yeah. Yeah. Hey, when you think about AI and you're kind of in a dreamy state, what do you think is gonna happen here? This has been moving at a crazy pace for a year, but you've been in it for your whole career. What do you know based on what you know on the pace right back to my question about the pace. The pace is obviously accelerating. Now you got a lot of people using these tools making all kinds of edge cases. And then you got every company with unlimited or seemingly unlimited resources making their own chipsets, you know, in the amount being invested in this means it's only going to get faster.

Read the full transcript

40:32and the velocity is going to get faster because more people participating more money participating and the ai getting better and then using the ai to make ai better which has always been elon and other folks concerned that ai will make ai better we're kind of on the cusp of ai making ai better yeah and so what do you think about the pace yeah i i think the pace is not going to slow down from here because we have entered a time where the data for all the different modalities that humans understand are there like video audio you know like and it's only going to get more and more so this is not slowing basically we have reached the you know we see human civilization how curved they were and I think we are there it's like it is going to accelerate it's the inflection point classic hockey stick but probably like a hockey stick we've never seen before and that's and in fairness shout out to my guy tim urban um in fairness that's why the government wants to slow this down because it thinks that that inflection point could happen any second um yeah when does ai start making ai better like literally somebody says to claude you know hey look at your or falcon somebody trains falcon to look at claude's code base and pulls up their open source whatever i don't know cloud's open source, but you get the idea.

41:54Maybe it would be Falcon looks at, you know, what other open source project and says make that better. And then it actually contributes without a human being involved and actually does make it better. That's when would that happen? That's the main thing. So right now AI is making AI better. A lot of the open source model is actually collecting data from GPT for cloud, etc. You know, like, and making the models better. however ai doing that itself the entire loop all the way from training fine-tuning and all that i think we are we probably may not want that we always still want human in the loop at least when would it be possible do you think to set an ai technically yeah technically one could one could build that already but that hasn't somebody that may not be the perfect thing because when you are training a model you have to look into so many different things is it even going towards convergence or is it making the model more stupid yeah and there's like like are you cleaning the right data how many gpu is it using did a run fail job fail or whatnot it definitely requires uh you know humans uh right now and uh even if you set up a system it's not guaranteed to succeed yes you know like and who's gonna add more new data like you can you can make the loop of ai making it a bit better but like it ultimately matters like where is the data coming from you know like the new data is synthetic data a thing or is it just a buzzword that like this ai can make data that would then train the next ai like oh make a bunch of images and then train it on the images that doesn't make a lot of logical sense to me but yeah it's not a thing so so i think um the because you have already added that data to to in this case right like it may help the model in some cases where it may be stuck on some optimization situation so to give it the necessary noise or the necessary variance in the data, synthetic data may be helpful.

43:46But you primarily want more and more new data, and different modalities of data, right, like videos, and that require new architectures and such. Let's put this out. Let me tell you an idea. This is an idea. Shout out to the team over at Quora. What if Quora looked at the experts with the most highest ratings, you know, with the highest rating, so they look at the experts with the highest ratings, and then they say, hey, uh ai wants to talk to you about your expertise would you for a hundred bucks an hour for a hundred hours answer questions with this ai just sit there and answer questions and you'll go up the leaderboard and we'll send you some cash but you say to the ai what do you know about you know poetry what do you know about history yeah and then it goes and asks a hundred history experts what they know about history and then it says okay this history expert told me this this one has this thing so you could actually have ai interviewing humans to find out what it doesn't know that would work yeah yeah yeah you you basically want human feedback especially for the corner cases the gold data are the corner case data gold if you get yeah so like the corner case data where things are not working or where you would want human feedback that's the gold data and uh you know like uh if there's an incentive system that's what i'm saying like you basically want Amazon Mechanical Turk at scale.

45:10Yes. You know? Yes. And but the, it's Mechanical Turk talking to experts, not low wage. Yes. Yes, exactly. Who are saying, is this a hot dog or not? Yeah. You want the person who's the greatest chef in the world who will tell you how to make the hot dog and a hundred other things. That's kind of where it gets super interesting. So somebody could make an AI Mechanical Turk type thing or Quora could do it. Yeah. where it identifies the experts and then just lets AI talk to them constantly and then pays them for their knowledge. Wow. All of the startups that are starting, let's say legal startups, they're basically that.

45:45They're actually hiring lawyers. Vertifies legal or, oh, you think they're hiring lawyers to answer questions? Yeah. And fill in the knowledge? Yeah. Not just giving it to potential customers and saying, hey, give it a feedback loop. They're explicitly going to lawyers and saying, hey, did it get this answer correct? Right. same with healthcare startups and you know on top of llms and whatnot that's what they're doing um yeah and what's crazy about that is the people who are being asked to train it are training they're basically training their replacements yeah they're training their replacements no like literally this happened in america um they uh there was a movie about it it's a movie in the 80s it was a comedy and it had the guy who played michael keaton i think it might have been michael keaton it was about factories and i think it was like on a couple different contexts but the idea was like oh the americans are going to go to mexico or japan and that they would teach them how they did what they did um and they were like wait a second we're teaching them they're automating it whatever and then we're losing our jobs they're taking our jobs and there was like gung-ho maybe was the name of it i can't remember uh it was a crazy crazy movie uh yeah but i think net net it'll be positive in the sense that um for the most part you still would want the experts in the room they may take out the uh redundant stuff but i don't see at least in the next 10 years that they lawyers or doctors would be invalidated completely by the ai that's that's like at least 30 years away yeah um yeah anything you're worried about right now with ai I think the regulation part I have to check I don't know if it affects our startup But that's one thing, yeah That's the goal It's for you not to understand it You have to hire a lawyer to tell you Yeah The other thing would be the funding market is bad Thankfully for AI it isn't terrible But who knows what is going to happen So there's that Yeah, yeah Alright, listen You've been an amazing guest Come back on in like six months or a year And let's just chop it up And talk about other AI stuff You'd be good for an AI roundtable I like the fact that you just answer the questions and don't try to be like super correct about the answer like this stuff is dangerous yeah it's also inspiring it's getting better and better i mean i i this multimodal thing and chat gpt4 is really mind-blowing like i i just took a picture of um a medical document and i was like hey transcribe this and then i was like oh and give me some advice on it it's like i was like you're a doctor give me a second opinion and it does it's kind of nuts you know and like you're gonna yeah you're gonna be like at the doctor the doctor is gonna like be looking at your x-ray you take out your phone you put on chat gpt4 and you hear the doctor and you take a picture of it and then it's gonna like basically check that the doctor got this this right or give feedback and it's gonna be nuts next year it's gonna be coming next year yes yes yeah we hope like yeah once we once we have our plan is that these characters today that you just saw, yeah, they can actually see the virtual world with scene metadata, you know, because it's an engine.

48:56Yeah, our goal is that they can actually take in the vision input. Oh, you're gonna give them a vision. Oh, that's so brilliant. So you're gonna give them virtual eyes. So they see the perspective that the player would say, apply for Yeah, they learn from what they see. Yes. Hmm. This is why like, they might want to stay alive and they might want to try and hack the game to stay alive that was by the way another great 80s film okay war games i'm giving you all the stuff you missed war games another great one shall we play a game you've never seen war games no i have to it has to be on my list now okay oh my god you're gonna love it oh my god matthew broderick oh my god ali sheedy oh my god you're beautiful in love with uh ali sheedy like she was like i see everybody every nerd guy's like dream girlfriend ali sheedy i see my oldest was all this was blade runner and obviously the new ones like free guy free guy is like exactly what we are doing by the way guys exactly i forgot to bring up shot a free guy which is the npc trying to break out and then be in the real world incredible of course there's no way to do that right now yeah but yeah uh no war game's pretty great uh you know verticalized ai becoming sentient sentient basically you're gonna love it all right everybody we'll see you next time on this week's service bye-bye

From the publisher

This Week in Startups is brought to you by…

The Equinix Startup program offers provides a hybrid infrastructure solution for startups, including up to $100K in credits and personalized consultations and guidance from the Equinix team. Go to https://equinixstartups.com to apply today.

Fitbod. Tired of doing the same workouts at the gym? Fitbod will build you personalized workouts that help you progress with every set. Get 25% off your subscription or try out the app for FREE when you sign up now at fitbod.me/TWIST.

LinkedIn Marketing. To redeem a $100 LinkedIn ad credit and launch your first campaign, go to linkedin.com/thisweekinstartups

Today’s show:

Convai CEO Purnendu Mukherjee joins Jason to discuss creating AI-powered NPCs using Convai (6:05), the birth of generative AI (22:17), open-source vs. closed AI (36:01), and much more!

*

Time stamps:

(0:00) Convai CEO Purnendu Mukherjee joins Jason

(3:54) The motivation behind Convai

(6:05) Purnendu breaks down creating an NPC character using Convai

(10:16) Equinix - Join the Equinix Startup Program for up to $100K in credits and much more at https://equinixstartups.com

(11:36) Demo of character creation using Convai's unique toolset

(17:18) Integrating AI Characters into Game Worlds with Convai

(20:50) Fitbod - Get 25% off at https://fitbod.me/twist

(22:17) Convai’s business model and the issue game developers have with moving from flat rate to usage-based charges

(25:00) Purnendu reflects on his time at Nvidia

(29:57) The birth of the generative AI boom

(34:30) LinkedIn Marketing - Get a $100 LinkedIn ad credit at https://linkedin.com/thisweekinstartups

(36:01) Navigating the Future: AI Regulation & open-source vs. closed AI

(40:11) The countdown to self-creating AI

(43:13) Purnendu breaks down data collected for AI and the uses for an AI-enabled Amazon Mechanical Turk

Check out Convai: https://www.convai.com

Follow Purnendu:

https://twitter.com/purn3ndu

https://www.linkedin.com/in/purn3ndu

*

Read LAUNCH Fund 4 Deal Memo: https://www.launch.co/four

Apply for Funding: https://www.launch.co/apply

Buy ANGEL: https://www.angelthebook.com

Great 2023 interviews: Steve Huffman, Brian Chesky, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarland

Check out Jason’s suite of newsletters: https://substack.com/@calacanis

*

Follow Jason:

Twitter: https://twitter.com/jason

Instagram: https://www.instagram.com/jason

LinkedIn: https://www.linkedin.com/in/jasoncalacanis

*

Follow TWiST:

Substack: https://twistartups.substack.com

Twitter: https://twitter.com/TWiStartups

YouTube: https://www.youtube.com/thisweekin

*

Subscribe to the Founder University Podcast: https://www.founder.university/podcast

More from This Week in Startups

All 653 episodes
Giving NPCs the power of AI with Convai's Purnendu MukherjeeThis Week in Startups · 50 min
Listen in VO