Stanford & Google’s Westworld-like AI experiment, BloombergGPT, SF CRE collapse | E1718

0:00Hey, everybody, we have a ton of news for you today. First up, there's a big story out of Stanford. researchers have released a paper in which they basically created Westworld, if you've ever seen the show or the movie, or read the book by Michael Crichton. Inside a Sims-like sandbox video game, they created a bunch of NPCs, had them talk to each other, emergent behavior came out of that. We're going to talk about what Google and Stanford researchers just released. It's pretty, pretty wild, folks. It dovetails with simulation theory and Blade Runner as well. Then we're going to talk about the commercialization of large data sets.

0:34Bloomberg has announced their own chat GPT competitor, essentially their own LLM called Bloomberg GPT. And we'll talk about what this means for financial analysts and figuring out market sentiment and how these data sets are going to be verticalized. And will it get rid of tons of jobs or just augment people, automate people or deprecate jobs? We're going to talk about that very important issue and that transition from augmentation of human intelligence and performance, automation of human performance, and then deprecation of humans themselves. Finally, two quick hits on San Francisco commercial real estate.

1:11Turns out some people have some interesting ideas about transforming Soma into a new type of hub. I think it's a brilliant idea. It gets Mayor Jason's endorsement. An Australian startup has blown through$75 million in one year and shut down. It's in the 20-minute instant delivery space. And so we'll talk about founders creating things that maybe consumers don't need slash are unwilling to pay for. It's going to be a great solo dolo show. Please stick with me and us. This Week in Startups is brought to you by LinkedIn Marketing. To redeem a free$100 LinkedIn ad credit and launch your first campaign, go to linkedin.com slash thisweekinstartupspilot.

1:53Grow your business sustainably and operate more effectively. Pilot provides the most reliable accounting, CFO, and tax services for startups and small businesses. Head to pilot.com slash twist and get 20 % off for the first six months. And House of Macadamias is the next big health trend. Get 20 % off your first purchase and a free bottle of cold-pressed macadamia oil at houseofmacadamias.com slash twist by using code TWIST20. Hey everybody, welcome to This Week in Startups. This AI news is coming fast and furious. It seems like everybody is making tremendous progress every day, and so we got to keep up with it here.

2:37Today, researchers from Stanford and Google did an interesting experiment with NPCs, non-playable characters in a sims like video game environment again non-playable characters is what npc stands for if you play video games you you know this because they're the people who you know in a video game might be like the shopkeeper they sell you armor or something like that and the term npc has become an insult online for somebody in social media maybe who you know doesn't have really intelligent takes they're not important in the world putting that aside that people have taken the video game term and applied it to the real world.

3:12Let's talk about what Google and Stanford released today. It's a paper that details this experiment they conducted. The title of the paper is Generative Agents, Interactive Simulacra of Human Behavior. Interactive simulacra. Simulacra are defined as, quote, copies that depict things that either had no original or that no longer have an original. Basically, these are like Sims characters, if you ever played that game, and they're in a sandbox, and they named the sandbox Smallville. Here's a screenshot, if you're watching us at youtube.com slash this week in. It's a small 2D village. It sort of looks like an old Pokemon Game Boy game.

3:552D, you can walk around. Here's a supply store, the park, houses, a college dorm, and you've played these 2D simulations. And inside the sandbox, the researchers created 25 characters, agents is what we might call them, just like you might consider a chat GPT and agent, you know, you talk to it in a chat format. So these were trained to talk to each other. And then they were trained to have memories of those conversations and then reflect on them. So the researchers used this architecture that stored and blended and applied all these memories. Okay, so now we're getting into like Blade Runner territory, right replicants memories, are they real, etc.

4:34In order to generate behavior that was believable, they used a large language model, right? Large language models are what OpenAI and other folks are doing. The researchers in this case connected their architecture to OpenAI's GPT 3.5. And amazingly, they weren't even using GPT-4 to use this. And GPT-4 is just a magnitude better. But this is kind of a simple concept here. The architecture was comprised of three main components, according to the paper memory stream, a long term memory module that records in natural language, a comprehensive list of the agents experiences, okay, their memory stream reflection, which synthesizes memories over time, enabling the agent to draw conclusions about itself and others to better guide its behavior.

5:14And then three planning, which translates those conclusions and the current environment into high level action plans, and then into detailed behaviors for action and reflection. So this is just like if you've seen some of those simulations where you're like, hey, write me a movie in the style of Quentin Tarantino where five people are bank robbers, they're collaborating on robbing a bank and things go wrong and give me the dialogue. And then at the end, three of them die. You know, you say do it in this voice. And it's kind of similar to that, right? Except they've organized it into memory, reflection and planning.

5:47Really interesting stuff. stuff. So these reflections and plans are then fed back into the memory stream to influence the future behavior. Just like when you're using chat GPT, you have that thread, and it organizes your conversations with the language model. So you can pick up where you left off and the conversation isn't starting from zero every time you do a prompt, or, you know, what we previously would call a search on Google, these prompts now have a history to them. And the history impacts what happens in the future. So here's a graphic they shared of how the memory architecture works. And so what we're seeing here is there's a memory stream that you can retrieve memories from, you perceive the world, then it goes into your memory stream, then you retrieve memories, and then you can go act.

6:32But retrieve memories could build a plan that then goes back into the memory stream, you reflect on it. And I guess the idea is that this would create more high level actions or more interesting behaviors in the individual. So what people are starting to realize is with chat GPT four, with chat GPT 3.5 with Dolly, and with creating simulations like this is, well, maybe what we do in terms of consciousness and pulling memories from our brain is not all that sophisticated. It is incredible that we can, you know, I could say hickory dickory dock and you do the next line, right? The mouse ran up the clock, or I could say, you know, a bird in the hand, and then everybody's mind right now, hundreds of thousands of people hear this, will say, you know, it's worth two in the bush.

7:20It may not actually be that special how our brains work. And so this is what we're starting to realize is, we have this corpus of data known as the internet, known as every word ever written by humans is now online, not in books in the library. So when you have it all stored, and then you build these language models, we can start to predict words, we know what tone is. And so now all of this is coming together. And that is, I guess, you know, as impressive as this is, what we might actually be learning from AI and these language models is how unimpressive we are as humans. And what we thought was this incredible vocabulary we all have and this ability to communicate, it might not actually be all that special.

8:09It might not be. And maybe when we hear a Bob Dylan song, or a poet, or a great writer, we're like, wow, that is the pinnacle. But it seems to me like these language models are going to be able to deconstruct how that happens and be just as impressive as us. So after they created the memory architecture here, they gave the 25 agents identities, and then they gave them a one paragraph identity description in natural language. This included things like the agent's job, their family relationships and other seed memories. This is serious Blade Runner stuff where they incepted memories in these four year lifespan replicants.

8:43For one agent named John Lee, they said John Lee is a pharmacy shopkeeper at the Willow Market and pharmacy who loves to help people. He is always looking for ways to make the process of getting medication easier for customers. John Lynn is living with his wife, May Lynn, who is a college professor and son, Eddie Lynn, who is a students studying music theory. He loves his family very much. He has known the old couple next door, Sam Moore and Jennifer Moore, for a few years. John Lynn thinks Sam Moore is kind and nice man. John Lynn and Tom Marino are friends and like to discuss local politics together.

9:13So after creating the sandbox and 25 agents, then giving these agents identities and memory architecture, they let them loose. They basically hit the play button on the game. The agents then started sharing information with each other. In other words, instead of us writing prompts like we do in chat GPT-4, they said, well, you know, the pharmacist likes Sam Moore, and he thinks he's a nice man, and Tom Moreno and him are friends, they like to just go out to local politics, so they gave them prompts. And so, once you get them talking to each other, it would be, you know, like the joke when people put two people on a conference call together, they say, hello, hey, who's calling?

9:49And the person who puts the two people on the conference call, you know, a pizzeria and a Chinese food restaurant, you know, and they start talking to each other, and it just creates confusion, but a dialogue emerges. It's kind of like those kind of pranks. So this is like a prank being played on the agents. They basically got them started. Will they stop or not? Okay, let's talk about marketing to senior level executives, not the rank and file, the senior executives, the people who make the purchasing decisions, the people who okay the purchasing decisions. Well, when you're selling something business to business, B2B, you need to market to the decision makers.

10:25The rank and file, they're probably using your product, they got the free version, but you need to get those decision makers on board and it's hard to find them. You're not going to find them on other social networks. And if you do find them on other social networks where people are dancing or arguing about politics, it's the wrong place to meet them. You want to meet them when they're thinking about business. Business equals LinkedIn, LinkedIn equals business. And LinkedIn has 180 million senior level executives and 10 million of them are C-level. LinkedIn ads is specifically built to reach those folks and they have a platform that nobody else has.

10:56You can reach those eyeballs when they're thinking about business. You can reach them by title, you can reach them by company size, geography, etc. And because people on LinkedIn are doing business, they're ready to receive your message. LinkedIn equals business, business equals LinkedIn. You know that. Audience is exposed to brand messages on LinkedIn are six times more likely to convert above average. So here's your call to action. Make B2B marketing everything it can be and get$100 credit on your next campaign. A hundy waiting for you at linkedin.com slash thisweekinstartups. No spaces, no dashes.

11:28linkedin.com slash thisweekinstartups. Type out those letters, you get a hundy. Well worth it. You're getting like 10 bucks for every character you type out. Terms and conditions apply because they're giving you a hundy. So the people who constructed this would also give some prompts to get this whole craziness going. For example, one agent named Isabella was prompted, this is by the humans who created the test, to throw a Valentine's Day party. So quote from the paper, starting with only a single user specified notion that one agent wants to throw a Valentine party, the agents autonomously spread invitations to the party over the next two days.

12:00The agents made new friends, asked each other on dates to the party and coordinated when to show up at the right time. Isabella invited 12 other agents to the party and then seven agents did not show up for the party. Ooh, drama. three of them had other plans and the other four didn't show up for no reason at all quote from the paper rajiv the a painter explained that he was too busy he said i'm focusing on my upcoming show and i don't really have time to make plans for valentine's day yeah he's a grinder you don't have time for parties i like it just like human behavior so here's a graph uh the diffusion path for isabel's valentine's day party you know she says i'm planning a valentine's day party at Rob's cafe on February 14th from 5pm to 7pm.

12:39And she, you know, asks everybody to come and then they all have conversations with each other is a fascinating idea here. It really does open up this Pandora's box of what happens when a bunch of agents, a bunch of verticalized AI start talking to each other, they're given plans. And this kind of dovetails with simulation theory, some people believe we're living in a simulation and that we all are these characters we're isabella and that some being out in the wider universe created what we perceive as the universe the planet earth and we're all in our own simulation this is simulation theory and so as you watch this as you see video games emerging grand theft auto sims gpt4 keep in mind and you've seen the movie the matrix I wonder if we ourselves are characters in a simulation.

13:29And these agents also form new relationships and remember them without prompting. So Sam does not know Latoya Williams at the start, but while taking a walk in Johnson Park in the simulation, Sam, the character, the agent, ran into Latoya and they introduced themselves. And they mentioned she's working on a photography contest. In a later interaction, Sam indicates a memory of their first interaction, so it gets real, and he said to Latoya, the agent, hey, how's your project going? Latoya replies, hey, Sam, it's going well. This was not prompted by humans. So, once they got this thing going, they said, you know, when you see somebody, say hi to them, and, you know, it just started reflecting on what they did with each other.

14:11So, they started coordinating with each other. This gets really weird. They gave Isabel and Maria two pieces of information. Isabel, you will throw a party. Maria, you have a crush on Claus without any further instruction. Isabella invites people to the party, decorates the venue and asks Maria for help. And Maria invited Claus to the party. Okay, an interesting wrinkle in all of this, the researchers could confirm that these interactions were not lucky hallucinations, like when they're just mistakes by the language model, because they could go back and check the memory logs for each agent. So they kind of knew how the history had emerged here, the agents memory streams that would include logs of them interacting, synthesizing and remembering new bits of information that they had generated.

14:52So is this artificial general intelligence? No, obviously not. This is a bunch of vertical AI is talking to each other. And then us starting to simulate what our interactions with each other are as humans, and then applying it to language models talking to each other. So how does this build into the real world? Well, this is similar to what Tesla is doing with their autopilot, They take all the real world data of actual miles driven, they drive all these miles together. And then when they're driving all these miles together, they find, I don't know, a particularly hard intersection. And then they say, okay, here's every piece of information.

15:30Here's, you know, 10 ,000 cars that drove through this intersection that, you know, is painted improperly, or maybe there's been like three accidents there that have been recorded by the city. let's run a million cars to this intersection. And let's see if a million cars do this, and we give them some parameters, different speeds, different levels of accuracy in the lane they're in, maybe, you know, they drift up to 20%. They change their speed up to 50%. Maybe a bicycle goes to the intersection randomly. Those simulations that Tesla is doing with their models is what makes their self driving so good.

16:06Because they don't have this is one of the interesting thing that's happening according to the reports out of Tesla. They don't have a lot of accidents. It's kind of hard to crash a car with a bunch of cameras on it that stops pretty conservatively like the cruise cars as well that are self driving in San Francisco. These things are very conservative. So they don't get in a lot of accidents. If they do get an accident tends to be somebody hitting them like a bicycle runs into them or a car rear ends them. It's not like they're blowing stop signs or red lights. They're designed to be super conservative.

16:35So how do you simulate accidents, you do it in a sim, which is what they're doing here. So what does it mean? It means video games are going to get a lot more interesting. You could create all kinds of unique video games, random video games, where the characters are just emergent behaviors. And you start to think, well, this is like Westworld. If you've seen the recent Westworld TV show, you know, they play certain scenarios over and over again, people come for a vacation, and it's a real world sim. So what if you took this, and you combined it with the animatronics the early Walt Disney animatronics in It's a Small World.

17:10I know this sounds dystopian and nightmarish, but this is what Michael Crichton envisioned for Westworld, which originally was a book and a movie that then became the TV series, which is people would go and interact in a world in real life, humans made out of, you know, biological materials with robots. So you take the Boston Dynamics robots you see flipping around, it's completely conceivable in our lifetime that you could have a west world like simulation where we go in and we play paintball or play a war simulation with those robots they're designed to not kill us of course uh but we could fight you know on the beaches of normandy in a simulation with boston dynamics robots that are developing or it could be something non-violent it could just be you could go to a simulation of a party.

17:58And of course, if you can do that properly, well, then maybe when we're old, there'll be a robot in our house that gets to know us. And in our dying days, instead of having a nurse or, you know, an aide come to our house, you could have this robot, again, like a Boston Dynamics robot, or Tesla's making a general interest, a general robot, they could be learning about us over time, they could be given the programming, be compassionate, have an interest and people ask provocative questions. They might read, they might listen to this podcast. They might read my biography someday or read every email I've ever written and then just start interacting with me with that as the backstory.

18:36Maybe they've read all of your social media. They've looked at all your Instagrams and this robot who's living with you as you are getting towards the end of life is like, hey, remember when you went skiing in Japan in 2023? I was, you know, watching some videos of it. Would you like to, you know, put on a VR headset and experience it again? because you seemed like you were really happy. It's just wild to think of what could happen here. This could be simulations of proteins in your body, right? Interacting with each other. We're moving into a space where I think we're going to close the loop in a very short period of time, a couple of years, you know, single digit years on kind of how consciousness works and how we and our intelligence works.

19:16And so if you've heard the Wolfram Alpha episode of This Week in Startups from a week or two ago, he kind of talked about this and so brave new world it's moving faster than i think any of us thought because this technology specifically what open a is ai is done which is one of the greatest i would say top 20 launches in the history of technology because that exists people who are doing this kind of research it's almost like they have an amazon web services they've got storage they've got compute power that they can just run experiments on and that's really the interesting thing that's happening here is because so many projects are hitting critical mass at the same time, the experiments are starting on third base, you don't have to build a bunch of compute, you don't have to buy a bunch of computers, rack them and run a natural language model against it, you're going to be able to pick from 100 different server farms with 1000 different AI models on it, and then start putting these together.

20:13In fact, this could be put together with mid journey, and they could create new characters, two of the characters could have a baby in this, and then it would generate a new image and build the character automatically. This could get very interesting. All right, everybody, in the early days of a startup, you need to be focused on product and customers, but you can't forget about getting your accounting dialed in. As an early investor, I've seen this so many times, so many accounting horror stories. I have people who are doing cash based accountants instead of accrual based accounting, they forget to do their taxes, they're paying for things improperly, or they're spending money that's not tax deductible, they're misclassifying expenses.

20:56And what happens? You're doing a fundraising, and it gets killed in due diligence, you need to get your accounting right from the beginning. So when you're a founder, you have to balance getting product market fit building your team, but also not forgetting to set up the proper accounting system. The good news is the perfect solution is waiting for you. Pilot is an amazing service that provides accounting, CFO and tax services for startups. Any problem you're having, they've seen it 100 times. They've seen it 1000 times. They've seen it 10 ,000 times. Let them handle your books while you focus on your startup.

21:30And then when you get big enough, no problem, hire a CFO, bring all your accounting in house, they understand they're going to help you as you get to the CFO level. When you get the CFO, yeah, CFO is going to take care of things. You know what a CFO costs? Do you know what CFOs want to work on? 20 million or more in revenue. Okay, so pilots got you covered. twist listeners can get up 20 % off for six months at pilot.com slash twist. What a generous offer. pilot.com slash twist for 20 % off your first six months. I think an interesting transition is to think about how this is going to affect business in the short term well bloomberg uh is building their own llm called bloomberg gpt two weeks ago bloomberg announced this llm large language model right guess the next word kind of a thing um and bloomberg as you know is uh one of the largest financial technology companies in the world terminals cost about 25k per year per user this is how michael bloomberg made all of his money but they also have tv shows like their news network they do journalism they write stories um and bloomberg has over 10 billion dollars in revenue according to different sources a lot of people use these their large language language model was trained on the company's own financial data and data from the web so it was trained on over 700 billion tokens gpt3 was trained on 500 billion.

22:56They did not disclose what GPT-4 was trained on for competitive purposes. But about half of Bloomberg's 700 billion tokens were sourced from Bloomberg's own financial data. So they have a lot of little pieces of information because they have trades, right? They have every piece of data on gold and when they traded at what. It took about 53 days, Bloomberg reported, to train Bloomberg GPT. 53 days of computations run on 64 servers, each containing eight NVIDIA A100 40 gig GPUs. One of these GPUs, these NVIDIA A100s that people talk about, they cost about 12 ,000. And so they're very expensive. And that means each of the servers cost about 100k.

23:43You multiply 100k by 64 servers, you know, it costs about 6 million in hardware just to get this up and running uh and this is why nvidia stock is going crazy it's doubled in the last six months because people are realizing hmm uh there is actually a market for these gpus at scale and everybody's going to buy hundreds of them and run data against them but you know it's a de minimis amount of money for a company like bloomberg obviously um and so the financial data sets were 54 of what they did so that's really interesting uh now you know chat gpt if you were to ask it to analyze stocks and trades where does it get its data from like probably like news articles on the web maybe not the granular data in data sets that bloomberg has and also bloomberg has proprietary data so you can be sure bloomberg is not going to allow google barred or bing or chat gpt or anybody else to use their data if you want that data you're going to get access to it only by having the you know 25 000 a year thing so um public company filings which i'm sure everybody will have access to because they're public companies sec that was two percent of the training data press releases from public companies was one percent or so So it's really sort of interesting when you think about it.

25:05Public data sets were the other 48%, right? So private was 52. And the pile is an open source data set that has been used to train multiple GPT LLMs in the past. So the pile, which I've never heard of, is a public data set, I guess. And C4 was 20 % of the training data. C4 stands for colossal clean crawled corpus. It's an open source common data set that's been used to train LLMs. And Wikipedia was 3 % of it. so there are public data sets that everybody uses and i think wikipedia deserves a lot of credit here because wikipedia really structured a lot of data that is uh i think by definition allowed to be used by anybody because of their creative commons license as long as you give credit and i think you have to cite so i think that's going to be one of these legal issues that'll be worked out over time it'd be so what's this all going to be able to do well you might be able to ask it, you know, which stocks were, which companies were doing particularly well 10 years ago, went through a management change and are doing better now.

26:08And explain how volume has changed over time. And tell me about which company you think is going to go through this pattern before you could say, hey, these 10 companies, Peloton, Coinbase, whatever, give me 10 technology companies that went public over 10 years ago were worth over$10 billion, but went through more than two management changes, and their stock prices down 50 % from their peak. Okay, now you get that list. Okay, put it in a table. Tell me the price earnings ratio. Now tell me which one has done layoffs, right public news data, and which one has the largest revenue per employee. Which one and then compare that to the ones that have the largest revenue, the largest earnings per employee, not top line revenue, but bottom line, right?

26:56So this is kind of weird queries you give, that you might say to an analyst in a brainstorming meeting at your financial company, but now you're just going to type it into a box. And then you start getting back this information. I've been doing this kind of stuff with chat GPT when I did my tweet the other day, about what were the greatest product launches of all time. I literally did this, tell me the greatest product launches of all time, put it in a table, tell me the year it was launched, tell me the company and tell me how many units were sold. Now, it didn't get that perfectly, but it started to talk about, you know, GPS, it started to talk about the Mac, the iPod, iPad, iPhone, all those kind of obvious things.

27:33Sentiment has always been hard for people to understand, like, are people negative on a company or positive on a company? Why is that? How does it affect stock price? It might be that negative sentiment about a company in the New York Times, and let's call it. Gosh, I don't want to trigger anybody. But let's say the woke press, the left press, maybe if Mother Jones, and the New York Times and MSNBC, and certain Twitter handles, who are quote, unquote, woke, or I'm using the modern term, not the original term, who are, you know, highly liberal, anti capitalism, pro socialism, or social Democrats.