Sora Disappoints, ChatGPT Pro Tested, Inference Time Reasoning & More with Sunny Madra | E2062

0:00I think OpenAI is going to lose their lawsuit. I'm saying it right now. I'm predicting it here. I think it's going to be an injunction against open AI and they're going to have to settle for billions. You heard it right. I think it will be the largest copyright infringement case in history. I think it will be a billion dollar settlement with the New York times and other people are going to join it. If you are a content creator and you feel your stuff is stolen. Billion dollar. Okay. I think it's going to be a three comma settlement. I honestly do. Or judgment. Trace commas. Oh. this week in startups is brought to you by lemon.io hire pre-vetted remote developers get 15 off your first four weeks of developer time at lemon.io slash twist open phone create business phone numbers for you and your team that work through an app on your smartphone or desktop with listeners can get an extra 20 off any plan for your first six months at openphone.com slash And Zendesk.

0:54The best customer experiences are built with Zendesk. Qualifying startups can join their startup program and get Zendesk products free for six months. Visit Zendesk.com slash twist today to get started. all right everybody welcome back to this week in startups i am so excited because the prodigal son my boy my bestie sandeep madra is back you know we were on such a tear every week you were coming in you were doing all these great ai demos we're placing bets it was completely degenerate and we loved it and then you got a little busy grok has been surging you raised a bunch of money we saw that in the press yum yum i got a little tasty poo of your company definitive intelligence which was of course bought by grok grok of course is the inference chips that chamath invested in i guess eight nine ten years ago and funded they bought your company and here we are give us the latest on grok and then let's get right to our demos yeah no i mean look uh apologies i i missed doing it with you but you know my one of my uh resolutions for next year is make sure we're doing it every week but it's been it's been busy for us you know post financing we've been doing a lot of deals you know spent a lot of time in the middle east met some of your friends out there as well jay cow oh great yes really inspired by what's happening out there to be honest why are you inspired what explain to people what's so inspiring about what's happening in saudi uae you know and kuwait doa bahrain oman you know there's just like a lot israel a lot of going on there everywhere well i you know if i could distill it down to kind of three things which i find really inspiring.

2:26Let's just start with a lot of young folks, right? They have a lot of young folks and that makes for a population that has a lot of energy, right? You probably even see that being in Austin, right? And so - Big time in Austin. That's one of the things I love about it. Yeah. Get that young energy. That's number one. Got it. Two, they have a kind of a core business, let's call that the oil business. And they can use that to basically elevate themselves into the next industry for the country, right? Any industry. They could participate in any industry. When you have that kind of a chip stack, you can sit at any poker table.

3:03You're invited to every game. And look, they've made a distinct choice to make a big bet at the AI poker table, right? And they're doing that across the region. And then, you know, strategically, the area is central to about, you know, within a thousand kilometers there's four billion people right within 600 miles right there's four billion people because you have india is a hop skip yes you got all of africa you got all of europe all going into the dubai airport or the saudi airport yeah exactly and you know you've spent time out there there's a there's a really good energy right there don't deal with a lot of the stuff that you know we've had here which is changing now good to our bunch of our friends You're referring to woke nonsense, regulation, and an outright hatred of capitalism.

3:51Yes. And, you know, they want to win and they're making it happen. And they love America, which is also great, too. I think it's incredible. I think it's well said. You know, it's kind of the equivalent of if the United States was sitting there and you had but one rival, China. Yeah. Right? And China, you really can't participate in that poker game. You're invited. we were invited for 20 years to that poker game yeah and i'm like you know what can't play poker anymore and it's like well whose fault is that why'd the game break it's like i don't know and whoever the host was broke the game the game broke somebody stiffed the game i don't know why the game broke but we can't play with the chinese anymore our government is stopping it their government is stopping it and you can't trust the game in china if they're going to rug pull you and take every education startup which xi jinping did and say you know what education is owned by the state all your investments go to zero that's kind of like you know authoritarian behavior that makes people not trust investing in a region everything we do based on some level of trust now europe is in decline you very rarely see a company break out there the nordics berlin sometimes london you do have exceptions and i know people are trying over there but let's face it it's a giant retirement community and that's why we all love going there period full stop it's like epcot center great summer vibes so you great summer vibes love it but doing business there is hard so then all of a sudden this region emerges and you know it's really uh charming to see a group of people who are like hey the way you've done things in your democracy in the west has gotten the best results objectively with the king abdullah scholarships and all these great scholarships they gave people in the 80s 90s 2000s i understand they just sent all their kids to america and to europe to get educated they came back now you got all these 30 40 50 year olds gen xers millennials gen z who have crazy degrees perfect english you would think they grew up in jersey or boston or something by their accents because they've spent as much time in america and in boston going to harvard mit and you know myu whatever then they did there this is all like an incredible setup to they want to do business with us that is the right side of history which is another reason to love doing business there i see it and you going there me going there brad gerson are going there all of us participating there is exciting because they want to build okay great but i also think if we think about the larger planet it would be very nice for india the middle east region which is obviously it's a lot of different cultures yep and the west africa europe africa too yeah and africa too but you know africa is a frontier market uh which is it's emerging and you know there's various levels of stability and investment so a great market but it's just interesting that the people who are writing the checks go build are aligned and they're aligned against russia china you know and maybe authoritarian countries so it's a really beautiful thing that i think is happening it's easy to criticize it there's a lot of issues but we'll leave those aside for now um and i'm just excited that you're spending time there as well yeah we got to get out there together and so um i had dinner with your Omar once as well.

7:01It was great. We'll do it. Make sure we're 2025 goal. I'm going to announce something in 2025 in the region. Oh. Yeah. So I'm going to be there. Okay. Yeah, it's going to be exciting for the audience of the show. But in 2025, there'll be an announcement, two different announcements, two different regions, two different projects. And so it'll be quite nice. Count me in. Count me in. I'll be there twice a year. I'll be a little advisor on there because I'll be there quite a bit. Okay. You know, the reason I want to do it too is I want to expose my family to it, my daughters. I want them to see what's going on in this region because if you're not in America, if you're coming from another country, people are deciding, do I want to be in Riyadh, New York, the Valley, Austin, Miami, Doha, Dubai, Abu Dhabi.

7:41Dubai, Abu Dhabi, yeah. This is the destination for smart people in the world, so it's super exciting. Okay, let's get to demos. You haven't been here in a while. There's been stuff dropping on my head like you wouldn't believe, and I am so impressed with Gemini. I have the Gemini app on my phone. i'm just going to put it out right now have you been playing with the gemini 1.5 and the deep research i mean i i i use it all the time it's kind of part of my workflow it is at parity with chat gpt4 in in my experience as a user and i'm finding it has access to some data images flights other things that i'm starting to see get pulled in from the google suite of services what's your general since we haven't talked since 1.5 and the 2.0 and all this notebook deep research has come out just generally and do we have any of those demos lined up yeah we do so we were actually going to do something and maybe we can expand it so you know i don't know if you've done um 4.0 pro yet but i have 4.0 pro that's a 200 a month product from exactly i haven't ordered it yet here's the reason i haven't ordered it they don't allow you to upgrade individuals in your enterprise plan to it.

8:53I got to like, it's a mess. Like just, Hey, Sam, somebody clipped us and sent it to Sam Altman. Just like what I'm in, I have a paid enterprise account. I have a personal account. I want to pay you 200 bucks a month and there's no upgrade button. So what's the deal? Yeah. Well, there's an upgrade from your individual account, but I don't know how to do it across the enterprise account. That's the thing. So I had kind of a similar issue. So I had to do it on my personal account. So one of the things I was going to say, J. Cal, and we can maybe do like a multifaceted test here. Why don't you pull up your 4.0?

9:20I'll pull a 4.0 pro and you have a very standard prompt that you come up with okay uh you know the one when you are doing like the kind of the uber analysis so let's both put them in at the same time okay and and let's compare the results how about we do that as like kind of our first that would be an interesting one i'd have to go find it let's do okay um okay here we go okay so send that to me in the chat and then i'll drop it in pro and then we'll do a comparison between what pro the 200 a month, which is meant to be your PhD level versus your regular, maybe undergraduate level. All right, founders, are you tired of doing all your own software development?

10:03Do you need help, but you can't afford all this time it takes to find great talent? Are you dreading the endless interviews and email chains just to find somebody great? It takes six months, It takes a year. Well, what you need is Lemon.io. Lemon.io has thousands. That's right, thousands of on-demand developers who can help you. And they've done the work already to vet these developers, making sure that they're results oriented and that they're super experienced. Of course, they got to have competitive rates. So they're going to take care of that as well. And great developers are so hard to find and integrate into your team unless you're using Lemon.io because they handle all that for you.

10:45They only offer handpicked developers with three years of experience at a minimum, and they have to be in the top 1 % of applicants, right? Something goes wrong, don't sweat it. Lemon.io will find you a replacement developer ASAP. So many of our launch founders have worked with Lemon.io and they've had great experiences. So to lemon.io slash twist and find you're a perfect developer or the perfect tech team in 48 hours or less that's right and twist listeners get 15 off the first four weeks stop burning money hire developers smarter and faster at lemon.io slash twist so here we go i am uh opening up my chat gpt window yeah i'm using oh one pro you're gonna use oh one yeah okay not pro one okay not yeah because i don't have it here we go all right i just did it and you can see mine it's going through this uh you know thinking assessing you're seeing i have the screenshot up here right it's kind of going through it's see that on the right side you're seeing how it's basically working through this problem right it says it's gathering data i'm working through a hypothesis on u.s daily car trips pegged at 1.1 billion seems like a logical starting point gives you a little color commentary refining trip estimates i'm pulling data from the nhts 2017 estimating 940 million trips daily assessing technologies influence calculating robo-toxy fleets etc etc etc so you see it's doing some logic it took my prompt and uh just to give people an idea of what the prompt was was build a detailed model estimating the cost of creating a robo-toxy fleet for all car trips in the usa including waymo cruise as a case study the model should account for total car trips in the usa uber and lift trips fleet efficiency fleet operations required fleet size public transit fleet costs injured induced demand and the output should be provide a comprehensive model with data backed up assumptions and link sources that estimates the total cost of building a robotaxi fleet in the entire u.s yada yada yada so this is kind of like a crazy insane thing that i'm asking you to do right and is yours done so mine is still working on it right so yeah mine has been done for a while yeah and then does it yeah it just did a very quick one here and we'll see yeah uh it took 22 seconds to do mine i can uh go ahead and share my screen and i'll show you my share yep and because mine is still working on it and this is where we're going to assess the difference between having a 200 a month yeah 20 so here it is you see mine and the output was below is a modeled estimate for scaling a robot taxi fleet cover all u.s trips plus 20 of public transportation trips with a 20 % induced demand factor.

13:27What induced demand factor, if people don't know, is more people will take more rides, right? So people will take 20 % more rides because they're cheap and they're available. Key data sources, total US daily trips in 2017, Uber, Lyft trips, company filings, SEC filings. It tells me public transit ridership, American Public Transportation Association, RoboTax, the operation fleet assumptions. So US car trips, 0.95 billion a day u.s car trips 350 billion a year which makes sense a billion a day uh u.s and lift trips 4.5 billion a year just over one percent of the total trips robo taxi trips per vehicle per day i put it at 20 to 30 trips a day assuming 25 a day maybe it's more i don't know if uber if uh tesla can do more with their uh fast charging and fast turnaround to clean it i assumed uh five days off for each car for maintenance of 360 days on the road per car pretty aggressive i think uh trips recorded yeah you say there's all my calculations and it says here uh with an induced fleet size you would uh have around 422 billion rides with induced demand you would need 46.9 billion vehicles now that doesn't count p so you know this is this estimate probably needs to be doubled to demand to handle peak demand right uh like people leaving the warriors game or something anyway you're gonna need uh you need five trillion dollars at a hundred thousand a vehicle you need 1.5 trillion dollars at 30 000 a vehicle to replace the entire fleet in the u.s this is not to just do ride sharing this is to replace all rides in the u.s nobody has a car what did you get and you know this is a good test right you know you've got these projects yours did it in a few seconds cost 20 bucks a month mine cost 200 a month okay 10x of value give me 10x value okay so first of all i think it's very important to see here it thought for two minutes and 42 seconds right which is incredible right i mean just just let's level set for a second all of this incredible work's happening either in your case in a few seconds in our case you know two minutes and 42 seconds here in my case all right first of all i think it goes it goes below as a step-by-step illustrative model with clear stated assumptions and references and calculations.

15:45So it's a lot more organized than yours, I think, J-Cal, right? So it goes, total car trips in the US. So basically, you know, walks through it and its simplicity gets to a billion car trips a day. Same answer, basically. Yep. Okay. Annual car trips gets to 365 billion, right? Yep. It says maintain more conservative. It actually does this interesting thing saying, hey, we'll go to 340 as slightly lower, still representative based on the NHTSA data that it has right then it tries to calculate uber and lyft trips annually it pulls those from the s1 filings from 2018 and 2018 of lyft pretty interesting right i like the reference here i don't think yours had that right it did not and it yes here it actually did the time for context it's given this context section 5 billion rides from uber lyft versus 340 billion total rides mean the current ride share currently represents one to 1.5 percent demands which is a calculation I just did.

16:36So it's like thinking a little bit more. Yeah. So it's more like a J-Go. Then it's like fleet efficiency, right? So it kind of walks through this. So it's assumption is, hey, each fully autonomous robotaxi can do 20 to 30 trips a day. We'll take a midpoint. I like how it does that. Average trip duration, 15 to 30 minutes, some downtime between rides, high utilization scenarios, right? Okay. It's kind of all the stuff that you were rattling off on your own, right? Then it's got fleet operation constraints, right? Which is, you know, six hours a day for charging, cleaning, maintenance, five days a year for overall major maintenance shutdowns.

17:12So basically it gets, say, you know, 360 days a year it can be utilized and it gets to daily utilization. And, you know, again, I like this breakdown, right? Which is 25 trips per day, 360 days, 9 ,000 trips per year. So if we're strict with the numbers, basically we get 8, 9, 7, 5, but we'll use 9 ,000 as a rounded figure. I like how it's kind of humanizing that bit too making it easier for us to handle and then it's like required fleet size to handle 100 of the u.s car trips right so just going through all the math 340 billion trips 9000 a year requires 37.8 million robo taxis what did yours get to on that one jake i think it said something like 46 okay because it's taking into account the induced traffic which i put into the instructions and it put in uh picking up some public transport we had the same instructions So I should have had that.

18:00Same instructions. Yeah. Now it's like it adds the public transport capturing 20 % with FSD cars. So it starts to run through the logic here. And it basically, again, still at 38.2 million rides. Now your fleet cost, it sort of starts to calculate it. And then just for, you know, the induced demand, because you and I were talking about this, it's well known that you have these induced demands. So let's say 20 % increase in total trips, you know, because of widespread capacity. and summary after all that yeah the summary is like 340 billion trips 25 a day i gotta tell you it's not that different it's not 10 times better it feels a little more polished um i wonder if it's our instruction set so i think it'd be good to have somebody from open ai on the pod who worked on this project so let's just send a little note to them the results here are not different the results here are 90 the same there are some formatting and so the u the ux i think is determining the value of llms right now it doesn't feel to me like the core llms are getting much better because i don't think i think they've got as much data as they're going to get or as you know 80 or 90 as much data as they're going to get so then it becomes the interface then it becomes the instructions and how it interprets what you want and how it gets to know you in the personalization that's my feeling of the gains that are happening now formatting like canvas um you know this product artifacts yep the pro yeah agents eventually i think we're now in the fit finish and polish of the language models and the language models themselves um are they starting to plateau because they've run out of data this is something i saw ilia was saying hey we stole all this data he didn't say steal but let's call it what it is we stole all this data we got all this data some of it's public some of it's not okay whatever it is it is putting that aside i think that they're now stuck founders know that every missed call is a missed opportunity customers don't want to wait they will call someone else if you don't pick up but if you use open phone you're never going to miss another customer call and guess what it's super affordable and easy to use.

20:13People used to spend tens of thousands, hundreds of thousands of dollars putting in a corporate phone system. And now for just$15 a month, OpenPhone will give you a business phone line and complete control of your destiny. Want to know who's answering customer calls and how they were handled? OpenPhone can do incredible things like sync with HubSpot and give you AI-powered call summaries, automated responses to ensure you don't miss a single ring. Well, that's all built into OpenPhone. And if you've got an existing phone number, they will let you port it over at no extra charge. Get 20 % off your first six months.

20:47What an amazing offer at openphone.com slash twist. That's O-P-E-N-P-H-O-N-E dot com slash twist for 20 % off for six months. The LLMs feel stuck to me. And now it's about inference, interface, how it interprets instructions, how it personalizes you, and then proprietary data sources like reddit twitter uh google flights data you know which comes from other um databases you know the data the deeper data and the instruction set in the interface am i right or am i wrong it's a great summary jacal and it's interesting that you know we do this experiment on something that costs 10 times more i would feel you know if i had to use the output the output of the 200 one gives me a little bit more background on things yes but you What you've actually done in the way you prompt is that you've been able to direct the model that's not as powerful to do what the more powerful model is doing.

21:51right so maybe if we just said build a model of what it would cost to replace all u.s rides with robo taxis with as many details as possible okay and i'm going to give you the same one to do that is just a sentence now now let's see if it does anything close to what i did because you're right when i did this i kept a notepad open on the side i kept notion on the side and i was putting in my architecture of how i would solve the problem which is how many rides are there how many public transit rides are there induced i introduced those three topics i did i let it know that i wanted induced in there for 20 i let it know i wanted to know uber and lyft's percentage i wanted to know just in the u.s so i did give it a framework yeah this is really interesting it came to without my instructions an annual operating cost of 150 billion a year and total initial deployment cost of eight 2.8 trillion yeah wow so this came up with a totally different number and it didn't explain it very well but it's getting them as citations i think but see what what you've done jcals what you've shown our our listeners and our watchers is that if you are willing to put the time in to create better prompts and i think this is important for the industry you can basically get um what is the equivalent of 10 times more expense on a model just by putting the time and you know it was good it's i think this is a really really good example uh i'm running mine now so it's good these ones take a bit longer to run so just let it run as we talk through it but look it just found the daily passenger trips it got that right for trips robo taxi a day it picked 20 and it got it from a citation from litman 2019 so i guess somebody had done a paper at some point that said 20 is the right number we came up with the number 25 or 30 yeah the reason we came up with that number is we gave it we want six days off a year and we want six hours off a day to charge so we we gave it like hey five days or six days of maintenance a year which seems you know like if you got to take tires or it gets a fender bend or needs to be repainted who knows what could go wrong on these cars uh for maintenance and then charging certain amount of charging um and it came up with 55 million times 45 000 so it used the cost estimates of i think a robo taxi as opposed to a waymo pretty fascinating um charging infrastructure included 300 billion in charging infrastructure which it doesn't need to include i don't think but maybe actually maybe you do need to do that because if there were that many you would need much more you do actually need to include that because you would be charging every single car all day long that's actually a really good point maintenance operations software 100 billion a year mckinsey and company rand corporation 2018 insurance regulatory so people have been working on this data so it just did a better it just did citations it didn't kind of use my framework what did you get yeah is it still doing yours mine's still running it's probably going to take another minute here i my guess is but like so anyway i don't think it's worth it i think we learned something here but i'm going to buy it anyway because yeah 2400 a year versus 240 a year in business to spend an incremental two thousand dollars for an average salary in corporate america of let's say i don't know eighty thousand dollars i'm including people who make 40 and people who make 150 but we'll pick 75 000 you know for two twenty five hundred dollars you got to ask yourself does an employee an information employee get three percent more efficient with one of these products pretty clearly yes oh yeah but they have to use it and this is the thing that is making me a little I can't get people to use it I can't I you know I yeah I know if people are using it or not I can't get people to use it people it's this is a habit I think the world's going to bifurcate between people who use this as their default all day long and people who don't and it's going to be really sneaky can I give you a hack there please this reminds me of the time I would say it was like just past the mid 90s when the internet it was just making its way into the corporate world there was a set of people which is you know the our our vintage which were dying to use the internet and there was a set of people that would not use the internet and they fact they were didn't trust it and so and what what was it we were just younger and we were more kind of uh tuned to technology tech savvy we're had more energy more fascinated no kids my suggestion to you is as the summer is coming up either you do with interns hire two to three people that are sub 20 jcal and i will tell you under 20 everything they do they use open ai because you know what what is the one thing that you got through experience you got mentorship you got to work through that so how they account for that is by using um these ai tools and so as long as you have someone with good energy so imagine the 18 or 17 year old version of yourself oh yeah i'm super keen but i don't know a lot of these things oh how am i going to go figure it out well i'm going to figure it out yeah i mean you're not allowed to hire by age in the united states here right unless you were like casting for a movie i think you could actually do it there well maybe not age you'd have to be doing by look so but there is generational differences so while you can't hire for age if you do have entry-level jobs it generally will skew younger because the salary is yeah is it what you could do it as internships for college credit yeah and yeah there is a distinct difference between how young people use these tools and older people and getting older people to use them it requires sometimes change a little a change in behavior which requires pushing and so here's what i did it's called new tab override it's by soren hence show and what it does is when you load a new page and uh in your firefox browser and this will work on brave firefox and brave use the same it lets you put in an option of a url and then you see where it says focus here set focus to the web page instead of the address bar this is clearly this is a very important thing to do what this does is when you open a new tab it puts you in the url bar right yeah here it puts you in the cert box if you do this so when you start typing you don't have to hit tab or click your mouse to get in yeah super clear if you were for me this needs to be on your computer i will randomly pull it up during a zoom meeting and say do a new tab i mean i do this kind of stuff i'm Does that make me crazy that I spot check?

28:10I spot check. I say, pull this up. So if you work for me, be prepared. Yeah. Because I want you to, yes, chef. Yes, chef. And you know what? We're all chef in this analogy. I always tell everybody we like a good yes, chef. Just acknowledging that we're making progress here. All right. Okay. Mine finished. Let's quickly wrap this one up. Okay. Mine finished. Again, this time it was three minutes and 18 seconds. So the prompt was shorter. Okay. This did a better job. you see your your premise is correct you can get out of a standard model in the more expensive model it's doing the sub prompting isn't it yes for you it's all correct yes what do you call this prompt generation feature where i can give it less prompting but get more prompting because it's doing the prompting for me so i think the is there a term in the industry for it inference time reasoning so what it's doing is inference time reasoning itr iter yeah okay so it's iterating so we're going to use that we create an industry term inference time reasoning reasoning got it so when you do the inference that's the query yes it's doing at that time additional reasoning as opposed to doing the reasoning when the llm was built on a bunch of H100s and the models.

29:29The main difference is like, say, when we started on this adventure of, you know, chat GPT, and I'll just go to the summary here, but when we started on this adventure of chat GPT, the minute the first token is predicted, every other token is already determined. And so it's just going through, you know, sort of what's going to happen. What happens in this inference time reasoning, it has the ability to kind of stop part way through and then work and say, let me go and let me do an offshoot on some of these things, bring those answers back into my main line, right? It's an oversimplification, but that's sort of what's happening there.

30:01And so it's not just a standard prediction of tokens, which is what we saw in the original iterations. That's why we're seeing it. But what we've shown here, and I tend to agree, if you prompt engineer with more sophisticated prompts, you don't have to do as much inference time reasoning, and you can get very similar results. And so fascinating here that it did come to the same 2.7 trillion number that you were at. and the same operating cost. Hey, startups, when you're a business, you got to treat your customers right. Unreasonable hospitality is the standard today, but you're going to need tools.

30:37You're going to need a platform to help you do this. And that platform is the Zendesk Suite. The Zendesk Suite is going to give your startup all the tools you need to deliver exceptional customer experiences so you can build stronger relationships without growing your headcount. That's key, right? Every dollar matters. You got to control headcount. You got to control spend. So use the tool that Shopify, Squarespace, Uber, and Instacart all rely on. It's called Zendesk. Let's take a look at another customer, Unity. Very famous company. They saved$1.3 million with Zendesk automations and self-service, and they saw an 83 % increase in their first response time.

31:15These companies love Zendesk because it's so easy to set up and it scales with you as you grow. They'll also give you all the metrics to make your reporting easy, keeping you and your business agile and investor ready. And that's because you're their esteemed customer. And they've created the Zendesk for Startups program just for you, where you get unlimited access to all the Zendesk products, expert insights, all the best practices, and entry into their amazing community of founders, all at no cost for the first six months. That's right. They want to support you zendesk.com slash twist get ready to scale with the best in customer support with six months free nothing to lose it's really cool when you compare this to what gemini is doing so i ask it the same prompt here uh build a model of what it would cost to replace all u.s rides with robo taxis as many details as possible and it says here cost of it and it says what it's going to do and i didn't put these details in here but it says in the itr inference time reasoning it said build a model that it says with as many details as possible by find the total number of rights taken the u.s annually find the average cost per ride of each model transportation find the estimate so it's actually come up with its own reasoning analyze the reports create a report ready in a few minutes start the research and it's doing it right now feel free to leave this chat as you knows i'll let you know it's done and it researched 69 web pages look at that 69 not 420 and look at all these it's it's showing you its work this is why i think this is a better product right now for me i like to see what it's doing you know um and it's analyzing all the results here i guess we're about a minute into it this is gemini advanced 1.5 pro with deep research and if you don't have the gemini app the gemini app does not have deep research in it yet it does have the other features it's as good as chat gpt's app gemini and google have reached parity in my mind with it now did you see some talk about the gemini api and that api the gemini api is gaining steam on everybody is that true are people developers using it i saw a post uh for that today i i think directionally it's correct like definitely there's been huge amount of growth on on on uh you know jam and i i'm not sure i think the tweet i saw was from open router or something like that where they said it's you know greater than 50 that may be you know open routers view of it i still think you know it's probably not 50 but happy to happy to be proven wrong there if google folks want to come out they said it was high as 50 uh which yeah uh so you know which by the way uh if you want 350 000 in google credits uh you can get them at get startup credits.com get startup credits.com okay you know i have all these startups meet with us 28 000 people apply for launch go to launch.co to apply for funding from our firm join our programs etc after they apply you did with gcp well gcp did it for all in summit and they did it for this week in startups and they did it for accelerator we also have oracle provides credits azure microsoft azure provides credits credits and digital ocean provides credits to our startups the only one who doesn't is aws aws they have like a rack rate thing but aws is not very supportive of anybody but y combinator they've got like a weird thing they also don't buy ads or whatever so which is but you know aws is great i don't have any hard feelings towards them yeah but they're not supportive um in that way they kind of picked yc and i think yc is very sharp elbowed sometimes so they're like yeah we're team yc we're not team everybody else okay that's fine i have half the number of applications of yc right now and next year i'm gonna match them uh all right so here we go we've got this done and look at this it did a nice thing total rides in the u.s annually it estimated it got bus rides it included bus trips well i didn't ask you to do that based on the national survey americans make approximately 1.1 billion trips a day 411 billion trips annually or about 1500 trips per person wow it added that that's pretty interesting and then it has here look at this it built a table mode of transport car bus train freight so included all of those and estimated rides average course for a ride and it put the amount wow estimated cost of manufacturing and deploying early estimates up to 400 000 maybe that's in billions or something uh tesla is projecting 25 000 he got that right for their robotaxi baidu 77 waymo 180 wow that's interesting deployment cost estimated cost of maintaining and put that in without saying uh estimated operating cost per mile projected average distance travel per ride i mean this is incredible total cost yeah cost per robotaxi ride manufacturing cost deployment cost yeah and it just figured that out wow this is better let's be honest no it is the superpower is that that open in docks on the top right i mean well i mean that is i think if i want to i can just open this up in google docs yeah creates a document for you yeah which is your next step and now you're starting to see this now if i save this document in the future it's going to know that i did that document and it's going to be able to use your documents in your email so if i was emailing with i don't know somebody running a robo taxi or i had 10 friends who had shares in tesla uber or whatever and we had conversations i wonder yeah if on my side it's going to take that into account and say hey in your email in your gmail there was a conversation with dara or this analyst at warby parker warberg pinkis and they helped you do that and they gave you some data there do you want me to include that data so this could get very interesting very quick folks i believe google is the sleeping giant grok also doing a very good job that uh the other grok yeah which we'll get into okay let's do a couple more demos here well just quickly i want to close out on that statement so let's do one thing we got to get back to our grading oh one pro versus oh one and then let's grade uh 1.5 with deep research so three grades i'm gonna give a b plus to pro i felt like it did a really good job yeah and i would pay for it and i would give an a to the new 1.5 from gemini with deep research with deep research i'm giving it an a only because i believe the output of both of those with itr i feel for the average i'm creating it on my feeling and what i think it will do for the people who work for me the people who are not putting in the deep thoughtful prompts they're going to i think get a lot more out of uh gemini 1.5 with deep research or oh one pro the 200 a month now they google it's 20 bucks a month so i give it an a plus on a value basis on a value basis on a value basis i'd be like a plus and a and a b so there'd be a big gap there i'm saying in a corporate america this pricing does not matter for the value matter yeah yeah it doesn't matter because you're spending more on people's parking so just throw the shit in the garbage sorry uh and let's just talk about how much it will impact my employees who use it my team members who use it my founders uh who we invested and partner with I believe B and A.

38:37A B and an A. I'm not giving pluses and minuses today. Yeah. So my interpretation is I actually, I'm going to give them both A's. And what I really like about what OpenAI has done is it is giving the reasoning process along the way, which I think is very powerful for people that are using this in a work context versus when it's just spit out at you so i liked how it was sharing its its reasoning along the way so i'm i i kind of lean towards that which deep research does as well gemini's advanced 1.5 pro with deep research i got to talk to sergey and the team over there when you're naming these things it's gemini that's the product yes it has a version nobody cares about the versioning just it's gemini don't say advanced don't say 1.5 just call it gemini and then have gemini with deep research and abstract out the the version numbers for nerds but i think this is too confusing for consumers right it's getting even worse like if i look at my menu here which i'm sure you have the same choices i have 1.5 pro 1.5 flash 1.5 pro with deep research 2.0 flash experimental and 2.0 experimental advance yeah you know this is okay google's premise was here's the box you type in what you want you search or you say i'm feeling lucky i'm feeling lucky it's a sniper shot takes you right to the thing that was cute and fun but there was just a search box so here for gemini i think it should just either do a quick search whatever the best search is or you should have deep research and then if you want to you have somewhere where you can kind of tweak the model but it's just too confusing for consumers all right so we make a progress on this one okay one last thing on this one is if you could only use one which one would you pick i'm gonna stick with open ai pro oh i'm gonna stick with uh gemini deep research because i think google has access to data that open ai does it and i believe the gap is going to grow okay that's we'll we'll come back to that one i don't know if i bet on that but i'm just using it now one last thing on this one okay okay how about this how about this two high school interns for the summer i don't do internships unless it's friends of the firm okay i do it as a favorite bank you know why because in those 10 weeks they take up all your time and resources and you train them and then they're gone but they're supposed to use the the these tools that's what i think it is for the rest of the team that's i just prefer to hire people at school i'm going to university of texas shout out to jay hartzell president of ut i went to ut and i am so impressed by the ut graduates i went to a game longhorns whatever that is uh go longhorns and uh i am all in on ut we went to a football game i went to a football game a pigskin well i mean a hundred thousand people i was on the field man it was awesome uh but a more awesome there's 55 000 students at ut and they're smart and like i think the top one or two percent there are like ivy leaguers but they're blue collar ivy leaguers yeah with one thing i i did see this thing and it was there's ut at austin then there's also university of austin ut is the public austin yeah of course yep big giant school they're funded because they have my understanding is they have land and under the land they found oil so they are super funded in texas if you if you if you're a texas resident ut is like eight nine or ten thousand a year you can get in and out of ut for 40 dime skis which is half the price of a private school in the bay area for one year because it's 60 or 70 and you got to give a 10k donation or else they admonish you and give you a hard time for sure so one year of private school in the bay area or new york dalton whatever this nonsense is an entire degree from ut college education college degree university of austin now if you're out of state they charge you a rack rate and so a little bit higher yeah but 80 or 90 of the people go to ut are in state this is amazing okay now let's go over to uh university of austin joe lonsdale and a group of these you know kind of free will and awesome free-thinking libertarian-ish republican types on the right but i would say maybe they would be considered moderates you know like i think barry's probably a moderate yeah you know classically like she probably would have voted for a clinton democrat or you know a matt mit romney as much as she would vote for a trump or whatever so if she did vote for trump i don't know if she did um putting that aside they just had their first class it's like 50 students or something it's a startup school they bought some university so it's accredited and they want to teach from first principles take all the woke out okay got it but i mean i'll be honest in ut ut doesn't have like a woke movement there like when they had the protests or whatever it was it started and ended pretty quick but in a long way of saying you know i've been thinking about you know i have this founder university and i'm gonna bring it in person i think in the next cohort uh is my plan in texas and have in austin in austin and i want to get a space this is the big announcement you know it's part of what i'm doing there and so i'm trying to figure out if I do that with a university or if I just do it in a space or if I do it remote and in person if I do it every day for an hour a day or if I do it two hours a week and get a co-working space so you know it's a lot on my plate right now but I'm trying to figure out how I can have my own university the founder university teaching how to be a founder and that's why I spent so much time getting founded at university and giving people instead of they pay to instead of paying tuition we give 25k to the top 10 percent of students to start their company at a one million dollar valuation for 2.5 percent yeah which is a good deal for us most people argue it's a great deal it's kind of like the y combinator accelerator but we expect only one out of three of those to pull through and get another round of funding so we're taking high high high massively high risk bets so if you were to net it out if two out of three don't even make it to the next round of funding it's really like 75k at three million for two and a half percent because you're taking into account how many people would wash out and just not make it to year two.

44:46I really enjoy that kind of part of the job and seeing a lot of good stuff. What else? Let's go lightning round. Let's continue on the path. I do want to give a shout out to a couple other things along the way. So lightning round for the next few minutes here. Okay. Just some of these are not demos, but they're important things to call out for. Recently, Meta launched Llama 3.3 70b. And what What I wanted to call out on Longmo 3.370B is, just in terms of how fast things are iterating, and we won't do a demo with this because, you know, I think it's just easy to call it here. But if you look at a comparison against Gemini Pro 1.5, which you were just playing with, with deep research, right?

45:22You can see that it is starting to make really good inroads against its, you know, previous competitors. In a benchmark test. In benchmark tests, exactly. But this is a relatively small model. So let's not count meta out here that's all i'm trying to share here oh no met is doing a mitzvah for the industry by going open source and not trying to make money on this and they're letting other people use it there were some weird caps like you couldn't have 100 million users or something but i think zarkberg is going based and he looks at this like the open compute platform he knows he's got a network effect he'll defend it everybody can use his language models and he is going to be the backstop against sam altman's closed ai and which is so paradoxical by the way and what i do want to call out here right this is this column right here is llama 370b this is gemini's gpt4 oh yeah and look at the pricing down here you're talking 10 cents for million tokens output 40 cents for million tokens output your dollar 30 and five bucks and two two dollars and 50 cents and ten dollars so So it is getting there in being comparable, but from a price perspective, crushing, which is something that Zuck and Meta have always been incredible at.

46:37Well, they are obviously investing heavily in this. What do you estimate these platforms are losing providing services at this pricing? Or are they breaking even? What are they doing? Do you have any insight into their infrastructure costs and how much they might be losing, making or breaking even? I fundamentally believe if you are, and look, I'm a bit skewed here, but if you're not building your own infrastructure, including chips from scratch, it's very hard to be competitive because you do have to pay an 80 % margin to NVIDIA along the way, right? And so fundamentally, I do not believe that anybody is losing a ton of money on these things, but they're not making a lot because the big chunk of the margin is being taken out by NVIDIA along the way.

47:22So this is the challenge for the industry, and this is why NVIDIA could be a short or, you know, could have topped out here because people are now realizing there's an 80 % margin there, which means they're going to have to compress that margin and lower their pricing to compete with Amazon, Apple, Gros. Yeah, I mean, everybody's providing inference chips, et cetera, at greater and greater prices. Do you make custom ones with people or do you only make your own? No, our chip runs all models. So, you know, folks come to us and we run them. And, you know, and so that's kind of where I think things are going to start to net out is that you see that price difference there.

48:00Let's not ignore that. Let's make sure everyone keeps watching that because I do think there's pricing and there's capabilities and those things are kind of starting to go in some interesting directions. Let's keep speed rounding here. Okay. Okay, speed rounding. so next one let's do sora because you had brought that one up sure and so have you have you tried sora yet jaco i haven't tried it but i've looked at the demos i see the demos so yeah so basically you know we've got it here uh the the generations do take a while but like what you can see here is you know they've made it available they've made a very clean ui they've given you the capabilities to do anywhere from five ten seconds right and we can do a couple of different variations and so none of it looks real the interesting thing is it all looks fake i i was i was going to tell you that so my overall observation is uh cling which we reviewed before which is that one of the chinese-based ones it looks the most realistic and i fundamentally believe that's the case because it's trained i think on a bunch of proprietary copyrighted data and because it's done that it's able to do it now my understanding is a lot of the folks a lot of the training data that's provided to these is being generated via game engines and game engines are good but do you get a feeling this feels a little bit game engine this feels like if you if stock imagery and a video game had a baby so we have nailed it if you look when you see this stuff it looks like cgi done you know in the czech republic or poland eastern european country south paria done by somebody doing like a corporate video or something in other words it looks professional but not industry leading like disney or george lucas or you know uh jj abrams would would accept so jj abrams george lucas spielberg gorsese you know anybody making a you know secession tv show nobody would accept any of this it's all 60 of what they would accept 70 so this would be great for them to storyboard and to be able to show hey here's what it looks like but it's not good enough for prime time it feels like it's a couple years off if ever because you know while the chinese have no problem stealing disney's archive and doing this and they'll have models out there that will let you do anything you want with the marvel characters the disney character soon i think marvel should release an a model in partnership with one of these companies and for your disney plus subscription you can create disney character models and uh you can make short videos and they're only exist inside the disney app and you can send them to friends this would be a killer feature this is one of the chinese ones i was going to play it's like a two minute video but this one i want to get your reaction to this one i'm watching it okay that looks like stock there that looks real that looks pretty real yeah like almost like they took a george clooney film and like a professionally shot george clooney film in italy you know the italian job or something yeah this looks like they took how they stole from hollywood to get this effect yeah so doesn't it look closer it's it is distinctly closer in that example to a hollywood film than a stock photography library yeah yeah well this shows you and i think we didn't bring this up but the open ai whistleblower who apparently committed suicide or was whack uh there's a lot at stake here i mean i know i sound like a conspiracy theorist but they're uh i think open ai is gonna lose their lawsuit i'm saying it right now i'm predicting it here i think it's going to be an injunction against open ai and they're going to have to settle for billions you heard it right i think it'll be the largest copyright infringement case in history i think it will be a billion dollar settlement with the new york times and other people are going to join it if you are a content creator and you feel you're calling it billion dollar okay i think it's going to be a three comma settlement i honestly do or judgment trace trace commas trace commas well listen this is not unprecedented in the world things like this can happen we have seen records be broken when there is serious damage and so and i i think it's going to the reason you know this tragedy that occurred you can look it up folks is a 26 year old whistleblower inside of opening eye who apparently is unalived and we don't know why um and he was a key linchpin in this test testimony um i'm not saying i think he was murdered by an opening eye employee obviously but this is really weird looking very weird looking uh like many weird things occurring in the world these things i always thought were weird and then everything that's happened in the last six months and it was like you know what anything's possible now anything i mean listen if if people who don't like putin you know fall out of windows at an alarming rate statistically you know who's to say it couldn't happen here right i mean i'm going to be so arrogant to say there couldn't be somebody yeah uh oh look very sorry for the family of the gentlemen hopefully they figure out what happens there um not not but quick grade on sora i mean i give sora still a b could be much better i'm sticking with my b there i didn't i'm not like i don't know what the use case for these things is okay i feel like you're not there yet like 2.5 or whatever that was where you were like or like yeah remember gmail could guess the fifth word yeah and you're like okay okay all right but the jump we saw today from the chinese you know no no the jump in the reasoning oh the reason guess the third or fourth word yes uh would you like to yeah you know it says have dinner or whatever yeah that guessing game that it was doing five years ago in gmail leading to gemini with deep research that jump occurred in five years well if that happens with this in five years we'll be sitting here going make us a sopranos episode and then we're going to watch it in my movie theater entirely this you know make us a lost episode you know yeah that'd be great that'd be fun well no i mean it's it's gonna happen and there's no reason your favorite character in the star wars series you couldn't direct a film uh or your daughters couldn't you know or your kids couldn't work together to say you know we always do like a little um yeah what do you call it like a you know like a um talent show you know at holidays we have to a little talent show with the kids so if we do a little talent show it's going to be like hey i directed this film about ashoka from star wars right it's coming so i do that all the time i do seinfeld episodes in a modern with a modern scenario they're always kind of fun to do try that in in your thing of choice george starts having chat gpt uh do all of his communication he's just like i'm so bad with women i'm just gonna i've always had some he's or he's using it at work just yeah he just had it do what he thought this historical figure would say in certain situations so instead of doing the opposite costanza he does whatever stalin would say whatever you know like some lunatic would say He says, make me into Einstein and myself, and all my responses should be like Bob Dylan and Einstein.

55:33All right. Two more really quickly, and then we're done. You gave a grade there. Okay. This, I think, is interesting for folks because this comes up quite a bit. And I had this live example. I actually tweeted about it, so I think it's fascinating. So, I used, in this case, perplexity. And I said, hey, why did Syria fall now versus in previous years? Who are the rebels and what parties are supporting them? And, you know, to be honest, you know, I actually wanted to know what happened. It was purely curious. And I think it did a good job in terms of explaining what happened in terms of real-time information.

56:05Then I did the same thing in Grok, right? The X-Grok, X-A-I-Grok. Yes. And so, even though these folks have access to real-time information, which is great, and, you know, I think there's a huge advantage for G-R-O-K, there is, you know, the folks that are connecting out to real-time sources like perplexity i find and so my my task for you here jason is as things are unfolding because i know you're always having to look up stuff either for episodes of yes we can startups or all in um really try this in both perplexity for real-time events and in yes and grok i find grok is really doing a good job of catching the zeitgeist on x but i do worry about all the anonymous accounts and the anonymous accounts at scale and their bias you know like i don't want king koa the anonymous million person account that sax is always retweeting and is in love with sax like i get it it's obviously some right-wing person in their mom's basement or it could be a russian or it could be any other chaotic actor and they've got a million followers like that stuff for me is really dangerous like an at scale account with a million followers that's getting paid by x right that has revenue sharing on but we don't know who it is now i get you want to protect people and i get it's a pseudonym so you can look at their account and there's 20 of these accounts that have now hit prominence half of them have a real name half of them don't so with the real names at least you know like this is a 25 year old who's just a fanboy of yeah you know whatever the left the right in between this pod that pod joe rogan you know uh rachel matt or whatever it is the bias is clear it's these anonymous accounts that are hitting scale that have thousands of other anonymous accounts and this is where like the data at reddit the data at twitter gives you an advantage in that it's consumer driven and it's fast and furious but now you're going to have an incentive and i don't i've never heard anybody say this so i'm floated here there is a big incentive now for a foreign actor or spam accounts to come in and use a hundred paid accounts if i had a hundred paid accounts based on twitter and reddit and hacker news and i start building these pseudonyms up and i start having conversations with myself on these platforms not only am i getting the value of influencing people in real time i'm getting the value of influencing the language models now let's let that sink in so a foreign actor you know you think about like uh you know somebody who has uh an axe to grind from the middle east or an axe to grind from china or north korea whatever it is and any any political bias you could send a hundred really smart people you could have literally 100 people where yeah you could have 10 smart people getting paid a hundred thousand dollars a year in a war room for a million dollars a year with 50 different accounts each rotating them you know with a turret display and then feeding the llms their biases and now they're going to in real time because i think open ai has a real-time deal with reddit and reddit has reddit answers just using reddit as an example you know are you going to be able to trust the data on reddit i I would trust it before LLMs existed, but I don't know if I'm going to trust it after LLMs existed.

59:32Well, and that's kind of almost like why you need LLMs though, JCal, right? Because you need LLMs to look at not just those hundred accounts, but maybe a thousand, right? And look at more, because that's the advantage of the era that we're going into is that you can have sort of a infinite number of parallel agents looking at the data and then aggregating it all, right? And then trying to get to the ground source of truth. I think it's like there are dozens of accounts that reach true influence on these platforms i think like it's low hundreds so i don't think it's that difficult to shape reality to shape the shape the message shape the message just look at wikipedia my wikipedia page other wikipedia page there is a small number of people who are really influential under 100 who are the super editors of wikipedia and some of them get paid uh covertly and you know by pr firms and there's like a whole grift going on there in the back end And just like the review systems on Amazon and other places, I don't know that these LLMs are going to figure out who the bad actors are and who are great content creators who slowly introduce bias.

1:00:35It's a good opportunity for a startup though. Imagine to do that, like, you know, to look at all that and analyze that and look at real news sources. And that's an interesting idea. All right. Okay. Well, maybe we cut that out here and we make our own influence startup. maybe we should leave this out and you and i should fund a startup do covert intelligence operations for corporations and individuals so we could start our own little cia interesting idea yeah all right this has been another amazing episode of this week fun to be back jacal it's great to have you back let's come back i mean i think every two weeks is the cadence that you need because you've got a lot going on so i think maybe we could even be monthly for now But you got to lock in because people love to get some deep madras take on everything.

1:01:23And you're on the inside. That's what the show is all about. Insiders sharing with other insiders to catch you up. If you miss this, you're going to fall behind, folks. These are the important discussions of our time. Thisweekinstartups.com. If you want to search the AI archive powered by our friends on Podcast AI, they're not a sponsor, but I am an investor. If you would like me to invest in your company and Sonny's and LP of mine, all you have to do is go to founder.university. see if you have an idea with two or three friends and you're in year zero or one the accelerator launch.co slash apply launch.co slash apply i'm saving up to get the m to get the launch.com i think our friend uh friends at yahoo still own it all right everybody we'll see you next time bye