AI Model Showdown: Grok 4.1 vs. Gemini 3 | E2211

0:00I heard Gemini 3 is out. I heard Grok 4.1 is out. Everything is happening so quickly in our industry. Look at what XAI did. Grok 4.1 is really good across a number of metrics. Compared to Grok 4 fast, it's a dramatic improvement. When 4.1 was released onto the charts, it went straight to the top. And I think it just shows that there's several different AI labs here in the US that are able to successively create state-of-the-art models. And I think it also pushes back, Jason, against some of the doomerism that's been kind of building in the last three months? One of the great things is, I guess, that now that we have these leaderboards, it's motivated all these companies and gotten these teams super excited about claiming the top spot.

0:42That's where I'm just getting a little concerned of, you know, like, are we actually making progress at solving the real world problems people have? Or are we getting good at acing the SATs? And that is a question for me. This Week in Startups is brought to you by Zyte. Zite is the fastest way to build business software with AI. Build apps, forms, websites, and portals that connect to the tools you already use. Go to zite.com slash twist and get 50 % off your first project. Every. Running a startup is hard enough. Every takes care of incorporation, banking, payroll, benefits, accounting, taxes, and more so that you focus on building, not back office admin.

1:26Visit every.io. And Goldbelly. Gold Belly ships America's most delicious, iconic foods nationwide. Get 20 % off your first order by using the promo code TWIST at checkout. All right, everybody. Welcome back to This Week in Startups. There's tons going on in the news. I heard Gemini 3 is out. I heard Grop 4.1 is out. Everything is happening so quickly in our industry. I'm having a hard time keeping up. No, it's been really, really busy. I want to start though, Jason, with the Cloudflare outage. I don't know if you were awake or asleep. I know you've been going through some jet lag. The internet broke this morning.

2:03It actually made getting the docket ready kind of hard. It's funny when a news story really impacts us because services like ChatGPT and X and other kind of major planks, you might say, of the podcast economy broke. And for several hours, Cloudflare had these really embarrassing downtime outages. And the good news is that now that we're here, it has been resolved. They've come out and apologized for it. If you're curious, everyone, what drove this? They said that, quote, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made.

2:35And Jason, you'll know this, that cascaded into a broad degradation of our network and other services. Sure, whatever that means. It was bad, though. Yeah, you know, Cloudflare is this great CDN, I guess, content delivery network. And if you have a website and you get these DOS attacks, these denial of service, DDoS, where people would send tons of pings to your server, try to slow things down and cause a disruption in service, they figured out ways to, when all those attacks come in, to just drop them. And because it's a network, you know, if some hacking group has, you know, 10 ,000 servers or maybe even individual machines on the Internet that they've hijacked to send in these denial of service attacks, just to think about it simply, they'll hijack 10 ,000 Windows machines, send a bunch of attacks to a website, and then all of a sudden you can't read the Drudge Report or New York Times or whoever their target is.

3:29They know those IP addresses. They know what it feels like when those kind of attacks happen. They know the signature and they just drop those requests. It's pretty sophisticated. It's pretty basic. But Cloudflare has become kind of a standard for this CDN. I remember when we were running Weblogs Inc. back in the day, this 20-year-old knowledge. But we started using these different CDNs, different caching servers. We put up our own caching servers. We were handling these issues ourselves. Now everybody abstracts it to providers. What's the value of being able to abstract it to a provider like this?

4:02Is, well, you can focus on other things in your business. What's the downside? If everybody trusts, everybody's got their eggs in one basket, whether it's Amazon Web Services or Google or Azure or in Cloudflare, everybody gets affected at the same time. That's kind of what I want to talk about because normally this is a little bit outside of our wheelhouse, Jason, but there was a Cloudflare outage also in June. There was an AWS outage in October. There was a Google Cloud outage in June and there was an Azure outage in October. So you're a startup founder. And you are running a software business, as most of them are.

4:39You are using cloud services, as most of them are. Communication strategies for downtime, mitigation. Just I'm curious what your advice is to founders as they go through this apparently recurring issue. It's a recurring and non-consequential issue at this point. Like the downtime happens for an hour or two. And frankly, we're all permanently online. So it's actually good. Everybody gets a little break. Like I would be concerned if your service is down for more than a couple of hours. If it gets into like, okay, it was down this morning, but it's still down this afternoon. That's when people start to go, wait a second, maybe I should look at another option.

5:16And you just don't want to get into the, okay, it's time for me to look for another option. If you're a startup founder and your service is down in the morning for an hour, no big deal. If it goes into the afternoon, now you got a big deal. People will start looking for other solutions. When AWS and their US East One region went down, everyone kind of lost service. If you lose service with everyone else, Jason, is the penalty just not that high for your individual company? Because that's good news and better than I expected. If it was AWS, everybody gives you a mulligan. No big deal. No harm, no foul.

5:44Now, you might, if you're a financial services company, you know, if you're a Robin Hood or you're Stripe and you go down and it costs people money, that's a whole different ball of wax. You might need to give a credit, you know, and things can get, yeah, a little bit ugly. People lose money. If you are selling pay-per-view and it's Netflix and it's that boxing night, yeah, that is a big problem. So I think, you know, it's service dependent. The financial companies have a different order of duty. But, you know, if you look at the video game industry, they'll be like, hey, we're having downtime Saturday afternoon.

6:22You can't play your game for three hours. expect downtime and they communicated to you ahead of time. I don't mind that, but like, you know, Tuesday morning, I mean, I think Cloudflare likes to say they power or protect about 20 % of the websites online. That's a pretty big chunk of the internet to bring down yourself. Going back though to what you said about their footprint and how they help protect websites from DDoS attacks and they do content delivery around the world. Cloudflare just bought a company called Replicate, Jason. And this is a startup that I hadn't spent enough time looking at.

6:51raised a couple of rounds, including a series A from Andreessen and a series B from Andreessen. And what's interesting is they're kind of offering serverless AI access. So if you want to just basically not deal with an individual provider of AI services, you could go to Replicate and just kind of via one API hook access a bunch of different AI models, and they handled all the technical backend. And so it kind of makes sense to plug that into Cloudflare's global infra a footprint because then you can kind of have AI at the place of your consumers with limited lag and so forth. But like, is it kind of crazy that I didn't really know anything about Replicate before this and it raised several rounds?

7:31How many companies are there doing this sort of thing? This is a company I'm not aware of either, but yeah, being able to use an API to just send something to a model and get a result back is kind of how people are running these systems now instead of you can run your whole application on AWS or Azure or Google Cloud, Oracle Cloud, and then just be calling APIs from Grok, Claude, Anthropic, or ChatGPT. That does make it a lot easier, doesn't it? You don't have to like stand up a bunch of hardware. You don't have to stand up like a bunch of instances. So it has become quite modular. I do think at a certain point in time, people are going to want to stand up their own language models with their own data to just not educate other LLMs and to keep their data private.

8:27So I think the future is going to be, you know, a lot of people running their own deep seek instance, you know, over at a Google Cloud or Azure, et cetera. Yeah, I do think I'm just seeing it with startups. Let's say you have a startup that does some obscure thing like, I don't know, building codes, right? And you're going to fine tune this model to do building codes. And it's going to be for architects and builders, developers to understand all these different codes on a global basis. You know, your Starbucks, your opening stores everywhere, you need access to all these codes. And you're taking all the time to go collect that data and to normalize it and train the model.

9:09Do you want to train ChatGPT's model or Anthropix model, have them get the benefit of it? Or do you want to just create an island, put all your data into it and not have them see the prompts, not have them learn from your prompts, not have them learn from what are the queries you're sending from your customers to them? I think people are going to become very, very cautious about the large language models becoming competitors. And I said it before on the shows, everybody listening to me knows it's gonna be a big end of the year, November and December. You're trying to get a lot of things done at work, but Thanksgiving's coming, Christmas is coming, and you're gonna need some help gifting and hey, getting ready for your Thanksgiving table.

9:55And we've got the perfect partner, my favorite, Gold Belly. Yum, yum. Get yourself some amazing gifts for your partners. Get some amazing food for your table like Martha Stewart's salted caramel chocolate cake. How about this? John's of Bleecker Street, New York style pizza. Amazing. Lombardi's Pizza. These are some of the best pizza makers in the world. It's a great way to reward your team members. I do it. I order great stuff from around the country. Just surprise my team with it when we have team meetings. So if you're looking to reward your team for doing a great job in 2025, maybe you want to surprise your loved ones with a special holiday treat.

10:32Here's how you do it. You go to goldbelly.com, Use the promo code TWIST and you get 20 % off your first order. That's goldbelly.com and use the offer code TWIST. Some of them are positioning themselves as like neutral third parties. They're not going to be in your business. Other ones, you know, that have to spend a trillion dollars. Quite possible they need to make two trillion dollars. How are they going to make that money? I think they're going to compete with you. So for this startup that was working hypothetically in building codes, and they took the time to go find all these building codes.

11:07Let's say they hired a hundred researchers to full time for a year and they spent$10 million on these hundred researchers to go collect all this information, normalize it, whatever. And then they feed it into somebody else's language model and now they learn this information. Not good. So in that case, Jason, the corollary or the follow-up question is just, are the startups that you're seeing that are kind of rolling their own models, are they using, I presume all models from China than the open source stuff from your moonshots, your deep seeks, et cetera? I think this is the year this is happening.

11:42So TBD. But we'll, you know, let me ask a couple of them if they're willing to talk. So producer Marcus, like maybe we should see if there's somebody in our portfolio who's doing this. We can just ask in founder university, et cetera, anybody working on open source AI models and training themselves and running themselves and maybe they can come on the show and talk a little bit about it. But it is definitely a trend that's going on. And I saw Andreessen Horowitz was publicly saying, hey, a lot of our startups are using DeepSea. That's what I was going to say. Martin Casado, I think, said that it's like 80 % of startups they see are using open source models from China.

12:16They're probably using other models as well, by the way. It's just... Not exclusively, but I mean, like, you know. I think that's how people took that quote, though. People took it as like, oh, they're picking the open source Chinese model over open AI's model. I think it's in addition, or maybe they're experimenting with them. So in other words, they're aware of them. They've got an instance running and they're sending some jobs to it. And why wouldn't you? I mean, I think ultimately what's going to happen is we're going to have a large language model on our Mac mini M17. If it's an M, I think I have an M4 right now, and I guess someday I'll have an M5, you know, somewhere around m10 you know i'm gonna have four m10s or an m10 will have four microprocessors in it and some amount of bandwidth and ram that will give it the ability to run my language model locally then when i do that search and i say hey show me all the photos of us skiing and uh you know a bulldog in the background because i'm looking for those pictures of the bulldogs playing on the mountains while we're skiing uh it's just going to do that locally not in the cloud it's going to index my photos locally.

13:24It's going to have all my Gmail local. It's going to, you know. So do I then have like a server in my basement, Jason, that has like my family collection of GPUs or am I just running a single Mac mini pro 2015 edition or 2030 edition, whatever? I think that the ultimate manifestation of all this is going to be Apple will have local language models and they'll be pitching us on they're encrypted they're private your data's safe just like they do for iCloud we can't get your iCloud stuff if the government comes and asks us to crack it open we can't get there just like you know I guess Sam Waltman gave a warning like hey if somebody sends us a letter we're sending them everything you've talked about and when this kid tragically committed suicide recently they had all the logs right like yeah they're gonna dump the log.

14:16So if you're in a relationship with ChatGPT, they have all that. Whatever you said to your ChatGPT, they've got it stored. But the Apple Point's great though, Jason, because if you wanted to have that model succeed, you would need to have a smartphone, a desktop computing experience, a Slate computing experience, a headset computing experience, a future glasses computing experience, and you need to be in cars, perhaps do something like CarPlay. They literally have all the surfaces already that they would need. Whoever has the operating system, I think is going to have of a real operating system plus privacy is going to be a really interesting combination.

14:51I've talked ad nauseum about my love of the Comet browser and watching my browser, you know, go to United Airlines and find a flight for me and put it into, you know, the shopping cart was like, hmm, that's interesting. Or I used it the other day. I said, go to Blue Sky because I don't have, I'm not following anyone in Blue Sky. And I just said, I think I said, follow anybody on the Explore page. um, who's like featured, just follow like 200 people. And it went and it followed like 20 people. It's like, ah, 20 is the limit. And I was like, why is 20 the limit? Okay. Do it five times. It wasn't good five times.

15:26The idea that Apple is watching what you do on your phone. And there's, um, this give Apple intelligence. I don't know if you've seen this setting, you can pull it up. We've talked about this. Yeah. Give Apple intelligence access to what you're doing on this app. So if you go to the settings for any app on your phone, there's a thing like give Apple intelligence access to it. What does that mean? What that means is Apple intelligence is watching how you use your phone and they're watching how you use, I don't know, the Contacts app, your phone app, your superhuman app. And then over time, you're going to be able to say, go do this in the StubHub app.

16:07Go find Nick's tickets. But it will have recorded and studied it over and over again. People asked online quite a lot about how to turn this off. But to me, if you're going to use any sort of intelligence on your iOS device, you're going to want it to connect into other applications, Jason. So just. So Apple Intelligence Siri, yeah, learn from this app. Next on the docket, we have Grok 4.1. We'll talk about Gemini 3 in a second, but I think it's worth stopping and saying, look at what XAI did, because this is a company, Jason, that started later than OpenAI and has really done, I'm just going to say, impressive things.

16:40I know a lot of my friend group aren't the biggest Elon fans, but I do think it's important to give credit where it's due. And Grok 4.1 is really good across a number of metrics. One, they did a lot of work on reducing the hallucinations, which I think is a very important step compared to Grok 4 fast. It's a dramatic improvement. And I think most importantly, based on the usual kind of panoply of things that Ella Marina tracks in its battleground for different AI models, when 4.1 was released onto the charts, observe. It went straight to the top. Grok 4.1 and Grok 4.1 Thinking took the absolute pole positions.

17:17But this is a real accomplishment. And I think it just shows that there's several different AI labs here in the US that are able to successively create state-of-the-art models. And I think that's very encouraging for startups. I think it's encouraging for the AI economy writ large. And I think it also pushes back, Jason, against some of the doomerism that's been kind of building in the last three months. It feels like AI needs a win. And I think this is an example of such a win. Less hallucinations, obviously better. And one of the great things is, I guess, that now that we have these leaderboards, it's motivated all these companies and gotten these teams super excited about claiming the top spot.

17:55So kind of cool that these things, you know, somebody came up with this, this concept of the arena and all these different tests. I do wonder if building for the tests at a certain point is kind of like kids mastering the SATs. And like at a certain point, maybe that's not what's important. Maybe like there are other things. So I wonder if these tests are changing. I don't have enough information on that. Like how are these tests changing? I know early on, some of these, one of the tricks that was occurring is when you give people an incentive, like a leaderboard like this, people then try to game the leaderboard.

18:38What's a way to game the leaderboard? Doing unnatural acts. Like if you know it's gonna ask certain types of questions, then studying for those types of questions or feeding those exact questions or those multiple versions of those questions into your training set. In other words, kind of gaming the system. Precisely. And I think that that's where I'm just getting a little concerned of, are we actually making progress at solving the real world problems people have, or are we getting good at acing the SATs? And that is a question for me. People are worried about, I'm trying to remember the exact phrase of this, but it's like diffusion or when the model gets so suffused with people knowing how it asks questions that it kind of loses meaning because you can tune.

19:26And if you think back to Meta's launch of Llama 4, Jason, if you recall, they actually made a bunch of different versions of Llama 4 and then ran them through the same test to get the best results. And they got mocked for this pretty mercilessly because it was clear they were trying to cook the numbers to appear impressive. The reason why I shared LMArena information versus just kind of a raw rundown of individual benchmarks is that this is their users in a head-to-head battle choosing which result is better with the model names masked. And so it shows what people that are actually using these models think.

19:56And I'm hoping that provides a better perspective on actual progress here. I'm a big enough deal now that I can afford to hire my own admin team. Look at me. They handle all the details of running my company. But if you're a startup, you need to spend your time obsessing about your product, not filling out paperwork and doing all these books. That's why I want to tell you about Every. They've worked with over 1 ,000 startups from first-time founders to VC-funded teams and more. And they've got the experience to help your company navigate whatever's coming around the bend. Maybe it's time to incorporate your startup so investors take you seriously.

20:29Every is going to take care of those filings for free without any unnecessary delays or legal fees. Hey, maybe you just got some new funding and it's time for you to scale up with every. You'll get 3 % cash back on every dollar you spend on your company's card. Hey, maybe it's time to say goodbye to a member of your team. It happens. Well, they've got an employee offboarding checklist. And you know, I love a checklist. It's going to help you ensure a smooth transition while protecting your business, both legally and financially. Plus, you'll get a$2 ,000 bonus when you move$250K or more into your Every account.

21:01So for your incorporation, banking, payroll, benefits, accounting, taxes, and other back office administrative needs, visit every.io. That's E-V-E-R-Y.io. But anyways, let's move on to this morning's big news. Similar idea, new AI model, this time from Google. Gemini 3 Pro is out. So limited early access. It's rolling out as we speak. Earlier, Jason, you mentioned how the benchmarks are a little bit dodgy. So I don't want to over-index on this particular data set, but I think it is worth just sharing with people what the numbers show. So here's the Gemini 3 Pro benchmark results. I just wanted to talk about the difference in how things have changed.

21:44If you look at their Humanities last exam results, Claude Son at 4.5 gets about 14 % right versus about 38 % for Gemini 3 Pro. big improvements in math and screen understanding. And these numbers really are materially better than other models that are kind of currently state-of-the-art. And I just really think this pushes back against the idea that AI improvement has slowed dramatically. So I view this all as very, very bullish. And there's a little bit of a teaser though. One last thing here. If you take a look at this chart right here, there's an upcoming variation of Gemini 3 Pro called Gemini 3 DeepThink.

22:23does even better and is crushing the Arc AGI benchmark, which I think is very important. So I believe the kids say that Google cooked here and that a lot of folks are really going to dive into this pretty much right away because this is going to be. Looks incrementally better. You know, to be honest, we need to have an expert on here to tell us if it's actually better or not, or like the actual extent, because this history's last exam, I do know, having talked to some folks, they are some pretty hard questions. I would like to have an expert on here who created that. I wonder who created histories.

22:58Somebody created this. Somebody creates the new questions on it. I want to know who that person is. In a more real world application than Jason, to put it into more kind of functional context, browser use, a Twiz500 company, gave it very strong marks. Instead, it was much better than previous models from the company. Box's Aaron Levy compared it to Gemini 2.5 Pro and found that in the box context, it was much better across a number of metrics. And Stripes' Patrick Collison gave it a pretty strong review. You might say he gave it a pretty hard task and was very impressive when he put together a compendium of recent research into genetics.

23:32So some people in the field are actually reporting. That's good to know. Got rave reviews from some leaders who actually use it at their major company. So Aaron Levy likes it. Stripe boys like it. That's pretty good endorsement that they're seeing a difference. It's at least as good as people hoped, unlike GPT-5, which was not as good as people hoped. So at a minimum, the vibes are persisting. So that's pretty good. And then just for fun, Jason, I thought we'd talk about the Polymarket perspective here. There's a funny little wiggle here in this Polymarket. If you're on the audio version, we're looking at which company has the best AI model by the end of 2025.

24:06The end date, of course, is December 31. And there was this really interesting wobble, Jason. We're zoomed into a one-day context here. But look, Crock 4.1 came out and people got really excited about it. Maybe XAI is going to win. And then Gemini 3 came out and got and crushed it again. So the market is once again betting that Google is going to run the AI game through the end of 2025. How does this one resolve? I always bring the same thing up, which is we need to understand how the bet resolves. So how do they define which one has the best AI model again? The market will resolve to, yes, if any model owned by Google has the highest arena score based off the chatbot arena LLM leaderboard.

24:43So it's the chatbot subset of the LLM arena leaderboard that we discussed earlier. Again, head to head, no models listed. People just pick which one they think is best. So it's kind of a blind taste test. And Google has to have the highest score in that area. And it didn't when Grok 4.1 came out. And now it does again, now that Gemini 3 has come out and surpassed it. That means Google is going to have the best model. Period, full stop. Like if the sharps are saying it, there's somebody on the inside who's making this bet, I bet. I bet there's people inside the, there's some group of developers.

25:18This is my hypothesis. There's some group of developers who run LM Arena and there's people inside of Google who are betting on themselves. Like a basketball, like Michael Jordan betting on himself to win a game or cover the spread. It's like, it's in his control. so when I see like every that sort of level of consensus I kind of think it's the developers themselves working on the language model or the person and or the people who actually run the leaderboard who are watching uh the leaderboard activity and they have some inside information this is why these you know polymarket is like really changing the world because there's an opportunity to make money from your knowledge.