Code Interpreter is GPT-4.5: A Summer AI Technical Roundup [feat. Swyx and Alessio of Latent Space]

28:05Going all the way back to the history of open source models, in Luther AI, when they trained GPT-J and GPT-Neo as the first reproductions of GPT-3, they released the data first. Stable Diffusion, when they trained Stable Diffusion, they released Lion 500B first. And that's, I think, reflective of the normal sequence of events. You release the data, then you release the model weights. But now we're just skipping the data part. And I think it's fair. It's a way to guard yourself. I think one of our conversations, I think it was Mike Conover, when he was talking about comparing our current AI era versus the 2000s era in search engines.

28:48He basically said all of the public publishable information retrieval research dried up because all those PhDs went to work at Google and Google just sat on it. And this is now a fight for IP. And I think that is just a very rational way of behavior in, I guess, a capitalist AI economy. So one of the things that we were talking about before, starting with the code interpreter 4.5 and why or GPT 4.5 and why they might not call it that, is the emergence of this sort of regulatory, if not pressure, certainly intrigue. Do you think that there's potentially an aspect of that when it comes to why people are so jealously safeguarding the data?

29:27Is there more risk for being open about where the data is actually coming from? The book's three examples is probably good. So MPT trained their model on a dataset called Books Tree, which is 190 ,000 books, something like that. And then people on Twitter were like, well, this stuff is not, you know, in the free, you know, it's under copyright still. You just cannot. Public domain. Yeah, it's not in the public domain. You can just take it and train on it. But the license for some of these books is like kind of blurry, you know, on like what's fair use and what isn't. And so there was like this old thing on Twitter about it.

30:04And then MPT, you know, Mosaic first changed the license and they changed it back. And I think Sean, Sean Presser from Luther was just tweeting about this yesterday. And he was basically saying, look, as ML engineers, maybe it's better to not try and be the main ethics knight and just say, hey, look, the data is open and let's try it. And then maybe people later will say, hey, please don't use the data and then we can figure it out. but like proactively not using all of this stuff can kind of keep the progress back and you know he's more coming from the side of like a luther which is like doing this work in public so for them it's like hey you know if you don't want us to train all this is fine but we shouldn't by default not do it versus if you're meta you know they said that they trained llama on like stuff available on the internet they didn't say they trained llama on stuff that is licensed to train on.

30:59It's a small difference. The other piece of this that I wanted to sort of circle back to, because we kind of breezed over it, but I think is really significant. We did get a little lost in this conversation around open source definitions. And I don't think that's unimportant. I think that people are rightly protective when a set of terminology has a particular meaning and a massive global corporation sort of tries to like nudge it towards something that is potentially serving their ends versus actually being by that definition. But I also think that your point, which is that functionally relative to the rest of the space, it probably doesn't super matter because what people mean is almost more about functionally what they can do with it and what it means for the space relative to more closed models.

31:43And I think one of the big observations has been that the availability from when Llama One was fully leaked, the availability of all of that has pretty dramatically changed, one, the evolution of the space over the past few months, and two, I think from a business standpoint, how the big companies and incumbents have thought about this. So another big conversation this month, going back to sort of the venture capital side of your life has been the extent to which companies or startups are, or big companies are not wanting to sort of sign on with some startup that's going to offer them, you know, AI, whatever, because their technical teams can just go spin up sort of their own version of it because of the sort of, you know, availability of these open source tools.

32:33But I'm interested, I guess, in bringing the sort of open source, you know, in air quotes side of the conversation into the realm of how it has impacted how companies are thinking about their development in the context of the AI space? I think it's just raising the bar on what you're supposed to offer. So I think six, nine months ago, it was enough to offer a nice UI wrapper around an open AI model. Today, it isn't anymore. So that's really the main difference. It's like, what are you doing outside of wrapping the model? and people need more and more before they buy versus building. Yeah, I think it actually moves the area of competition towards other parts of productionizing AI applications.

33:22I think that's probably just a positive. I feel like the competitive pressure that Meta is putting on OpenAI is a good thing. One of the fun predictions that I made was in the next six months, GPT, OpenAI will open source GPT-3, which is not open source. It's so far behind the state of the art now that it doesn't matter as far as safety is concerned. And it basically keeps OpenAI in the open source AI game, which would be nice to have. Of the things that people have been building, you called out a couple context window expansion, but have there been any that really stand out to you as super interesting or unexpected or particularly high potential?

34:02One of our short term podcast guest uh the mlc team they worked on wrapping llama 2 to run on uh macbook gpus so i think that's like the the most interesting gap right it's like how do we go from paper token to like unlimited local use that's one of the main main things that keep even people like me from like automating a lot of stuff right it's like i don't want to constantly pay open ai to do menial stuff but if i could run this locally and do it even if it's five times lower i would do it. So that's a super exciting space. Yeah. I would say beyond that, there hasn't been that much. I mean, it's only a few weeks old, so there hasn't been that much emergence coming from it.

34:43I would definitely say you want to keep a lookout for basically what happens in post-LAMA 1, which keep in mind, it was only in February. The same thing that happened with Facunia, Alpaca, and all the other sort of instruction to you and sort of research type models, but just more of them, because now they are also commercially available. We haven't seen them come out yet, but it's almost a guarantee that they will. You can also apply all the new techniques that have emerged since then, like JSON former, because now you have access to all the model weights to LAMA. And I think that will also create another subset of models that basically was only theoretically applicable to sort of research quality models before.

35:30And so now these will be offered commercially as well. So like, yeah, nothing like really eye-popping, I would say. But it's been five minutes. Yeah, it's been a very short amount of time. And the thing about open source is that the creativity on lock is very hard to predict. And actually, I think happens a lot in the, let's just say, the less official part of the economy, where I've been focusing a lot recently on the sort of AI girlfriend economy, which is huge. I feel like it's not polite conversation that the amount of AI girlfriend and AI husbandos, AI wife who's in AI husbandos that people have been talking about.

36:08But it's real. There are millions of users. They're making a lot of money. And it's just virtually not talked about in polite SF circles. It feels like one of those areas that's going to be an absolute lightning rod when it comes to the societal debates around this technology. You can feel it. People are going to hone in on that as example A of a change that they don't like. That's my guess, at least. So I have a really crazy longer term prediction, maybe on the order of 30 to 50 years. But AI Girlfriend for Nobel Peace Prize, because what if it solves the loneliness crisis? What if it cuts the rate of terror and school shootings by half or something?

36:51That's huge. My wife and I have joked about how every generation, there's always something like they always think that they're like so far ahead and they think that there's nothing that their kids could throw at them that they just like fundamentally won't get. And without fail, every generation has something that seems just totally normal to them that their parents generation writ large just like has such a hard time with. And we're like, it's probably going to be like AI girlfriends and boyfriends. We're going to be like, yeah, but they're not real. They're like, yeah, but it's real to me. You know, or having debates with our future 13-year-old.

37:26Our kids are only four and two now. So it feels like maybe the right timeline. Yeah. I've heard actually of all people, Matthew McConaughey on the Lex Rebman podcast. What? Yeah, yeah. No, he was great. Shout out. Shout out. Shout out, Matt. They were kind of talking about this and they were noodling this idea of like computers helping us being better. so kind of like we have computers learn how to play chess and then we all got better at chess by using the computers to like learn and like experiment they were talking about similarly and interpersonal relationship maybe you know it doesn't have to be you shut off from from humans but it's like using some of these models and some of these things to actually like learn you know how to better interact with people and if you're like shy and an introvert it's like okay, I can like try these jokes or like these conversation points with a model.

38:17And like, you know, it teaches me, Hey, that's not okay to say, or like, you know, you should maybe be more open or, or I don't know, but I think that's a more wholesome view of it than like everybody just kind of runs away from society. And that's like 10 AI friends and doesn't talk to humans anymore. It's much less sexy to just say like AI friends, right. And even though like there's the, If you look at the possibility set, the idea that people might have this sort of, to your point, like conversational partner that helps them effectively work through their own things in this safe space, that doesn't necessarily lead to romantic attachment just because the movie Her came out, right?

38:59Right. It can just be a panel of experts. I do have plans to build a small CEO, which is my own boss, and just for me to check in. And I actually will flag out, just living here in San Francisco, you come across a lot of AI engineers who are interested in building mental wellness products. And a lot of these will take the form of some kind of journal. And this will be your most private thoughts that you don't really want to send anywhere else. And so actually, all of these will make advantage of open source models because they don't want to send it to OpenAI. And that makes a ton of sense. For people who want to try it out, I'll also give a shout out to circlechat.co, which is something I just came across from one of my friends here in the co-working space that I have, where it's one of those situations where you can actually try out, like having a conversation and having a group of AI friends chime in and see what that feels like to you.

39:50It's the first example I've come across where someone's actually done this. Super interesting. So Lama and Code Interpreter, I think, stood out pretty clearly as really big things to touch. I wanted to check in just as we sort of start to maybe round the corner towards wrapping up. Claude 2 and Anthropic, how significant was this? In what ways was it significant? Was it something that was sort of meaningful from expanding the capacity set for developers? Or was it sort of more just a good example of what you can do if you increase the context window? But that's something that might ultimately become table stakes later on.

40:25Yeah, I can maybe speak to this a little bit. It is significant, but not earth-shattering, clearly. I think it is the first time that Cloud as a whole has just been generally publicly available, used to be on a wait list. Yes, it has a longer context window, but to me, more significantly, it is Anthropic finding its foothold in the very competitive AI landscape. Anthropic's message used to be that, yes, we're number two to open AI, but we're safer. And that's not a super appealing thing to many engineers. It is very appealing to some corporations, by the way. I think having the 100K context window makes them state-of-the-art in one dimension, which is very useful.

41:08The ability to upload multiple files, I think, is super useful as well. And actually, I have met a number of businesses. I'm close friends with Sourcegraph, who are actually choosing to build with Cloud2 API over and above OpenAI, just because they are better at latency, better reliability, and better in some form of code synthesis. I think it's Anthropic finding its foothold, finally, after a long while of being in OpenAI shadow. Yeah, and we use Claude for the transcript and timestamps in the podcast. So shout out the 100K context window. You know, we couldn't do that. When we first started the podcast, we were like, okay, how do we chunk this stuff for like GPT-4 and all of that?

41:51And then I was like, just put the whole thing in here, man. And works great. So that's a good start. But I feel like they're always, yeah, second fiddle. You know, it's like every time they release something, people are like, cool. Okay. Some people like it. Most people are like, okay. I feel bad for them because it's like, it's really good stuff, you know, but they just need some help on the marketing side and the community buy-in. So I just spent this past weekend at the Cloud hackathon, which is, as far as I know, Enthalpik's first hackathon. I tweeted a pretty well-received video where I was just in the hackathon venue at 2 a.m.

42:30in the morning, and there was just a ton of people hacking there. There were like 300 people participating for Cloud. And I think it's just the first real developer excitement I've ever seen for Anthropic and Claude. So I think they're on their way up. I think this paves the way for a multi-model future. That is something that a lot of people are betting on. It's just the odds are stacked against Anthropic, but they're making some headway. I do think that you should always be running all your chats side by side against ChatDBT and Claude and maybe Llama2. so I immediately I have a little menu bar app that does that that syncs all the all the chats across and and yeah I can say I can legitimately say that Claude wins about 30 percent of the time as far as any time I give it a task to do I ask it a question which is not you know doesn't make it number one but it actually is very additive to your overall toolkit of AIs that you should use yeah it's certainly the first time that you're if you go on Twitter on any given day you will see people saying things like if you haven't used Claude you know for writing you have to try it now or so you know like people who are really who have made a switch who are have no affiliation who are very convinced that it is now part of the the suite of tools that people should really be paying attention to which I think is great yeah we shouldn't be at a stage yet where we're you know total totally in on one just one tool set I'll also mention I think this month, or at least July, was the first discussion of whether is too much context not actually a good thing.

44:06So there's a pretty famous paper, I forget the actual title of it, that shows a very pronounced U-curve in the retrieval abilities of large context models. And so basically, if the item that's being retrieved is at the start or at the end of the context window, then it has the best chance of being received. But if it's in the middle, it has a high chance of being lost. And so is 100K context a good thing? Are you systematically testing its ability to retrieve the correct factual information? Or are you just looking at a summary and going, yeah, that looks good to me? I think we will be testing whether or not it's worth extending it to 100K or a million tokens or infinite tokens.

44:45Or do you want to blend a short window like 8 ,000 tokens or 4 ,000 tokens and couple that together with a proper semantic search system, like the retrieval augmented generation and vector database companies are doing. So I think that that discussion has come up in open source a lot. And basically, I think it matches human memory, right? Like you want to have a short working memory. You know, I was thinking about it. The one other, obviously, big sort of company update that we haven't spoken about yet was around the middle of the month, Google Bar had a big set of updates. A lot of it was sort of business focused, right?

45:23So it was available in more languages. It was, you know, whatever the, the sort of from a feature perspective, the biggest thing that they were sort of hanging their hat on was around image recognition and sort of this push towards multimodality. But do you guys have any thoughts about that? Or was that sort of like not sort of on the, the, the high priority list as a, as an announcement or development this month? I think going back to the point before we're getting to the maturity level of the industry where like doing like model updates and all this stuff, like it's fine, but like people need more.

45:52you know people need need more and like that's why i quote interpreter it's like so good right it's not just like oh we made the model a little better like we added this thing it's like this is like a whole new thing if you're playing the model game if not you got to go to the product level and i think google should start thinking about how to make that work because when i search on google maps for certain stuff it's like completely does not work so maybe they should use models to like make that better and then say we're using bard in google maps search but yeah i don't know i'm kind of tuning off a lot of the single just model announcement so bard's updates i think that the multi-modality they actually beat gpt4 to releasing a generally available multi-modal model right you can upload an image and have bard describe it and that's pretty interesting pretty cool i think uh one of our earliest guests roboflow uh brad their cto was actually doing some comparisons because they have access to a lot of the vision models.

46:49And BARD came up a little bit short, but it was pretty good. It was close to the state of the art. I would say the problem with BARD is that you can't rely on them having reliable updates because they had a June update, I don't know if you remember, of implicit code execution where they started to ship the code interpreter type functionality, but in a more limited format. If you run the same questions that Bard advertised in the June blog post, that Sundar Pichai advertised in a video that he tweeted out, they no longer work in Bard. So they had a regression that was very embarrassing, obviously unintended.

47:28And it shows that it's hard to keep model progress up to date. But I think Google has this checkered history with its products being reliable. They also killed off Google Domains, RIP. And I think that's something that they have to combat, which is like, yes, they're trying to ship model products. I've met the bar people. They're good, earnest people. But they have struggled to ship products even more than OpenAI, which is frankly embarrassing for a company this size of Google. Outside of the biggies, are there any other sort of key trends or, you know, maybe not even key trends, but sort of bubbling interest that you guys are noticing in the developer community that aren't necessarily super widely seen outside?

48:09You know, one of the things that I keep an eye on is all the auto GPT like things, you know, in this month, we had GPT engineer and we had multi on who held a hackathon. and there's a few things like that, but not necessarily in the agent space. But are there any other themes that you guys are keeping an eye on, let's say? I'm sure Alessio can chime in, but I do keep a relative close eye on that agent stuff. It has not died down in terms of the heat. Even the auto GPT team, who, by the way, I work on, they're on the first floor, they're building that I work on. They're hard at work shipping the next version.

48:42And so I think a lot of people are engaging in the dream of agents. And I think scoping them down to something usable is still a task that has so far eluded every single team so far. And it is what it is. I think all these very ambitious goals. We are at the very start of this journey, the same journey that maybe self-driving cars took in 2012 when they started doing the DARPA challenge. And I think the other thing I'll point out, interest in terms of just overall interest. I am definitely seeing a lot of eval-type companies being formed and winning hackathons too. So what are eval companies? They're basically companies that you monitor the success of your prompts or your agents and version them and just share them potentially.

49:33I feel like I can't be more descriptive just because it's hard to really describe what they do just because they are not very clear about what they do yet. Langchain launched Langsmith and I think that is the first commercial product that Langchain probably the top one or two developer oriented AI projects out there and that's more observability but also will tend towards eval as well because they acquihired in AI eval projects as well so I'll just call out just the general domain of how to eval models is a very big focus of the developers here in SF. Yeah we've done two seats in companies doing agents, but they're both verticalized agents.

50:12So I think the open source motion has been auto GPT, do anything. And now we're seeing a lot of founders is like, hey, if you take that and then you combine it with deep industry expertise, you can get so many improvements to it. And then the other piece of it is how do you do information retrieval? So in general knowledge, like documents, everything is kind of flat, but when you're in specific like vertical, say finance, for example, if you're looking at the earnings from this quarter, like 10 quarters ago, like the latest ones are like much more important. So how do you start to create this like information hierarchy between documents?

50:50And then how do you use that? Instead of doing simple like retrieval from like an embedding store, it's like, how do you also start to score these things? That's another area of research from founders. Oh, I'll cut out two more things. One more thing that happened this month was SDXL. you know, text-to-image doesn't seem as sexy anymore, even though, like, last year was all the rage. But I do think, like, it's coming along. I definitely wish that Google was putting up more of a fight because they actually, at the start of the year, released some very interesting papers that they never followed up on that showed some really interesting Transformers-based text-to-image models that I thought was super interesting.

51:31And then the other element, which, you know, I'm just, like, very fascinated, by a lot of the, I don't know, like the, I hesitate to say this, but it's actually like the character and like the, character AI, let's just call it, character replica and all the sort of not safe for word versions of that. I do think that a lot of people are hacking on this kind of stuff. The retention metrics on character AI blows away, you know, a lot of the metrics that you might see on traditional social media sites. And basically AI native social media is something that is, there's something there. that I think people haven't really explored yet.

52:09And people are exploring it. No Shazir, basically the leading light behind the Transformers paper. Character AI is his company and he's always a few years ahead of it. So not to keep returning to this theme, but I just think it's definitely coming for a lot of the ways that we view things. Right now we think Co-Pilot and right now we think ChatGPT. But what we really want to speak to is a way of serializing personality and intelligence. And potentially that is a leading form of mind upload. So that gets into science fiction, actually. I don't want to go there, but I do see a lot of people working on that.

52:47Yeah, I mean, we just got a Financial Times report that says that AI personas from Meta, from Facebook could be coming next month. Oh, that's what the report was. There's one that's Abraham Lincoln, one that's like a surfer dude who gives you travel advice. So it's, you know, the sourcing is three people with knowledge of the project or whatever. And it's, you know, obviously no confirmation from Meta, but it's no secret that Zuckerberg has been interested in this stuff. Yeah. And, you know, the FT piece is actually, it's a good overview of why a company like Meta would care about it in very dollars and cents terms.

53:24And I want to state, like, the first version of this is very, very lame. Like when I first looked at character AI, It's like, okay, I want to talk to Genghis Khan if I'm doing a history class, but it's like what a 10-year-old would enjoy. But I think the various iterations of this professionally would be very interesting. So on the developer side of this, I have been calling for the development of agent clouds, which are clouds that are specifically optimized not for human use, but for AI agent use. And that is a form of character. It's a character with a different environment, with the different dependencies pre-installed that can be programmatically controlled, can give programmatic feedback to agents.

54:02And there's a protocol forming that some of the leading figures like AutoGPT and E2B are creating that lets agents run clouds. This would definitely terrify the AI safety people because we have gone from running them on a single machine towards running clusters of machines. But it's happening. So let's talk about what comes next. Do you guys have any predictions for August or if not predictions, just things that you're watching most closely? I think like for me, probably starting to see more public talk about open source models in production with people using that as a differentiator. I think right now, a lot of it is kind of like, oh, these models are there, but nobody's really saying, oh, I moved away from open AI, I'm using this.

54:50But in our, we run an early adopters community with about 1500, kind of like a Fortune 500 large companies leaders. And some of them were like, oh, we deployed Dolly in production and we're using it. We're not writing a blog post about it. So I think right now the perception is still everybody's using OpenAI and the open source models are like really toys. But I think we're going to get into September and, you know, you're not going to see a lot of announcements in August proper, but I think a lot of people are going to spend August getting these models ready and then going into end of the year and say, hey, we're here too.

55:24You know, we're using the open models. Like we don't need OpenAI. I think right now there's still not a lot of public talk about that. So excited to see more. Yeah, I'm a little bit, as for myself, this is very self-interested, obviously, but we had another agenda. You know, I wrote about the rise of the AI engineer. And I think it's definitely happening as we speak. I have seen multiple tags, like people tag me multiple times a day on like how they're reorienting their careers. I think people professionalizing around this and going from essentially like informal groups and Slack channels and meetups and stuff towards certifications and courses and job titles and actual AI teams in every single company, I think is happening.

56:05I just got a notification like two days ago that, you know, in Meta, apparently you can sort of name your name, a job title, whatever you want internally. And so the emergence of the first AI engineer within Meta has been announced. And so I think as far as the near term, I do see this career, this profession come into place that I've been forecasting for a little bit. And I'm excited to help it along. Awesome. Well, guys, great conversation. Tons of interesting stuff happening, obviously. Ironically, I think it's a relatively more quiet time in some ways than it even was. And my prediction for August is that we're going to see the extension of that.

56:46We're going to see sort of the biggest breath that we've had, at least from a feeling perspective, maybe since ChatGPT. But then we are going to rage back in September. You've got Facebook Connect in September. You've got sort of just the return to business that everyone does after August. But of course, I think the hackathons aren't going to stop in the Bay Area. So people are going to keep building. And it's entirely possible that something hits in the next four weeks that totally changes that. Be exciting to see. Looking forward.