Karpathy Leaves OpenAI as OpenAI Launches Memory for ChatGPT

0:00Today on the AI Breakdown, Andre Karpathy has left open AI as ChatGPT gets memory. Before that on the brief, NVIDIA has jumped in the on-device chatbot game. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube, our newsletter, and our Discord.

0:24Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. One of the big trends in LLMs right now is to try to put them in new contexts rather than just being a cloud-based bot. Specifically, people are looking into things like personalization, how they allow individuals and businesses to have LLMs that reference their own data. Two, many companies and projects are also looking at how to get LLMs running on mobile phones or PCs in either a fully offline or hybrid cloud on-device sort of way. This is, of course, where Apple's efforts are. Samsung and Google and the other handset makers are thinking in similar ways.

1:00And now it appears so too is NVIDIA. They've recently announced NVIDIA Chat with RTX, which they call your personalized AI chatbot. They write, Chat with RTX is a demo app that lets you personalize a GPT-LLM connected to your own content. Docs, notes, videos, and other data. Leveraging RAG, you can query a custom chatbot to quickly get contextually relevant answers. And because it all runs locally on your Windows RTX PC or workstation, you'll get fast and secure results. So like I said, there are a lot of projects that are working on something similar right now, but obviously a company at NVIDIA's size and scale doing so shows what a big deal this is.

1:35Brian Romley writes, Boom! Open source chat with RTX is here and I am testing it now. You can feed it YouTube videos and your own documents to create some reason get relevant answers based on your data. This is a massive challenge to OpenAI. Now, for those of you who are interested in this space, one project that I am watching is Open Interpreter. Open Interpreter frames themselves as a new way to use computers. Let LLMs run code on your computer to complete tasks. You can check it out at openinterpreter.com. Now, staying on the theme of NVIDIA and AI chips for a minute, we recently got comments from Jensen Huang saying that he didn't think that it was going to take the$7 trillion that Sam Altman has apparently been talking about to meet the world's compute needs.

2:12Part of the reason for that is that he said he expected prices to come down, and now the CEO of Databricks has said something very similar. For those unfamiliar, Databricks is a company that helps big enterprises and software use AI. It's competitive with, for example, Amazon Bedrock. They're very invested in the idea that there won't be one winner-take-all sort of LLM, but lots and lots of different solutions that run the gamut from open source to closed source that can meet any different company's particular needs. The CEO of that company, Ali Goetze, said recently that just as internet bandwidth constraints evaporated in the 2000s, quote, the same thing will happen with GPUs.

2:44Now part of where this came from was a comment in an interview where he said, The challenging thing is a lot of these startups started on a weird funding or GPU commitment model that I think is going to pose an extra challenge to them going forward. We're going to see a lot of turbulence in the next 12 months because of that. Going further, he said, People were buying these GPUs and doing Bitcoin mining on them. Then Ethereum decided to switch the model and they didn't need GPUs anymore. But then it turned out that OpenAI is training gigantic models and we do need GPUs after all. Then there was this one vendor, NVIDIA, and everybody rushed to get these GPUs.

3:12There's been such crazy scarcity, but in the 2000s there was a similar thing around bandwidth. It turned out that actually capitalism supply and demand takes care of that problem, and the price of bandwidth just plummeted and bandwidth was abundant everywhere. The same thing will happen with GPUs. So then, what happens to the startups that do GPU money laundering, which is when they get huge funds from big strategic investors? The valuation of those companies were high because they had to raise hundreds of millions of dollars just for the GPUs. So what happens when the price of those comes down, and there's people that are sitting with three-year commitments on these GPUs?

3:40That's going to pose a big challenge for anyone who's overcommitted on the GPU spend side. The whole interview with the information is pretty fascinating, and I highly recommend it. And even if Aldia is right, Wall Street is still very focused on that bottleneck that exists here and now. In the week since UK chip designer Arm Holdings announced its latest financial results, their stock price has nearly doubled, up 98%. Unrivaled Investing on Twitter wrote, My broader conclusion is that there's a huge mismatch between demand and supply for AI stocks, and this is translating into absurd valuations where the companies can still grow a lot, but due to lofty prices, the shareholders likely lose.

4:14The reality is that the winners of the AI revolution will be far spread and not concentrated in just a few players. The challenge for these investors is that a lot of these initiatives and companies are still very early days. Now, moving over to the new product side of things for a moment, a couple really interesting things. Cohere has introduced AYA, which they call a state-of-the-art model and dataset pushing the boundaries of multilingual AI for 101 languages. The project involved 3 ,000 independent researchers, 56 language ambassadors, and people from 119 countries, and is meant to try to advance multilingual AI, given that much of the state of the art in LLMs is of course developed in English right now.

4:48Stability introduced a new open-source text-to-image model called Stable Cascade that some are arguing is the best open-source text-to-image model yet. They write, During our evaluations, we found Stable Cascade performs best in both prompt alignment and aesthetic quality in almost all model comparisons. As with most stability releases, Stable Cascade is now available on a non-commercial license. Quick update on the legal side. Earlier this week, a U.S. district judge dismissed four of the claims in the Sarah Silverman lawsuit against OpenAI. The dismissed allegations include vicarious copyright infringement, violations of the DMCA, negligence and unjust enrichment.

5:21But the judge did allow Silverman and her co-plaintiffs core claim of direct copyright infringement to move forward. The plaintiffs now have a month to amend their class action suit in response to this ruling. So friends, never a dull moment in AI. If you want to hear more about the latest moves at OpenAI, including new features for ChatGPT and a majorly high-profile researcher leaving the company, come back soon for the main AI breakdown. Welcome back to the AI breakdown. OpenAI is once again dominating the news today for both good and not so good reasons, depending on your perspective. The first story to cover is that OpenAI researcher Andre Karpathy, who is arguably one of the best-known people at the company, has decided to leave.

6:00The information writes that Andre was working closely on an AI assistant project in collaboration with the company's head of research, Bob McGrew. We talked a couple weeks ago about how OpenAI is moving and increasingly trying to pull competition in the AI space to this agentic future, rather than just leaving it in the world of LLMs. Now, Andre Karpathy has had an interesting on-again-off-again relationship with OpenAI. He was one of the founding members, but then left and spent five years at Tesla. At Tesla, he was leading the development of the company's autopilot semi-automated driving software.

6:29About one year ago, he posted on X that he was coming back to OpenAI, and people were incredibly excited. Said OpenAI spokesperson Kayla Wood, Andre is departing to pursue personal projects. We are deeply grateful for his contributions and wish him the best. His responsibilities have transitioned to a senior researcher who worked closely alongside Andre. Now, after this got a ton of buzz, Carpathy himself took to Twitter where he wrote, Hi everyone, yes, I left OpenAI yesterday. First of all, nothing happened, and it's not a result of any particular event, issue, or drama, but please keep the conspiracy theories coming as they are highly entertaining.

6:59Actually, being at OpenAI over the last year has been really great. The team is really strong, the people are wonderful, and the roadmap is very exciting. And I think we all have a lot to look forward to. My immediate plan is to work on my personal projects and see what happens. Those of you who have followed me for a while have a sense of what that might look like. Now, one thing I had noted before this came out is that Karpathy has been tweeting a lot more of late. For example, he wrote a whole screed on the shortification of learning, basically saying that education and entertainment aren't the same thing, and that learning should be hard in some ways.

7:28And on Monday, he wrote, The point being here, that the Occam's razor explanation of a very smart guy just wanting to work on something different seems like in this case it might actually be true. Now, still in terms of your experience and my experience with OpenAI's products, the bigger news is the announcement of memory. Joanne Zhang, who does product at OpenAI, tweeted, We just launched a small experiment for memory on ChatGPT. How it works? It's quite similar to custom instructions, except ChatGPT is the one driving it. Basically, we taught ChatGPT to keep a notepad for itself. Every time you share information that might be useful for future reference, it'll hopefully add to the notepad.

8:20Two, you'll be in control. There are five different ways to manage memory. You can, one, just tell ChatGPT to remember or forget something in chat conversationally. Two, use temporary chat for one-off convos involving topics you don't want ChatGPT to pick up on. Three, delete individual memory snippets. Four, clear all memories. Five, disable memory altogether. Finally, she writes, it's a small experiment. I'm sorry if you're not in the experiment and want to be. We're taking a bit more time than usual with this feature, given how memory, almost by definition, will take a bit longer and more interactions to build.

8:48We can't wait to learn from this experiment and iterate on the feedback so that personalization features like this can be truly useful to everyone. Now, in the example screenshot they share of managing memory, they have things like has a two-year-old daughter named Lina, daughter Lina loves jellyfish, prefers meeting summaries to have headlines with bullets and action items summarized at the end, prefers assistance with writing blog posts to be more concise, straightforward, and less emotive, loves to travel, is interested in traveling to Mexico for April vacation. Basically, these are all things that ChatGPT has extracted from your conversations and uses to make future results of conversations with ChatGPT even better.

9:22They write as you chat with ChatGPT, you can ask it to remember something specific or let it pick up details itself. ChatGPT's memory will get better the more you use it, and you'll start to notice the improvements over time. Of course, the feature that they referenced was custom instructions, which was a really simple feature that came out last year that showed the direction that ChatGPT was headed. The custom instructions field simply says, what would you like ChatGPT to know about you to provide better responses, and how would you like ChatGPT to respond? So for example, one of the things that I put in there was that I do a lot of content and education around AI, and that the way I liked it to respond tended to be at an undergraduate comprehension level for people who have some technical understanding but who aren't developers or technical themselves.

9:59Those are things that I often put in individual prompts that I was able to embed in ChatGPT's responses in general. In many ways, it feels like memory is just a next step on that pathway that expands that capacity and just nudges ChatGPT towards even more personally useful. Now, given how it appears that OpenAI, or at least Sam Altman's goal with ChatGPT, is to have it become the most powerful AI assistant, it makes sense that it could learn more about you over time and customize and tailor its responses to your queries based on that knowledge that it has. Now, by and large, the feedback has been really good.

10:29People are very excited about this feature. SWIX is among them, but also has a little bit of advice, writing, ChatGPT finally getting persistent memory and temporary chat. I believe most chats should default temporary with opt-ins saved to history after the fact, But I wonder if OpenAI will ever recognize that in its pursuit of data. As for memory, it looks like this is almost a memgbt implementation. The flaw here is that you have to manually delete memories instead of having the AI figure out when to delete it. Net result is it is much easier to add memories than delete, and there is no demonstration of edit in place.

10:57Probably more work to do here to make it good, but this is a nice start. Now, speaking of starts, another start that is somehow related to OpenAI, although not of OpenAI, is that OpenAI's chairman of the board, Brett Taylor, formerly co-CEO at Salesforce, has officially announced a new company called Sierra that is designed to build AI agents for customer service at major corporations. The company has raised$110 million in that pursuit. TechCrunch writes, At its heart, the new company is a customer service bot. That's not actually all that earth-shattering, but the company claims that it's much more than that, with its software going beyond an extension of an FAQ page and actually taking actions on behalf of the customer.

11:33In a blog post, they wrote, Sierra agents can do so much more than just answer questions. They take action using your systems, from upgrading a subscription in your customer database to managing the complexities of a furniture delivery in your order management system. Agents can reason, problem solve, and make decisions. Now, agents are, of course, one of the big themes in AI development right now. And it's not surprising that someone with Brett Taylor's background, especially a six-year spend at Salesforce, has investors excited about the possibilities. I think honing in on a particular discipline like customer service is the way we're going to see AI agents come to market.

12:04And so this will definitely be one to watch. Now, of course, some people did ask about conflicts of interest with OpenAI, given that they also seem to be driving towards an AI agent future, to which Taylor said that he'll simply recuse himself, quote, whenever there is a potential for overlap. We'll see just how competitive things get over time and whether that will actually work. Lastly, we got a little bit more information from OpenAI CEO Sam Altman when he spoke at the World Government Summit, when he had this to say about GBT5. I was sort of laughing a little bit because this is going to sound like an annoying answer, but I think it is the important part.

12:35It's going to be smarter. There are all of these other things. We can talk about it'll be better at these kind of tasks, it'll be multimodal, it'll be faster, who knows what. But the thing that I think really matters is it's going to be smarter. And this is a bigger deal than it sounds, right? Because what makes these models so magical is that they're general. And so if it's a little bit better, if it's a little bit smarter, that means it's a little bit better at everything. And the thing that I think is most exciting is it's not like this model is going to get a little better at this task and not really better at these.

13:12It's not that. It's because we're going to make the model smarter. It's going to be better at everything across the board. So friends, lots happening in the world of AI in general, but certainly OpenAI in specific. However, that is going to do it for today's AI Breakdown. Until next time, peace.

13:38Thank you.