ChatGPT All Tools and What OpenAI Is Really Building

0:00Today on the AI Breakdown, we're looking at what OpenAI is really building, and why it's less about an application, and more about the future of computing in general. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube channel, our Discord, and our newsletter.

0:24Welcome back to the AI Breakdown. Undisputedly, undoubtedly, the biggest event coming up in artificial intelligence in the near future is OpenAI's Dev Day, which is happening on November 6th. This event was announced back in September, and it's the first time that OpenAI has held a developer-focused event. Now, speculation about what is going to be announced at this event is running rampant. Altman himself, when he announced it, made sure to say that there was not going to be any sort of GPT 4.5 or GPT 5, but he thought that people would still be excited anyways. One of the areas of speculation which we've talked about previously on the show is around autonomous agents.

1:06Summed up here in this Peaky Blinder meme, where a furious Cillian Murphy is saying, no effing autonomous agents. Today we will go to a little bit of speculation at the end about what might get announced on Dev Day, but that's actually not so much the point of today's episode. Instead, what we're looking at is a set of recent updates from OpenAI around ChatGPT in advance of the event, and what they suggest about how OpenAI sees what it's actually building. So first, let's look at what has changed. If you are a plus user of ChatGPT, when you go to GPT-4, you have a set of different models that you can choose between.

1:43You can choose default, which is their model that is not connected to the internet and that has been trained on data up to January 22, or as some people have reported, perhaps a little bit later. But the point being that that is their default model. And again, it is not connected to the internet. That is in contrast with the next model option that you have, which is Browse with Bing. Now, Browse with Bing was in fact turned off for quite some time after users figured out how to use it to get around paywalls. It's only a few weeks ago that it opened back up, but for many people, if they are doing tasks that involve understanding current information, Browse with Bing is in some ways the default model that they're likely to choose.

2:19Two of the models that made waves earlier in the year are Advanced Data Analysis and Plugins. Now, Advanced Data Analysis used to be called Code Interpreter. Advanced Data Analysis is probably a slightly better name, but neither really fully captures the idea that in this version of GPT-4, the system is able to actually write code to help it figure out how to solve problems given to it by the user. I and others have done just innumerable videos about how different people use Code Interpreter or Advanced Data Analysis, with some of the most frequently cited examples being things like segmenting customers, taking a spreadsheet of data, and then using it to figure out different clusters of a market.

2:55Another capacity that people have shown is to feed advanced data analysis some set of data and just ask it to figure out what it thinks is interesting, to come up with hypotheses, for example. It can create basic visualizations, graph public data, and visualize data in a ton of different ways. Even though it's slightly old news at this point, it does remain one of the most impressive features of any AI product. Now, of course, plugins add functionality to ChatGPT by creating some access to different data sets or platforms. The one that I use most often is XPapers, which plugs into Cornell University's Archive site, which is a free open access archive of nearly 2.3 million scholarly articles.

3:31Whenever a new AI paper is released, it ends up pretty quickly on Archive, and the XPapers plugin allows me to get summaries of those things just by sharing the link to the particular piece I'm interested in. Now, at this stage, there are an insane number of plugins that you can use. I'm scrolling through a list currently across all different categories that someone put together on an open Google document, and we are currently well over a thousand plugins ranging from music to travel to SEO and marketing to education to productivity to finance to crypto to science to business and more. The last GPT-4 model is of course DALI-3 and that's certainly the one that I've been using most frequently.

4:06I already had mid-journey deeply in my workflow, particularly when it comes to YouTube thumbnails or images for newsletters, and the ability to interact with DALI-3 through ChatGPT where I can actually use English language rather than just traditional prompting gives it a much better ability for me to get my weird request across, such as these images on the screen of a Thanksgiving table with Bitcoin. I really love this huge Bitcoin pumpkin pie with the little tiny Bitcoin pumpkin pies all around it. So the new update is two parts. First of all, there is a new ability to upload and analyze files.

4:38Specifically, you can chat now with PDFs. Now, this is a big deal because there are numerous plugins that do exactly that, and it is a very, very compelling feature with just a ton of different use cases. However, the second update is what some people are calling all tools, which basically means that now instead of manually switching between these different GPT-4 models, ChatGPT itself can actually figure out which are most useful based on the instructions of the user. So functionally, what this allows for is all of these different capacities to come together much more fluidly. Let's look at a couple examples of how people have demonstrated this.

5:12Darug Walsh writes, Multimodal AI is the future. I uploaded the great Gatsby PDF to ChatGPT, then asked Dolly 3 to create images from the key scenes. Gatsby's mansion left off the page. Having all tools in one chat is a game changer. Indeed, I think this sort of layer one use case of uploading an image and asking for a modified version of it, or an image that resembles it, is one of the simplest but clearest benefits of this all tools update. Previously, the Dolly 3 model didn't give you the ability to upload an image as a reference point, which was one of its biggest limitations. Now, Anu Akash also demonstrated how the integration of Browse with Bing could be useful, asking GPT-4 AllTools, what is the price of MacBook Pro M3?

5:52Create a visual of MacBook Pro M3 with that price tag. Now, of course, for those of you who've been following along, you know that the MacBook M3 is the new advanced chip that Apple is actually promoting as significant and important to the AI space in what really represents their first time talking actively about the industry and trying to compete directly for people who are in it. So with this prompt, obviously ChatGPT had to both browse with Bing to figure out the price, and then it used Dolly3 to create the image that had that information that it had searched up integrated into the output. Now, these are all very basic demonstration uses.

6:23They're literally all, I think, designed to figure out what the capacity of all tools is, rather than really stretch and get creative about how all these tools in one spot changes what we can do. But there is one more thing that some eagle-eyed folks noticed, which is that apparently all tools has an increased token context window as well. Whereas GPT-4 previously had an 8k token context window, it appears that all tools has a 32k token window. Now, it makes sense that they would try to roll out a larger context window given that they are now allowing people to upload PDFs. But still, this is a huge update and one that people have been eagerly awaiting for months and months and months now.

6:59But where I really want to get to is the conversation that surrounds this. And now, a word from today's sponsor. Are you interested in how two top-of-mind trends, AI and crypto, can work together? If so, I have the perfect podcast recommendation for you. Web3 with A16Z Crypto, the chart-topping show brought to you by venture firm Andreessen Horowitz. Web3 with A16Z Crypto is your definitive resource for the future of the internet, whether you're already building in these spaces or simply curious about what's next. If you need a place to start, they recently released an excellent episode with Stanford cryptography professor Dan Bonet and former Google Xer Ali Yahya in conversation with host Sonal Choksi about the intersection of AI and crypto.

7:41From fighting deepfakes and proving humanity to large language models like ChatGPT, they cover it all. I highly recommend checking it out, especially if you'd like to learn more about how AI and crypto will impact our everyday lives. Beyond Crypto and AI, this show is for creators seeking more ways to truly own their work, for business leaders trying to prepare for the future today, and for innovators exploring trending tech topics. So go ahead, listen to Web3 with A16Z Crypto wherever you get your podcasts. Alex Kerr summed up a huge number of tweets that I saw when he wrote, Many startups just died today because OpenAI added PDF chat.

8:17You can also chat with data files and other document types. We had a wave of products better suited as features rather than standalone companies. Wrappers are being squeezed by OpenAI on one side and incumbents on the other. It's a rough world out there. Now, of course, what Alex is referring to is the way in which companies that filled in the gaps of the ChatGPT product are now being out-competed as ChatGPT and OpenAI adds that functionality natively. This is always the risk of building on someone else's platform. What makes it more brutal, and I think what Alex is recognizing, is that there have been a lot of different pressures on startups that perhaps they didn't anticipate going into the AI space earlier this year.

8:54Some of those are the platforms like OpenAI themselves deciding to compete in a particular area, but some of them are just the speed with which incumbents and big players have adapted, and integrated versions of a lot of different services that previously startups might have been focused on, into the tool sets that people are already using. Mike Butcher from TechCrunch wrote, New ChatGPT demonstrates how small tech companies and products that relied on parsing PDFs will eventually be wiped out by AI platforms. AI isn't just coming for blue-collar jobs, it's also coming for engineers. Now, my note is that actually I think it's coming more for engineers than blue-collar jobs, but that's the subject of a different show as well.

9:27Now, there are two analyses that I wanted to share in completion because I think they do a really good job of summarizing and putting into context these moves in a bigger way. Edipi quote-tweeted Alex Kerr and wrote, I don't know why there is any surprise. Here's OpenAI's product strategy for the next two years. You will be able to upload anything to ChatGPT. You will be able to link to any external service like Gmail and Slack. ChatGPT will have persistent memory, no more multiple chats unless you want it. ChatGPT will have a consistent, user-customizable personality, including political bias.

9:58ChatGPT will be able to respond by text, voice, images, diagrams, and video still in this time frame. ChatGPT will become much faster until you feel it's a real person, around a 50 millisecond response time. Hallucinations and non-factual errors will decline rapidly. As self-moderation improves, question rejection will decline. Now, Adapai also added a little nuance to the agent conversation. When Browsing and Abled responded to them and said, Little Birdie told me AI is pretty far ahead in developing agents that run continuously and complete tasks. Adding memory and learning over time, reasoning, different fine-tuning, and determination of output quality is far from real true loops.

10:33Browsing and Abled wrote, Little Birdie told me that OpenAI is pretty far ahead in developing agents that run continuously and complete tasks. To that, Adapai responded, AI initiated actions. I hesitate to say agents because it's a loaded word. Current browsing and data analysis tools are already AI-initiated. Would expect to see that functionality be extended to more tools. Now, expanding this line of thinking even further was Rob Phillips at IWasRobbed on Twitter. Rob wrote, As an ex-Viv with Siri team engineer, let me help ease everyone's future trauma as well with the fundamentals of assisted intelligence.

11:06Make no mistake, OpenAI is building a new kind of computer, beyond just an LLM for a middleware and frontend. Key parts they'll need to pull it off. Persistent user preferences. The biggest unlock of assistance has always been to deeply understand what someone wants in the most specific way. This is the wow moment where computers stop being scary and start being truly helpful. We did this in 2016 on Viv when our AI knew what you liked for each and every service you used via Viv and mixed that in with context like what kind of flowers you told us your mom liked. This will need to include access to your personal information to infer preferences well.

11:37External real-time data. 50 % of the utility of an LLM comes from the base training and RLHF fine-tuning, but much more comes from extending its available data with external sources. Zapier, Airbyte, and others will help, but expect deep integration with third-party apps and data pipelines. Chat with PDF is a tiny, tiny part of this. If you're only building that, think much bigger. Actual computing on virtual machines. Context windows are limiting, so AI providers will continue benefiting from running tasks directly on a Python or NoDino virtual environment so it can consume huge amounts of data just like a computer today can.

12:08Today, these are short-lived environments used by data analysts and Julius, but over time, they'll become a new type of Dropbox where your data is persisted long-term for additional processing or cross-file inference and insights. Agent task and flow planning. Planning can't function without intent. Understanding intent has always been a holy grail. And LLMs finally helped us unlock what we spent years approximating at BIB with NLP tricks. Once intent is accurate, planning can start. Creating an agent planner is incredibly nuanced and will take significant integration with user preferences, third-party data sets, knowledge of compute capabilities, etc.

12:39An app store of experts. Apple initially made the mistake of building a closed app store. Then it realized it could monetize a cornucopia of creativity if they opened it up. Regardless of OpenAI saying they're focused on ChatGPT and only ChatGPT, it's inevitable they'll re-scope it and enable a long tail of specialized agents. Builders will be able to compose multiple tools together into workflows that can specialize. And AIs over time will be able to autocomplete these tools together as well, learning from the builders that came before them. Persistent contextual memory. Embeddings are helpful, but they are missing fundamental parts like context switching, conversational centroids, summarization, enrichment, etc.

13:12Most of the cost of LLMs today comes from prompts, but as history and persistence is embedded and the inference cached, this will unlock the ability to have long-term memory with pointers to critical subjects, topics, feelings, and tone. Core memory is just the beginning. We still need all the rich information our minds conjure when we think about a past sunset, a breakup, a scientific understanding, or sensitive context for people we interact with. Now guys, I know that this is long, but I want to continue because I think it's hugely, hugely relevant. Rob goes on, long polling tasks. Agent is a loaded word, but part of the intent is to have tasks that can be scheduled and self-completing regardless of the time horizon required.

13:47E.g., let me know when flights from Montreal to Hawaii are less than$500. This will require coordination of compute across API providers as well as virtual environments in the cloud. Dynamic UI. Chat is not the final end-all interface. There's a reason apps have affordances like buttons, date pickers, images. It simplifies, clarifies. AI will be a co-pilot, but to be a co-pilot, it'll need to adjust to what works best for a given user. The future is personalized as optimizations require it, so UI will be dynamic. API and tool composition. Expect AIs to generate custom quote-unquote apps in the future, where we can build our own workflows to compose together APIs without waiting for a big startup to do so.

14:25Fewer apps and startups will be needed to generate frontends, and AI will be better at composing an array of tools and APIs together coupled with a gas fee or tax. Assistant-to-assistant interactions. There will be countless assistants in the future, with each assisting humans and other assistants towards some greater intent. Alongside this, assistants will need to learn to interface across text, APIs, file systems, and other modalities used by both agents-slash-startups and humans as integration flows deeper into our world. Plugins-slash-tool stores. Specialized assistants can only be made possible by composing tools, APIs, prompts, data preferences, and much more.

14:58The current plugin store is super early days, so expect much more work to come, and expect many of those plugins to be rolled in-house as they become more mission-critical. And this is just a 10-minute brain dump. Much, much more is needed behind the scenes including internet search and scraping, community for intent, building, RLHF, etc., dynamic API generators and connectors, gas fees, tool builders, ingestion via glasses, earbuds, etc. If you think it's too late to be an AI, Just know the above is about 25 % of what it will actually take, with much more to come as we iterate and get even more creative.

15:30The point of all of this, this very long and super interesting perspective on where things are going, is that what you see with ChatGPT is the very first iteration, now maybe the second or third iteration depending on how you look at it, of a clean break with a former modality of computing. Thinking about it simply as an application that does some set of things completely misses how comprehensively it's challenging the way that we interact with computers, and indeed the way that we interact with the world and get anything done in general. Every time OpenAI or ChatGPT adds another feature, it has this feeling of having meant to have always been there.

16:06Of course, browsing the internet was supposed to be wired with image input was supposed to be wired with image output. How could it ever have not been so? And so to the extent that one is building a startup in this space, the old Wayne Gretzky notion of skating to where the puck is going has never been more significant. Those who design for the static world, in which some feature or another is missing from what we see now, are too likely to be steamrolled by the inevitable spur of progress. I, for one, am excited to see what OpenAI announces at their dev day, and I'm sure we'll be talking about it here.

16:36For now, I appreciate you listening or watching as always, and until next time, peace. Thank you.