The 10 Most Interesting and Useful ChatGPT Plugins So Far

0:00Today on the AI Breakdown, we are discussing the most interesting, relevant, important, useful chat GPT plugin so far as OpenAI rolls out plugin access to all of their Plus customers. On the brief, we discussed Apple's new text-to-speech features, Elon being mad at OpenAI, and much, much more. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. If you're enjoying it, please like, share, and subscribe. Welcome back to the AI Breakdown, all the AI headline news you need in five minutes or less. First up, Zoom has announced a partnership with Anthropic.

0:36The goal of the partnership is to bring the Claude chatbot across the Zoom productivity platform, which will start with Zoom's contact center product being used to help with customer support. Now, it's not clear yet how Claude might be integrated to other parts of the Zoom platform, including team chat, meetings, phone, whiteboard, or Zoom IQ. But one of the things that was notable about the announcement is that Zoom specifically called out Anthropic's constitutional AI model as one of the reasons for the partnership. They write,

1:12Next up, Elon Musk is not happy with OpenAI. In a recent interview on CNBC, he claimed he's the reason that OpenAI exists, that he was the one who invented the name, that OpenAI wasn't putting enough emphasis on safe AI development, and that separately he had also suffered ruptures in his friendship with Google co-founder Larry Page because Page as well, quote, did not seem concerned about AI safety. Musk said, quote,

1:41Now some might feel that Elon is just angry because he's behind OpenAI and doesn't really have a stake in any of these generative AI companies, but he's far from the only one who's upset about OpenAI's moat after yesterday's hearing. One of the big threads of conversation on Twitter has been around how Sam Altman's support for AI licenses vaguely defined at yesterday's hearing was really just another version of regulatory capture and pulling the ladder up after them. The accusation is that OpenAI was allowed to build all these technologies without any sort of licensing regime, but now wants one just because it's out ahead.

2:11I think there's a lot more to explore here, and I probably will in a separate video later this week, but for now, that has been a big part of the sentiment on AI Twitter. Speaking of Elon and Tesla, yesterday they dropped this video of their optimist bots walking around and learning about the real world, as they put it. The way that they frame Optimus is, quote, a general-purpose bipedal humanoid robot capable of performing tasks that are unsafe, repetitive, or boring. As many people have gotten excited about AI over the last few months, some are wondering if AI-powered robotics is the next big trend.

2:41Along those lines, we also got an update yesterday from a company called Sanctuary, where they introduced Phoenix, which is their, quote, revolutionary humanoid general-purpose robot designed for work. They say Phoenix is the first humanoid to be powered by carbon, a pioneering AI control system and represents a giant leap forward in our mission to create the world's first human-like intelligence in general purpose robots. From sci-fi to real life. Now, one of the big themes of yesterday's hearing was concern around what was real and what was not in a world where ChatGPT could communicate as well as humans.

3:11Along those lines, a Reddit post is making the rounds in which a Texas A &M commerce professor failed their entire class of seniors, blocking them from graduating because he claimed they all use ChatGPT. Basically, he took all of their responses to a question, put them into ChatGPT, and asked if ChatGPT had written them, which it said it did. Now, of course, the ironic thing about this story is that what the professor accused them of doing, i.e. overly relying on AI, is exactly what the professor did, relying on an AI system that wasn't designed to detect other AI writing to tell him whether he was reading AI writing.

3:44Last up today, Apple gave us a sneak peek of some of its upcoming AI-driven accessibility updates, and that includes a text-to-speech module that can actually mimic the user's voice. The feature is called Live Speech, and here's how it's described in Gizmodo. Apple is adding the ability for users to record their speech patterns away to, in the company's words, create a voice that sounds like them for when they are at risk of eventually losing their speech. The Personal Voice app asks users to read a random assortment of text prompts equaling about 15 minutes of audio. The system uses AI to then generate speech that's equivalent to your personal style of speaking.

4:16Now, Apple says this will be fine because it will only be available locally on a user's iPhone, but it's still bringing up a lot of privacy and security questions as voice mimicking technology becomes more commonplace. What do you think? Do you want an AI that sounds like you? Let me know in the comments. That's it for today's AI Breakdown Brief. See you back here soon for the AI Breakdown. chat gpt is rolling out plugin access this week to all of its plus users so today on the ai breakdown we're looking at some of the most interesting and productivity enhancing use cases that we've seen so far welcome back to the ai breakdown if you have been around chat gpt and particularly discussions of chat gpt versus google bard you will have heard the word plugins a lot So plugins are effectively a set of third party, in most cases, applications that allow ChatGPT or Google Bard to do different sets of things.

5:14So, for example, an Instacart plugin would allow you to interact with the Instacart grocery shopping app. A Kayak plugin would allow you to potentially search for flights, compare flights, etc. They are a utility layer on top of an AI chatbot like ChatGPT or Google Bard. Now, ChatGPT has been testing plugins in alpha for a month or so now, but last week as part of Google I.O., Google announced that their Bard plugins would be available to everyone. Following suit, this week ChatGPT started rolling out plugins to all of their plus users. So today what we're going to do is look at some of the most interesting and novel use cases for these plugins and figure out what's really useful, how people are exploring them, whether they live up to the hype, and what they suggest about the future of ChatGPT.

6:02Let's first of all do a quick demo for how to actually get access to these plugins if you are a ChatGPT Plus user. So down here you're going to want to go to your settings and then under the menu over here you'll see beta features. Now, this should theoretically have both the browser feature and the plugins feature. For some reason, some people only have one or the other. As you can see right now, I only have plugins. But anyways, you're going to want to toggle on those features so you can actually go use them. Now, when you come back out to the main interface, if you toggle over to GPT-4, you'll see down below plugins.

6:37You click that and no plugins are enabled for me. So let's go to the plugin store. Now, the plugin store looks like a basic web-based interface version of an app store. And indeed, many people have compared the state of plugins currently to the app store when it had less than 100 apps or something like that. You can select most popular and see that it's things that people use pretty frequently. OpenTable, Instacart, Expedia, Kayak. I think that makes sense. Or you can go to all plugins and just start going through page by page. And it doesn't take long to get a sense of what people are building so far.

7:09There is a lot of utility plugins. For example, VoxScript here enables searching of YouTube transcripts, financial data sources, and Google search results. There's a lot of summarization. World News summarizes news headlines. You can ask for the latest news from various sources around the world. Research assistants like Yabble, your ultimate AI research assistant, create surveys, specify audiences, collect data, and analyze. A lot of emerging financial applications like Publix real-time and historical market data. tons of integrated shopping and of course just some random cool things like playlist ai which says create spotify playlists for any prompt let's do that one as the demo just because it sounds pretty interesting here's the terms they access my spotify account i'm already logged in okay so let's try make me an energetic work playlist with hyper pop and synth wave tracks at least 10 tracks no repeats let's see what it does 100 gex we're on the right track and then you can actually see it goes a step farther than just putting the tracks here by creating the playlist using this plugin so now it's complete and let's see how it did boom there it is some fairly good choices we got night call a classic we got some 100 gex not so bad okay so now you get the idea of how plugins work.

8:29Like I said, they are a utility layer on top of ChatGPT. And we'll come back to why that's such a different use case in a few minutes. But before that, let's talk about some of the plugins that people are using and how they are using them. Now for these, rather than go through all of them on my own ChatGPT, I'm going to use examples from Twitter because I think people have already done the hard work of summing up some of the most interesting use cases. The first that we're going to talk about is Golden. And the reason that I wanted to put it first is that it comes at a problem that I think is going to be relevant for all of these other plugins, which is accuracy of AI information.

9:03Jude Gomilla writes, Without the plugin, ChatGPT knowledge is pulled from its LLM, which on the pre-trained model cuts off at mid-2021. The golden plugin gets data that is current, cited, and directly improvable if missing and incorrect. No more black box for our facts. The plugin uses retrieval enhancement where we fetch the latest data from the golden knowledge graph and infuse into ChatGPT as source knowledge to produce citable facts about entities and natural language conversation. We always want to know where information came from. For example, asking the very basic question of who is the CEO of Twitter to GPT4 results in the outdated answer of Jack Dorsey, but with enhancement results is the correct answer of Elon.

9:40Editor's note, he posted this before Linda Yaccaro was made the CEO of Twitter. Jude sums up,

9:48So like I said, I think that this type of information accuracy and information sourcing is going to be a big part of how AI grows and develops and becomes more trusted in society. So I thought I wanted to call that one out. Next up, let's turn to a super basic day-to-day kind of use case, which comes from Amar Reshi, who's a design manager at Brex. He says, I asked ChatGPT to buy my groceries today using Instacart's plugin, and it worked so well. Stayed within my budget, provided ingredients and recipes for seven meals, accounted for my schedule and diet. Then in this Twitter thread, he shows the conversation, and there are a couple things that elevate this, I believe, from more than just a novelty.

10:25First of all, he's effectively outsourcing thinking about what he wants to eat to ChatGPT, and given that humans basically spend most of their time thinking about what they're going to eat next, that's actually fairly time saving. Second, ChatGPT, with the help of the plugin, is able to go from ideas for meals to the actual recipes and ingredients in short order, again saving more time. Finally, it pushes directly to Instacart so he can press shop on that app. Now here it's worth noting that right now there isn't commerce integrated to ChatGPT so he's still being pushed off-site to Instacart.com to his account to actually make the order.

10:58Next up another time-saving personal assistant day-to-day type use case that shows off another capability of the new plugin system. So this one comes from DataChaz and he writes, another thing I like about ChatGPT's plugin system is its ability to select the most appropriate plugin based on the user's query. In this demo, even though I have multiple plugins enabled, ChatGPT correctly chose to use the Kayak plugin, which is exactly what I wanted. His prompt is, Book a trip for two from Paris to Venice under 300 euros. Show me the best prices from July to August 2023. ChatGPT gets right into it.

11:32And again, what he pointed out is that it actually figured out that although he had multiple prompts that were available, Kayak was the right one to use. So this is not only an example of travel planning, which is a use case for these plugins, but also the fact that the system is sophisticated to be able to help you navigate which plugin to use in any given circumstance. Next up is a use case that it seems like a lot of companies are exploring, which is financial research. In this particular case, the plugin being demoed is Portfolio Pilot, which gives research on public companies. But we also saw just in the quick browse that we did of the plugin store, a number of other financial services plugins like Public as well.

12:06I'm planning on a whole video soon about these financial use cases of ChatGPT and why they might be ones that we should be, if not concerned about, certainly more conscientious of the potential for misinformation around. But what they definitely represent is the idea that plugins are going to enable highly specialized research where they hone in ChatGPT to a particular research use case and give ChatGPT more information to execute against that research effectively. Let's move into the realm of productivity. one of the things that is a major use case for these types of chatbots that we're seeing come up more and more is effectively chatting with PDF.

12:41So here, sure enough, is a plugin called Chat with PDF. Chat analyze question with any PDF directly in the comfort of ChatGPT via plugins access. Here it is helping me with homework. He then gives ChatGPT a URL of a PDF and it starts summarizing it right away. Now there are a number of plugins and standalone tools for exactly this, so it seems like a use case that people are really excited about. Speaking of summarization, about a month ago, YK, who goes by CSDojo on YouTube, writes, I started building a chat GPT plugin that can summarize any YouTube video, answer questions about it, and give specific timestamps when asked.

13:15This again speaks to that productivity enhancement use case where we're taking big volumes of information, ingesting them, and then spitting out summaries that can help people understand key details in shorter order. I think the idea of giving specific timestamps where you can go back and review key information in the video makes this extra useful. Now, when it was shared by YK, this plugin was still in development. It wasn't necessarily in the store. And when I went back and looked in the plugin store, I didn't find it exactly, but I did find something called Video Insights and went to see if it did something similar.

13:46So I asked ChatGPT, can you summarize this video? And it was my video yesterday about seven takeaways from the Senate's AI hearing. And it came back with what was in the description and the timestamps. In fact, it said, unfortunately, I can't provide a detailed summary because the video is longer than 10 minutes. However, based on the description, it seems to focus on the Senate's hearing on AI and the various perspectives and concerns raised during the discussion. So while this might not be exactly what you're looking for with this type of summarization, it certainly shows the possibilities moving forward.

14:12Next up, one of the plugins that people are really excited about is called Perfect Prompt. And what this does is it makes your prompts better. Given how much of being successful with generative AI systems is being able to prompt well, this type of tool that can go from generic to very specific and useful prompt is something that people are really, really interested in. In this example from Min Choi, he writes, Perfect, 20 AI news trending with URL links. Perfect prompt suggests a rephrasing of what are the top 20 news stories related to artificial intelligence that are currently trending and can you provide URL links to each of them?

14:45Now, this also uses WebPilot, which is a plugin that allows ChatGPT to browse. And of course, that internet access is a huge part of what makes these plugins so much more useful than just ChatGPT alone. Now, one plugin that we've talked about before actually isn't a third-party plugin at all, but is one of the two plugins that OpenAI has worked on itself. And that, of course, is Code Interpreter. As Peter Yang puts it really well here, Code Interpreter gives everyone a personal data analyst. So here's Peter's example. He writes, I uploaded an Excel file and without prompting, it identified the source World Happiness Report 2021 columns and definitions.

15:20Now let's give it the incredibly lazy prompt, analyze this data and show me cool charts. Here are some charts that Code Interpreter came up with. Top 10 countries by happiness score, happiness score by region, happiness score via GDP per capita, happiness score via life expectancy. This all took less than a minute to generate. Next, I uploaded two separate data sets to see if it can do an analysis that combines the data. And it works. Here's Code Interpreter mapping happiness against life expectancy by country. And you can examine the code in the chat interface to understand how it produces the charts.

15:50The biggest problem with ChatGPT is that it tends to make stuff up. But Code Interpreter doesn't seem to hallucinate much at all because it's all based on data I actually uploaded. That's incredibly powerful. Now, if you want more on Code Interpreter, go check out my video from a couple weeks ago called Six Ways ChatGPT Code Interpreter is Already Being Used. There are a ton of very cool examples in there. Now we're going to close out with a couple of use cases that are a little bit more experimental and actually both come from Pietro Chirano. Pietro writes, couldn't wait for multimodal GPT anymore, so I built my own with plugins.

16:22Introducing Picasso, a ChatGPT plugin delivering multimodal-like performance. Picasso uses five different models to achieve this. Here are some examples of extracting recipes from photos and more. So basically you have here an input of a photo of a plate of spaghetti and ChatGPT, thanks to Picasso, is extracting information from that picture, giving the recipe, and then flipping the flow and generating two similar photos based on those inputs. We've profiled a number of tools on this channel like MiniGPT4 for going from image to text, so it's very cool to see it built in natively to ChatGPT with this plugin.

16:53Pietro's other fascinating plugin experiment is called DesignerGPT and he writes, my new plugin that lets you create any website directly in chat GPT. It seamlessly integrates with the stable diffusion plugin, and I didn't code that. GPT just knows how to use both. Absolutely unreal. Websites are hosted on a remote server via Replit. So the prompt he uses is generate a website for a beautiful architecture firm. Use any photos you want. And sure enough, what comes out is a website, basic though it is, for an architecture firm. Now, how much this is really useful versus just novelty, especially in the wake of so many AI-generated website tools.

17:27who knows, but it does show off the capacity of this ChatGPT plugin system, and I think that that was Pietro's goal, more as a thought starter about what can be done than necessarily getting to an endpoint that is super useful in and of itself. Now, it's worth remembering as we wrap up that this is all super nascent. There are less than 100 plugins in the plugin store right now, and really people are just beginning to experiment. Ethan Mollick cautions wisely, before the influencer thread starts, plugins and web browsing are both still early stage and not miracles yet. Browsing is especially early.

17:57So be careful about the everything just changed OMG posts, but do play around with the systems. They have potential. And in that spirit, Rachel Woods writes, not a critique, but just an observation. She says, after spending a big chunk of the weekend playing with all the new ChatGPT plugins, I have a few observations that are shaping what seems to be ahead for the Everything app, aka ChatGPT. I've found that plugin usage feels more utility focused. I need to get X done rather than exploration or like some of the meandering conversations I have with ChatGPT itself. This could be due to needing to plan ahead and choose which plugins you'll use for a thread.

18:31Oftentimes, I open up a chat thread not really thinking about where it'll end up. The current ChatGPT plugin UX forces pre-planning. Second, the plugins are fickle, like really fickle. If you're not familiar, the way plugins work is by giving ChatGPT an API and a manifest or a natural language explanation of how to use the plugin. If you thought hallucinations were a major limitation, the variability of plugins working or not feels even bigger. Third, it blurs the lines even more on copyright, data privacy, and ownership. A third party could collect a bunch of my content to power a plugin and doesn't cite their sources when used.

19:02Is that okay? If plugin developers will eventually need some usage data on how their plugin is being requested, say to optimize their manifest file, how much data will they get? The whole prompt or the whole thread? If you've forgotten how early we are in this AI wave, go play with chat GPT plugins. You'll probably quickly remember. I think these are great observations. It does absolutely feel like a utility layer on ChatGPT versus an extension of the sort of magic of meandering through different discussions. Now, for a lot of use cases, that's much better, but it does also create the types of issues that Rachel brings up.

19:32In any case, it is an exciting development for sure. It is making ChatGPT a whole different level of useful for lots and lots of different use cases. So go explore and let me know in the comments what you're using it for, what you found good, what you found didn't work as well. And we'll check back in in a couple weeks to see if some use cases and plugins are racing out into the lead. That's it for this AI Breakdown. If you are enjoying these, please like, subscribe, and share it. Go listen to the podcast version or subscribe to the AI Breakdown newsletter. I appreciate you watching or listening, and until next time, peace.