5 Reasons DALL-E 3 in ChatGPT is a Huge Deal

0:00Today on the AI Breakdown, we're talking all about OpenAI's huge announcement of DALI3 natively in chat GBT. Before that other brief, Amazon announces a big Alexa upgrade powered by AI. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube channel, our Discord, and our newsletter. Welcome back to the AI Breakdown Brief. All the AI headline news you need in around five minutes. Well, friends, we are officially in the fall. We are officially in the season of product announcements. In fact, just before I was recording this, I started to see things coming out of the Microsoft event in New York.

0:44Apparently, a wider co-pilot assistant is coming to Windows. Obviously, we will cover that in full on tomorrow's episode. But for now, we have to catch up with what happened over the last 24 hours. And we kick off with an announcement around Amazon and its Alexa assistant. The TLDR is that after many years of being a just medium or low-powered assistant, Alexa is now getting a generative AI makeover and will be powered by its own Alexa large language model. Now, this is something that Amazon has talked about before and was sort of one of the most obvious plays for them, but that does not change how significant a shift it might be.

1:16At their fall hardware event on Wednesday, Amazon started to premiere the new Alexa. Dave Limp, Amazon's SVP of Devices and Services, said, Alexa LLM is a true generalizable large language model that's very optimized for the Alexa use case. It's not what you find with a BARD or ChatGPT or any of those things. Now, in terms of what this will change, the biggest piece of it is that it will be much more conversational. In the same way tools like ChatGPT allow you to interact with computers through a natural language interface, the new Alexa powered by Alexa LLM will also give people the ability to speak more normally rather than have to adopt some specific nomenclature.

1:50Effectively, you won't have to do the Alexa version of prompt engineering anymore. Limp said that they funneled more than 200 smart home APIs into their LLM, which will give the new Alexa much greater capacity to actually interact with your smart home in meaningful ways. An example, The Verge writes, This contextual understanding will extend beyond knowing what other connected devices you might want to control to things like inferring when something's changed in your home. Explains Limp, if you add a new device to your home, you can say, Alexa, turn on the new light, and it will know what the new light is.

2:19It will disambiguate things, so if you put in a new smart plug or light, it will be easier to control. Another updated that Alexa will be able to respond to multiple requests at once. So for example, turn on the sprinklers and open my garage door and turn off the outside lights. That's something that Alexa can now figure out. That capacity extends to using natural language to create a routine without having to manually program the Alexa app. The example that Limp uses is a morning routine for his kid, saying, Alexa, every morning at 8am, turn up the light, play wake-up music for my kid in his bedroom, and start the coffee maker.

2:48Now, a few things about the implementation. One, obviously there is more room for this to go wrong than the limited, more controllable capacities of Alexa in the past. Obviously, with any LLM, there is a chance of hallucinations, and that could be particularly problematic in this type of context. What that means practically is that the new Alexa will be rolled out only in the US at first, and only through a preview mode, which is effectively a large public beta. The second relevant note is that this souped-up version of Alexa may not be free in the future. Limp said, quote, The idea of a superhuman assistant that can supercharge your smart home and more work complex tasks on your behalf could provide enough utility that we will end up charging something for it down the road.

3:24To me, that sounds like they are definitely going to charge something for it down the road, and they are just preparing people to get used to that reality now. Anyways, like I said at the top, this is in some ways one of the more predictable updates that we were going to get from generative AI and the rise of LLMs. The other one that people are waiting for being a Siri upgrade, but that doesn't make it any less significant for the people who will use it. Next up, we are getting a number of different announcements in and around the Microsoft ecosystem. They're having a big event in New York. Like I said, there is information and news pouring out of that right now.

3:54But what got announced yesterday was from one of Microsoft's subsidiaries, GitHub. That announcement was that GitHub's Copilot Chat, which is the chatbot assistant that lives in GitHub's developer environment, has now moved out of enterprise and business usage and is available for individuals who have GitHub Copilot subscriptions. Again, in many ways, this is an expected product update. We are, as we will discuss even more tomorrow, heading to a world in which there is going to be an AI-powered chatbot in every application you use, until such time as it either totally revolutionizes the interface of how we interact with computers, or it's determined that people actually don't like that interface.

4:26but for now, you can expect this type of experience to come to basically everything. Even if it doesn't work everywhere else, it seems like it's likely to be a pretty big hit when it comes to coding, and given how significant the AI assistant for coding use case is, I'm sure that this is an announcement that many will be pretty excited about. Now, it is not just individuals and businesses who are figuring out how to use artificial intelligence. The Pennsylvania state government has announced that it will prepare to start using AI across its operations as well. Speaking at Carnegie Mellon in Pittsburgh, Governor Josh Shapiro said Wednesday that his administration will be convening an AI governing board, publishing principles on the use of AI, and developing training programs for state employees.

5:02His argument is basically that this is going to be something significant for Pennsylvanians, and so the government needs to be able to understand it and adapt to it so that they can make sure that they can help their citizens do the same. As Shapiro put it, we don't want to let AI happen to us. We want to be a part of helping develop AI for the betterment of our citizens. Now, this is definitely part of a trend of increased attention being paid to AI at the state government level. Of course, earlier this month, California Governor Gavin Newsom signed an executive order to study the development of AI.

5:29And beyond that, lawmakers in at least 25 different states have introduced bills that address some aspect of artificial intelligence. All of which certainly lends credence to recent comments from Ray Dalio that AI will greatly disrupt our lives within a year. Speaking at Fast Company's Innovation Festival this week, the Bridgewater founder-turned-public-thought-leader for boomers, Ray Dalio, said that AI will soon be a quote great disruptor for all of our lives. All these changes, he said, are going to happen in the next five years. And when I say that, I don't mean five years from now. I mean that you're going to see changes next year, the next year, even bigger changes.

6:01It's all going to change very fast. Now, of course, while Dalio isn't an AI expert or anything like that, he is a public intellectual of significant repute, especially for that boomer generation that I just mentioned. And so I think there is significance that he is also talking about these issues as well. Speaking of talking about these issues, AI continues to get high billing in Congress and the Senate. Earlier this week, it was the Senate Intelligence Committee's turn to hold a hearing relating to AI. Invited witnesses included Dr. Benjamin Jensen, Senior Fellow at CSIS and Professor at the Marine Corps University School of Advanced Warfighting, Dr.

6:33Jeffrey Ding, Assistant Professor of Political Science at George Washington University, and Dr. Jan LeCun, who is a Turing Award winner and the Chief AI Scientist at Meta. Now, one of the things that makes Jan different than some of his peers, particularly his other 2018 Turing Award-winning fellows, Jeffrey Hinton and Yoshua Bengio, is that to the extent that they are scared about the AI safety future, Jan has very strong disagreements with their points of view. Much of that came out in Jan's opening statement, where he took extensive time to talk about how important open source really is. In that statement, he said, At Meta, we believe it is better if AI is developed openly rather than behind closed doors by a handful of companies.

7:09Generally speaking, companies should collaborate across industry, academia, government, and civil society to help ensure that such technologies are developed responsibly and with openness to minimize the potential risks and maximize the potential benefits. Jan also argues that open sourcing is a key geostrategic advantage for the United States. In that opening statement again, he said, By open sourcing current AI tools, we can develop and improve the foundational models faster than others, including potential adversaries we'll be able to access and build on those models. With U.S. leadership, we can cultivate this powerful technology based on our values, rather than relinquishing it to our adversaries.

7:43Leading the AI research and development efforts puts us in a strong position to enhance the safety of our systems and to warn about potential risks. Now, at the same time this was going on, people were discussing recent comments from Mustafa Suleiman, the CEO of Inflection AI and the former co-founder of Google DeepMind, about why open source could be problematic. These sides are quickly hardening in one of the most important conversations in artificial intelligence. Andrew Kritsch retweeted Jan's opening statement and said, If we dismiss his points here, we risk building some kind of authoritarian AI industrial complex in the name of safety.

8:14Extinction from AI is a real potentiality, but so is the permanent loss of democracy. Both our bad and all sides of this discussion need to acknowledge each other to expand our set of options for striking a balance. If AI is developed, used, and regulated responsibly, it can greatly expand the Pareto frontier of possibilities for humanity's future. And only through civil discourse like this can we achieve that. Anyways, that is, of course, a much longer conversation than we can have at the brief, but that is going to wrap us for today. Lots and lots going on, so stay tuned throughout the week for more.

8:44Next up, the main AI breakdown. Hey guys, one more quick note before we dive in. I so appreciate everyone who has taken the time to fill in our educational content survey. The TLDR, for those of you who haven't, is that we are considering launching a number of different types of AI educational content to better help you retrain, reskill, get to where you want to be when it comes to AI, including some really different ideas such as an AI learning community that would exist on Discord or some other shared space. But I really need feedback to know what people actually need and what people most want, and if you are willing to take less than one minute to give me that feedback, I would be ever grateful.

9:21Just go to bit.ly slash AI breakdown survey, and like I said, it'll take you less than a minute to fill that out. Thanks in advance, and let's get to today's show. Welcome back to the AI breakdown. Today we are talking all about the announcement from OpenAI yesterday that DALI 3 was on its way and it was going to be integrated directly into ChatGPT. On this episode, I'm going to go through five reasons why I think this is such a big deal. And while I am designing this for podcast consumption, I also would highly suggest that if you don't yet, you go subscribe to the YouTube channel as well, as this is obviously inherently a pretty visual story.

9:56By way of previewing the conversation, here are the five reasons I think it's such a big deal. The first is the advance in quality in one of the most important AI areas of image generation. The second is its integration with ChatGPT, which has both a user experience as well as a mainstreaming dimension. The third are the implications for multimodality as the next iteration of generative AI. Fourth is its impact on competitors like MidJourney. And five is what I'm calling competitive accelerationism, or the increase in speed of development and release based on market competition. How that's good and how it also might be not so good.

10:29Let's look at the announcement first. Yesterday, OpenAI tweeted, Our new text-to-image model, Dolly3, can translate nuanced requests into extremely detailed and accurate images. Coming soon to ChatGPT Plus and Enterprise, which can help you craft amazing prompts to bring your ideas to life. Believe it or not, there is tons in there, and many of those themes are announced in their basic product landing page. Dolly3, OpenAI writes, Understand significantly more nuance and detail than our previous systems, allowing you to easily translate your ideas into exceptionally accurate images. The first reason that DALI 3 is significant is the simple fact that in many ways it seems to be an advance on the current state of the art, which is Midjourney 5.2, which came out earlier this year in June.

11:09I'm going to get into a bit more of a comparison between the two in just a moment, but one of the things that DALI 3 seems to do really well, at least on first glance, and at least with the demo images that have been shared, is handling nuance and detail really well. It feels more precise, and because it's more precise, it ends up being more expressive. When a model is better able to understand, for example, inspirations or criteria or styles such as pointillism or cardboard cutouts or whatever it is, the end result is a lot closer to what the prompter imagines. Now, the other piece of this, again, compared to MidJourney specifically, is the fact that DALI 3 apparently handles text much better.

11:42One of the examples they give is a poster that says Explore Venus, which is something that you just can't do right now on MidJourney, although you can do it with some other models, including those from Stability AI. Anyway, the TLDR for this argument is that if this was the only advance, just a more precise, higher-powered image generation model that would still be significant. However, that's not the only thing about this, and in fact, in some ways, it's overshadowed by where this exists and how people are going to interact with it. DALI 3 is built natively into ChatGPT, and that means a number of different things.

12:12First of all, it means that it's more natively suited to natural language prompting than the sort of prompt engineering that people have to do with tools like MidJourney. Indeed, OpenAI says that part of what makes DALI 3 so much more expressive and so much better able to handle nuance is the fact that it is natively built on GPT. On the product page, this is one of the biggest highlights that they make. OpenAI writes, Modern text-to-image systems have a tendency to ignore words or descriptions, forcing users to learn prompt engineering. DALI 3 represents the leap forward in our ability to generate images that exactly adhere to the text you provide.

12:44The example they gave is a nighttime illustration of what might be mid-20th century London, in which they point out how a number of different parts of the natural language prompt come to life. A sidewalk bustling with pedestrians enjoying the nightlife A bustling street under the shine of a full moon At the corner stall, a young woman with fiery red hair dressed in a signature velvet cloak is haggling with the grumpy old vendor The grumpy vendor, a talented man, is wearing a sharp suit, sports a noteworthy mustache, and is animatedly conversing on a steampunk telephone Now one thing it should be noted is that right now, all we have in terms of understanding the quality of Dali 3 are the highly curated and selected examples that OpenAI has given us So far as I have seen, the company has promised that this will be rolling out into ChatGPT Plus and Enterprise in the near future, but so far I haven't seen anybody that actually has access to it.

13:28Which means, of course, that we have to take any big claims with a grain of salt until we contest it ourselves. Still, if Dolly3 really can handle this level of linguistic nuance when it comes to prompting, it is a total game-changer relative to what you get from something like MidJourney. The other piece of that is how ChatGPT might actually be a useful assistant in refining prompts. Again, OpenAI writes, Dolly3 is built natively on ChatGPT, which lets you use ChatGPT as a brainstorming partner and refiner of prompts. Just ask ChatGPT what you want to see in anything from a simple sentence to a detailed paragraph.

13:58When prompted with an idea, ChatGPT will automatically generate tailored detailed prompts for DALI 3 that bring your idea to life. If you like a particular image but it's not quite right, you can ask ChatGPT to make tweaks with just a few words. Now the other important piece of the argument about why it's so transformative to be integrated natively with ChatGPT is of course just distribution. Right now the biggest player in this text-to-image space is mid-journey, and to use mid-journey you basically have to live inside Discord. That is a huge barrier for normal consumers, and the fact that there are already hundreds of millions of users of ChatGPT that now might just natively have access to Dali, although yes of course it's just for Plus and Enterprise users at first, is obviously a really big deal as well.

14:36Now obviously the other piece of the integration with ChatGPT, which I've decided to break out as a third reason that this is a really big deal, is that this is a clear first step towards multimodality. Quantum physicist Kevin Fisher wrote, DALI 3 is a massive step change improvement to put DALI in chat. A year ago, pre-ChatGPT, I built an alpha that combined chat with images. A small group of people felt this magic together and still reminisce. This is a 100 % revolutionary release from OpenAI. Kevin quote tweeted himself from October 19th of last year when he wrote, prompt engineering is not the future, the future looks more like this.

15:10In it, he shares an example of this alpha, which has a chat-style interface that interacts with images as well. Dr. Jim Phan from NVIDIA is even more precise in his articulation of what DALI-3 means in terms of multimodality. He says, I think DALI-3 is not just a stance against Mid-Journey, it's actually a sneak peek of the upcoming epic battle of massively multimodal LLMs against DeepMind Gemini. Quote, DALI-3 is built natively on ChatGPT. This is the key phrase. DALI-3's extraordinary language alignment is built on a solid textual GPT foundation. Mid-Journey doesn't really have much reasoning brain, which is why so much prompt hacking is needed.

15:44Brain first, pixel second. That's the way to build a strong multimodal AI. So the third reason that this is such a big deal is that it's a first step in a preview of the multimodal future. Now, one thing that has been lurking behind and the fourth reason it's a big deal is its impact on immediate competitors like MidJourney. Chase Lean tweeted, OpenAI just announced DALI 3 their new AI image generator. Will it replace MidJourney? I think it all comes down to two things. One, how good is ChatGPT at understanding the user and writing prompts for DALI? 2. How good will MidJourney version 6 be? As Chase points out, expert MidJourney users know how to write prompts, but they're concerned about things like image quality, prompt adherence, image resolution, character consistency.

16:22Based on the recent demo, I'd say that Dali and MidJourney version 5.2 have similar quality. But here's the thing, Dali is way better at prompt adherence. Unlike MidJourney, Dali generates almost everything requested inside the prompt. It can also make some decent text. Combined with GPT, which is amazingly good at writing descriptions, this makes Dali truly formidable. A huge win for OpenAI. But MidJourney version 6 is on the horizon and is rumored to have much better image quality, prompt adherence, and an upscaler to increase image resolution. For the past two years, superior image quality has kept MidJourney in front of stable diffusion.

16:53If MidJourney version 6 is able to keep making the best images, it will continue being the AI image generator of choice. I think that that's an accurate assessment. I can only speak to myself and the people that I observe, but I make probably 50 to 100 MidJourney images per day. Some of them are for work. Many of them are just for fun or for creativity or for exploration. I can say beyond a shadow of a doubt that if MidJourney version 6 produces what I perceive to be better results and higher quality images than Dolly 3, it's not going to replace MidJourney, at least not fully, for me. Convenience is not enough even if convenience gives Dolly 3 a big edge when it comes to bringing more people into the image generation space in the first place.

17:29More than that, I think that obviously this announcement has to light a huge fire under Midjourney's butts to advance things more quickly. Nick St. Pierre went viral with a tweet that said simply, Midjourney needs to get out of Discord ASAP. And that's a pretty obvious one. A native app for Midjourney is no longer really an option. It's also going to force them to try to reach parity with things like text generation. And so I think if nothing else, you can expect that the next version of Midjourney is going to be a significant amount better, or they risk losing the race entirely. That gets us, however, to our fifth big point, which is maybe the most significant in some ways, and that's what I'm calling competitive accelerationism.

18:03Professor Ethan Mollick writes, It looks like the Gemini quickening has begun. With the upcoming release of the first model to likely beat GPT-4, at least temporarily, you can see a burst of announcements. GPT-4 multimodal and DALI-3 barred integrations like more. Competition in the space is increasing velocity. Now, of course, to the extent that I just mentioned the positive impacts of that, mid-journey getting better because of that competitive pressure, there are many who are concerned about exactly this type of competitive accelerationism. It was one of the big reasons that Jeffrey Hinton said he left Google, that the market pressure to compete and move faster was making companies act less safely when it came to AI releases.

18:40And physicist and MIT professor Max Tegmark said something very similar today. A Guardian piece came out this morning called AI-focused tech firms locked in race to the bottom, warns MIT professor. Physicist Tegmark says competition is too intense for tech executives to pause development to consider AI risks. Now, the piece itself wasn't specifically about this release of DALI 3, but more about this broader trend that Professor Malik was just noticing. You have every big tech company now firing on all cylinders to get ahead, or in the case of OpenAI to stay ahead, and that works at cross-purposes with slow, deliberate, considered AI safety discussions.

19:14It even works at cross-purposes with preventative mechanisms such as red teaming. Now, all in all, this is a much lower priority right now for the average person who's interacting with this DALI 3 news. Like I said, I think people are more excited about this product announcement than anything I've seen in months, maybe since GPT-4, but it is the background noise. It is a piece of this discussion that can't be ignored. And I think that Professor Malik is right to identify that the phase that we're entering, in which GPT-4 becomes no longer inaccessible, but matched or exceeded, is one which will have dramatic impacts on the shape of the industry because of this competitive accelerationism.

19:47Still, when it comes to DALI-3 specifically, it is very, very hard for me not to be just fully excited, ready to dive in. And so October, when they say this will begin rolling out in chat GPT-plus, can't come soon enough. Until then, I will be watching to see what people inside OpenAI create, and I will report back my favorite findings. If you want to share your text-to-image images, come join us on the AI Breakdown Discord. It's a great place for exactly that sort of conversation. I just dumped in a bunch of custom magic cards I created as a for example. You can find a link to the Discord at bit.ly slash AI Breakdown, and I'll see you there.

20:20Thanks as always for listening or watching. Until next time, peace.

20:27Thank you.