AI is the First UI Paradigm Shift in 60 Years

0:00Today on the AI Breakdown, we're talking about how AI represents the first UI paradigm change in 60 years. Before that on The Brief, Chuck Schumer lays out a new policy framework, Marvel courts controversy, and much, much more. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Like, subscribe, and share, and go to breakdown.network for more information. Welcome back to the AI Breakdown Brief, all the AI headline news you need in five minutes or less. We kick off today with what is gotta be just the beginning of controversies that relate to AI being used in popular media.

0:38Gizmodo writes, Secret Invasion uses AI for its opening credits, and the excuse is meaningless. The new Marvel Disney Plus show crosses a messy line for AI-generated work's presence in mainstream media. Gizmodo writes, If you hopped on Disney Plus this morning to see what Nick Fury and his frenemies were up to in Marvel's Secret Invasion, you may have noticed something a little off about the series opening title sequence. It looks strange to your mind's eye and for good reason. It's one of the first major shows to use AI-generated art in such a way. The story goes on to explain that the titles were developed by Method, who had also worked on previous Marvel shows including Moon Knight, Ms.

1:12Marvel, and Loki, and that with this particular task, they were in fact approached to use AI-generated imagery. This wasn't just something they turned in, it was a plan all along. Gizmodo writes, That hazy surreality was the intention behind using AI-generated imagery. Producer Ali Salim said,

1:35Salim went on saying that he didn't really understand how AI worked, but, as Polygon put it, was fascinated with the ways in which the AI could translate the sense of foreboding he wanted for the series. Salim said, quote, We would talk to them about ideas and themes in words, and then the computer would go off and do something. And then we could change it a little bit by using words, and it would change. Gizmodo's complaint seems to be about AI's use in major media in general. They write,

2:32On Twitter, people's reaction to this was absolutely a Rorschach test for how they feel about AI in general. Brian Long writes, So Marvel really used AI to make their intro for Secret Invasion. It's actually over. Stephen Ford writes, Marvel and Disney have infinite money yet used AI for the Secret Invasion opening credits. A slap in the face to literally every artist Disney has ever worked with, and something that overshadows the hard work everyone did on this show. Kelly McKernan writes, Marvel used AI for the Secret Invasion intro. That was a paying job for many artists. Add this to the heap of industry-wide lost work replaced by plagiarism software, aka generative AI.

3:08Absolutely disgusting. On the other hand, some digital artists were excited. Omliart writes, Whoa, so exciting to see AI art being utilized in all its imperfect beauty on Marvel's Secret Invasion. Kudos to Method Studios. It's gorgeous. Chris Castanova writes, Wow, an official movie sequence made with AI. And MK Toon writes, Everybody take a big breath. Marvel didn't feed the Secret Invasion opening into a computer and spit it out on Disney+. Thus, Method Studios, a VFX company that staffs human producers, human designers, and yes, human AI technicians worked on it. They used AI as a tool. So friends, this is absolutely the beginning of a type of controversy that we're going to see a lot of.

3:46And it'll be interesting to see in a couple weeks whether anyone cares anymore or whether we've just moved on to something else. Next up, a very different type of story. We're starting to get more information about how the Senate plans to regulate AI. Senate Majority Leader Chuck Schumer has outlined a new framework, if not actual new policies. Schumer has named the framework the Safe Innovation Framework for Artificial Intelligence. In a speech, Schumer said, After months of talk with over 100 AI developers, executives, scientists, researchers, workforce experts, and advocates, this morning I'd like to share my proposed framework for action.

4:17I call it the Safe Innovation Framework for AI Policy. The Safe Innovation Framework. I call it that because innovation must be our North Star. The U.S. has always been a leader in innovating on the greatest technologies that shape the modern world. But if people think AI innovation is not done safely, if there are not adequate guardrails in place, it will stifle or even halt innovation altogether. So it is safe innovation that we must seek. So, of course, this being Congress, that safe word has to stand for something. And in this case, it stands for security, accountability, protecting our foundations, and lastly, explainability.

4:48Now, while this is very light on actual policies, I tend to think that's a good thing. It suggests to me that they're taking this seriously and are trying to avoid any sort of knee-jerk reaction. There's clearly growing technical awareness. For example, here's how Schumer explains explainability. He says, Explainability is perhaps the greatest challenge we face on AI. Even the experts don't always know why these algorithms produce the answers they do. It's a black box. No everyday user of AI will understand the complicated and ever-evolving algorithms that determine what AI systems produce in response to a question or task.

5:20And of course, those algorithms represent the highest level of intellectual property for AI developers. Forcing companies to reveal their IP would be harmful, it would stifle innovation, and it would empower our adversaries to use them for ill. Fortunately, the average person does not need to know the inner workings of these algorithms, but we do need to require companies to develop a system where, in simple and understandable terms, users understand why the system produced a particular answer and where that answer came from. This is very complicated work. Now, if you're asking what the next steps are, Schumer is convening a series of what he calls AI Insight Forums.

5:50Quote, we need the best of the best sitting at the table. The top AI developers, executives, scientists, advocates, community leaders, workers, national security experts, all together in one room, doing years of work in a matter of months. The panels will include people of differing views, including some skeptics. Each insight forum will focus on the biggest issues in AI, including asking the right questions, AI innovation, copyright and IP, use cases in risk management, workforce, national security, guarding against doomsday scenarios, AI's role in our social world, transparency, explainability and alignment, and privacy and liability.

6:20Schumer says these insight forums are the first of their kind. They have to be the first of their kind because AI moves so quickly. If we take a typical path, holding congressional hearings with opening statements and each member asking questions five minutes at a time on different issues, we simply won't be able to come up with the right policies. By the time we act, AI will have evolved into something new. This is very encouraging to hear as someone who has watched a lot of congressional hearings over the last few years. Congressional hearings as they are currently constituted tend to just be chances for legislators to get in soundbites that they can use in future campaigns.

6:50So even if this new type of forum doesn't exactly work as planned, at least it's not that. Schumer concludes,

7:13to tackle every issue. Humility is the key word. The AI revolution is going to happen with or without us. I do not know of any other instance in human history when we reach new heights, uncover new truths, or mastered new innovations only for us to turn back. It's in our nature to press ahead. So friends, let us not turn back. Let us not look away. Instead, let us forge ahead determined and unafraid to lay a foundation for the next era of human advancement. Obviously, we will continue to follow up on the US policy discussion around AI, but I will only say this. I don't know that I've ever seen another issue in my lifetime that has gone from not on the agenda to the very top of the agenda so quickly.

7:52And that I think is actually cause for celebration. It suggests to me that politicians actually understand the magnitude of what we're dealing with. Now we're going a little long on the brief today, but there is so much happening. And these were both stories that could have been features, but let's still try to catch up on a few of the other stories as well. You might remember a month or two ago when Dropbox announced a set of layoffs. The layoffs weren't people being replaced by AI, but they were people whose skill set wasn't a good fit with Dropbox as it reoriented itself to compete in an AI-powered world.

8:22Well, now, just a little bit later, we've started to see how Dropbox is thinking about its approach to AI. Yesterday, they announced a number of new initiatives, including a$50 million AI venture fund, a universal search bar which can search across third-party tools, and something they're calling Dropbox AI, which summarizes and extracts info from files stored in Dropbox. Meanwhile, one company that did cut a big part of its workforce for AI is the Build tabloid in Germany. Build is the biggest newspaper in Germany and has announced that it will cut about 20 % of its workforce for AI. In a memo obtained by CNN, the company said that traditional roles like editors and proofreaders will, quote, no longer exist as they do today.

9:00Next up, global cybersecurity company Group IB says that they've identified over 100 ,000 sets of ChatGPT credentials within the logs of info-stealing malware that trades on dark web marketplaces. The number of compromised accounts reached a peak in May of 26 ,802. So obviously, if you are using ChatGPT like any other service, please enable two-factor authentication and keep yourself safe. Moving on, Wimbledon has raised some higher, announcing that they will be introducing AI-powered commentary to coverage this year. AI-generated commentary and captions will be used for all of its online video highlights.

9:33Now, while the amount of AI content is limited, IBM said that the move is just one step along the way to generating AI commentary on full matches. Finally, another milestone in the ChatGPT plugin store. As of yesterday, there are now 500 available plugins. Now, of course, how many of them are useful is a subject for another video. That's going to do it for today's AI Breakdown Brief. If you're enjoying this, please like, subscribe, and share, and I'll be back soon with the main AI breakdown. Welcome back to the AI Breakdown. As I mentioned in the intro to the show, I am currently traveling. I have a few interviews coming up.

10:07There'll be an interview on Friday, an interview on Saturday, and then an interview again on Monday. But the plan for Tuesday, Wednesday, and Thursday is to do the brief, but potentially not the main part of the episode. We'll see how the timing works, but certainly I will get you guys a brief at least. Now today, as this show comes out, I am racing to catch a plane inevitably. And I came across an article from June 18th, just a couple days ago, that I thought was really interesting. It was written by Jacob Nielsen, who's a principal at the Nielsen Norman Group, which focuses on research-based user experience.

10:38The piece is called AI, the first new UI paradigm in 60 years. And Jacob summarizes, AI is introducing the third user interface paradigm in computer history, shifting to a new interaction mechanism where users tell the computer what they want, not how to do it, thus reversing the locus of control. On the one hand, this is the type of thing that's sort of wonky and behind the scenes. But on the other hand, this is a recognition of a big pattern in history that has shifted. And I think trying to understand that paradigm shift is really valuable. So today as I travel, you guys get a long Reads Thursday about AI, the first new UI paradigm in 60 years.

11:16ChatGPT and other AI systems are shaping up to launch the third user interface paradigm in the history of computing, the first new interaction model in more than 60 years. Section, the first two paradigms. Paradigm 1, batch processing. From the birth of computers around 1945, the first UI paradigm was batch processing. In this paradigm, users specified a complete workflow of everything they wanted the computer to do. This batch of instructions was submitted to a data center, often as a deck of punched cards, and was processed at some unspecified time, often overnight. Later, often the next morning, users would pick up the output of their batch.

11:54Usually, this would be a thick fan fold of printouts, but it could also be a new deck of punched cards. If the original batch contained even the slightest error, there would be no output, or the result would be meaningless. From a UI perspective, batch processing did not involve any back and forth between the user and the computer. The UI was a single point of contact, that batch of punched cards. Usability was horrible, and it was common to need multiple days to fine-tune the batch to the point where executing it would produce the desired end result. Paradigm 2, Command-Based Interaction Design Around 1964, the advent of time-sharing, where multiple users shared a single mainframe computer through connected terminals, led to the second UI paradigm, Command-Based Interaction.

12:34In this paradigm, the user and the computer would take turns, one command at a time. This paradigm is so powerful that it has dominated computing ever since, for more than 60 years. Command-based interactions have been the underlying approach throughout three generations of user interface technology. Command lines like DOS and UNIX, full-screen text-based terminals common with IBM mainframes, and graphical user interfaces, Macintosh, Windows, and all current smartphone platforms. Powerful and long-lasting indeed. The benefit of command-based interactions compared to batch processing is clear. After each command has been executed, the user can reassess the situation and modify future commands to progress toward the desired goal.

13:14In fact, users don't even need to have a fully specified goal in mind, because they can adjust their approach to the problem at hand as they get more information from the computer and see the results of their initial commands. Early command line systems often didn't show the current state of the system, with horrible usability as a result. For example, in Unix, no news was good news because you would get feedback from the computer only if your command resulted in an error message. No errors meant no information from the computer about the new state, which made it harder for users to compose the following command.

13:42The beauty of graphical user interfaces is that they do show the status after each command, at least when designed well. The graphical user interface has dominated the UX world since the launch of the Macintosh in 1984, about 40 years of supremacy until it possibly is replaced by the next generation of UI technology and, more importantly, the next UI paradigm in the form of artificial intelligence. Section. The newest paradigm. paradigm. Paradigm three, intent-based outcome specification. I doubt that the current set of generative AI tools like ChatGPT, BART, etc. are representative of the UIs we'll be using in a few years because they have deep-rooted usability problems.

14:18Their problems led to the development of a new role, the prompt engineer. Prompt engineers exist to tickle ChatGPT in the right spot so it coughs up the right results. This new role reminds me of how we used to need specially trained query specialists to search through extensive databases of medical research or legal cases. Then Google came along and anybody could search. The same level of usability leapfrogging is needed with these new tools. Better usability of AI should be a significant competitive advantage. And if you're considering becoming a prompt engineer, don't count on a long-lasting career.

14:46The current chat-based interaction style also suffers from requiring users to write out their problems as prose text. Based on recent literacy research, I deem it likely that half the population in rich countries is not articulate enough to get good results from one of the current AI bots. That said, the AI user interface represents a different paradigm of the interaction between humans and computers, a paradigm that holds much promise. As I mentioned, in command-based interactions, the user issues commands to a computer one at a time, gradually producing the desired result if the design has sufficient usability to allow people to understand what commands to issue at each step.

15:20The computer is fully obedient and does exactly what it's told. The downside is that low usability often causes users to issue commands that do something different from what the users really wanted. With the new AI systems, the user no longer tells the computer what to do. Rather, the user tells the computer what outcome they want. Thus, the third UI paradigm, represented by current generative AI, is intent-based outcome specification. A single example of a prompt for an AI system is, make me a drawing suitable for the cover of a pulp science fiction magazine showing a cowboy in a spacesuit on an airless planet with two red moons in heaven.

15:53Try ordering Photoshop circa 2021 to do that. Back then, you would have issued hundreds of commands to bring forth the illustration gradually. Today, Bing Image Creator made me four suggested images in a few seconds. With this new UI paradigm, represented by current generative AI, the user tells the computer the desired result, but does not specify how this outcome should be accomplished. Compared to traditional command-based interaction, this paradigm completely reverses the locus of control. I doubt we should even describe this user experience as an interaction, because there is no turn-taking or gradual progress.

16:23That said, in my science fiction illustration example, I'm not happy with the spacesuits. This might be fixed by another round with the AI. Such rounds of gradual refinement are a form of interaction that is currently poorly supported, providing rich opportunities for usability improvements for those AI vendors who bother doing user research to discover better ways for average humans to control their systems. Do what I mean, not what I say is a seductive UI paradigm. As mentioned, users often order the computer to do the wrong thing. On the other hand, assigning the locus of control entirely to the computer does have downsides, especially with current AI, which is prone to include erroneous information and its results.

16:57When users don't know how something was done, it can be harder for them to identify or correct the problem. Intent-based paradigm doesn't rise to the level of non-command systems, which I introduced in 1993. A true non-command system doesn't require the user to specify intent because the user acts as a side effect of the user's normal actions. As an example, consider unlocking a car by pulling on the door handle. This is a non-command unlock because the user would perform the same action whether the car is locked or unlocked. In contrast, a car operated by voice recognition, could unlock the door because the user stated, I want the car to be unlocked, which would be an intent-based outcome specification.

17:31And an old-fashioned car could be operated by the explicit command to unlock the door by inserting and twisting the key. Whether AI systems can achieve high usability within the intent-based outcome specification paradigm is unclear. I doubt it because I am an enthusiastic fan of graphical user interfaces. Visual information is often easier to understand and faster to interact with than text. Could you fill out a long form like a bank account application or a hotel reservation by conversing with a chatbot, even one as smart as the new generative AI tools, clicking or tapping things on a screen is an intuitive and essential aspect of our user interaction that should not be overlooked.

18:04Thus, the second UI paradigm will survive, albeit in a less dominant role. Future AI systems will likely have a hybrid user interface that combines elements of both intent-based and command-based interfaces while still retaining many GUI elements. Again, back to NLW for just a quick wrap-up here. I really do love the simplicity of the explanation of moving from a paradigm in which we tell computers what to do to one in which we tell computers the result we want on the other side and let them figure out how to do it. To put it in the context of a conversation which many people have had around AI in the last couple months, part of the excitement around autonomous AI agents, things like AutoGBT or Baby AGI, is the idea that we can give an AI an objective and let it figure out, one, the steps to do it, and then two, how to execute along each of those steps.

18:50I think that's a demonstrative example of this intent-based outcome specification that Jacob is talking about. I also tend to agree on first blush that whereas command-based interaction entirely replaced batch processing, there will remain use cases where that command-based interaction still makes sense, where GUIs still make sense. I wouldn't be surprised, however, if many of the things that I assume I would still want to use a GUI for do in future generations get replaced by this new paradigm. But if nothing else, GUIs will still have us boomers to extend their lives for a few more decades. Anyways, guys, that is it for today's AI Breakdown.

19:25Thanks to Jacob Nielsen for writing a really thought-provoking and interesting piece. Thanks to you guys for listening. And until next time, peace.

19:40Thank you.