Mobile LLMs? The Battle to Get AI on Our Phones

0:00Today on the AI Breakdown, we're looking at Stability's latest model and what it says about the future of LLMs on our smartphones. Before that on the brief, how you can get access to Dolly 3 right now. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our YouTube channel, our newsletter, and our Discord. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. We kick off today with very exciting news for those of us who have been just pressing refresh on our chat CPT Plus subscriptions, hoping to see DALI 3 arrive.

0:40Well, now not only us, but basically anyone with a Microsoft account can use DALI 3 directly via Bing. To get access to DALI 3, just go to bing.com slash images slash create. And from there, it will ask you to either sign in or to create a new account. Now, each week, free users get 100 boosts that increase the speed with which images are created, but can also create more images after that at a slower pace. As you've probably seen, one of the things that makes DALI 3 exciting for people who have been using tools like Stable Diffusion and Midjourney is that it seems to handle text a lot better.

1:14It's not perfect, but it's certainly a huge upgrade from what we've had so far, at least among the main services that most people use. Now, of course, given that people are getting access to DALI 3 now, there is a huge amount of discussion on Twitter slash X around how it compares to Midjourney. Dreaming Tulpa, I think, sums up a lot of what I'm seeing, which has to do with DALI-3's main performance differential, being at how good it can actually interpret the natural language prompts that people are trying to achieve. In other words, I haven't seen a lot of people say that DALI-3 is distinctly better than mid-journey when it comes to the quality of imagery, but instead that the natural language prompting allows people to get much closer to what they were actually imagining than they can using mid-journey prompting.

1:58Indeed, in many ways, MidJourney feels like an act of prompt engineering, while Dali 3 promises just full-on natural language inputs and the ability to refine once again with natural language. Tulpa says, I'm super impressed with how Dali interpreted the prompt below. While MidJourney's outputs are beautiful, it's nowhere near what I was looking for. MidJourney recently said they're going to improve upon this, and oh boy, I hope they do. This feels like the first time MJ got a serious competitor. TechHala did something interesting where they tested 10 different prompts, MidJourney versus Dali 3, and rated them on a scale using accuracy, aesthetics, detail, consistency, and believability.

2:33TechHala sums up, While I still believe MidJourney is slightly better overall, Dali 3 is very close. In fact, in terms of believability and accuracy, it's even ahead. Mad Pencil responds and says, MJ did mostly better on aesthetics and details, but Dali stays faithful to the prompts. Now, moving on to a little bit of Apple AI news. In a recent discussion with UK Press, Tim Cook said that that company was not only not planning on layoffs in the country, but that they would be increasing the size of their artificial intelligence team in the United Kingdom. Now, there really weren't more details than that.

3:05Cook just responded to a question around AI saying, we're hiring in that area, yes, and so I do expect investment to increase. This is obviously a boon to a country whose prime minister has said very clearly that he wants them to be a leader in both the regulation and the development of artificial intelligence. Now, we have just come off of September, and in many ways it felt at the beginning like a relatively quiet extension of the summer, only to of course over the last week and a half or so really pop off with the announcement of a huge slate of products. However, looking back, one of the things that was clear was just how much venture capital activity there was over the last month.

3:41Chief AI officer on Twitter sums up the top 10 AI startup funding rounds from last month, including that big$4 billion investment from Amazon into Anthropic, which, for the sake of completeness isn't exactly a$4 billion investment right up front, but up to$4 billion being invested over time. There's also a$500 million investment into Databricks. Now, Databricks also made headlines for a deeper partnership with Microsoft that some saw as Microsoft hedging their bets relative to their relationship with OpenAI. Other big investment rounds include a$223 million investment into Helsing, which is an AI company focused on the defense industry that is backed by Spotify's Daniel Ek,$200 million to Imbue,$125 million to Infabrica,$110 million to Dematrix,$100 million to Ryder,$100 million to Inceptive, $100 million to Prion, and another$85 million to Pyxis, which is AI for marketers.

4:32Nine nine-figure rounds in a single month shows just how much activity there is in this space. Now, of course, the other big thing that happened in September was the culmination of the writer's strike. The discourse around the compromise that they reach with AI has been really fascinating to me, as so far it seems a lot more like a Rorschach test in terms of how people interpret it than a clear win or loss for either AI or for the writers themselves. But whether it represents a win or a loss, another piece of news from this morning shows just how much AI is going to be a part of celebrity and pop culture in general going forward.

5:04Over the weekend, Tom Hanks posted a picture of himself from a video and said, Beware! There's a video out there promoting some dental plan with an AI version of me. I have nothing to do with it. Now, Hanks is interesting because he is very clearly not some visceral AI Luddite, or someone who dismisses the excitement around the technology. On a podcast recently, Hanks said, Anyone can now recreate themselves at any age they are by way of AI or deepfake technology. I could be hit by a bus tomorrow and that's it, but performances can go on and on and on. Outside the understanding of AI and deepfakes, there'll be nothing to tell you that it's not me and me alone.

5:35That's certainly an artistic challenge, but it's also a legal one. He added, without a doubt, people will be able to tell that's an AI, but the question is, will they care? There are some people that won't care that won't make that delineation. Finally today, another story at the intersection of artificial intelligence and health. ScienceAlert.com writes, AI identifies brain signals associated with recovering from depression. The piece writes, it could soon be possible to measure changes in depression levels like we can measure blood pressure or heart rate. So the piece is about a recent study, and effectively what that study was trying to show is that while currently, all we have to go on when it comes to understanding levels of depression is patients self-reporting their mood, that's problematic because so many things can affect one's mood.

6:16A stressful thing that happened in the morning can impact how someone's day was, just as much as any sort of actual underlying depression issues. Given that, they write, scientists in the U.S. used a combination of electrode implants and AI analysis to try to pinpoint changes in brain activity patterns triggered by deep brain stimulation. The result was that the team of researchers, which included people from the Georgia Institute of Technology, the Emory University School of Medicine, and the Icahn School of Medicine at Mount Sinai, did end up identifying a brain signal that could be used as a biomarker linked to recovery from depression.

6:47So far, it seems to be more than 90 % accurate in its feedback. Now, one of the things that makes the study most exciting is that each of these types of studies builds on itself, given that there's now a new data set for future AI to be trained on. As the piece writes, the AI was trained using images of the participants' brains at the start and end of the process, giving it the opportunity to spot neurological differences that the human eye might miss. One of the patients responded well to treatment for four months before relapsing, for example, and the recovery signal disappeared a month before the relapse.

7:15Now that the AI has been trained, it can be used in further studies like this, giving researchers a much better set of data than they get with self-reporting alone. One of the things that I see starting to shake out is that as politicians are talking about weighing the risks of AI with the opportunities, the area that is clearest to people around those opportunities seems to be in the health sphere. The more studies we get like this, the more likely it is people fight to continue to be able to leverage those benefits in the medical field, even as policy attempts to put guardrails around other types of AI in order to prevent future bad outcomes.

7:46In any case, that is going to do it for today's AI Breakdown Brief. Next up, the main AI breakdown. Hello friends, quickly before we get into the main episode, I wanted to tell you about one opportunity. It is the beginning of October and that means we are refreshing the very limited number of personalized AI consulting sessions that we have for this month. These are short high impact consulting sessions where I will get into what you are trying to learn, how you are trying to apply AI to your business or life, and do my best to get you up and running with resources to take your efforts to the next level.

8:19If that's something that's interesting to you, like I said, I make available an extremely limited number of slots for this per month, so shoot me a note at nlw at breakdown.network, and I will do my best to get you on the calendar. With that, let's listen to the rest of the episode. Welcome back to the AI Breakdown. One of the most anticipated evolutions of the artificial intelligence space is the move to be able to run large language models on mobile devices such as smartphones. Now, by and large, right now, models are too big to be able to run locally without serious performance degradation, but that hasn't stopped people speculating about local on-device type of LLMs being the future of the AI space.

8:56All year, we've had articles about this evolution in the LLM space. Back in July, the information wrote about it in a piece called Small Devices Could Soon Handle Large Language Models. The specific prompt for that piece was an announcement from an AI startup called OctoML, but it articulated some of the benefits as well. They write, Running large language models on the Edge could alleviate some of the exorbitant cloud computing costs facing AI companies by taking advantage of computing power sitting idly on their customers' laptops and devices. That would also benefit cloud providers, which have to ration access to server hardware for their own internal teams.

9:29However, as they point out, Historically, AI researchers have struggled to run sophisticated AI algorithms like LLMs on the Edge, since those models have to share computational resources and memory space with other important functions like, well, actually being able to use your phone, and are usually more compute-hungry than the voice recognition or computer vision models already running on devices. Now, the piece also points out that Edge AI has other benefits such as being able to run without an internet connection, which might be as banal a value proposition as being able to use it on an airplane without Wi-Fi, or as serious as having a medical device assistant during a high-risk surgery.

10:02They also discuss the benefits of latency. They write, in the case of Edge AI, processing data locally means there's no need to transmit data over the internet to a remote cloud server and back, speeding up the process. Finally, there is the benefit of privacy. Simply put, AI models work better when they have access to more customized information about the person using them, especially when you're dealing with cloud services that presents a risk, whereas if a model could run on device, users might have more confidence that their data wouldn't be leaving that device. Now, this seems to be part of the barrier that has held Apple back, for instance, from going deeper into the LLM and generative AI space.

10:37In an article a couple weeks ago about how Apple had increased its training spending to millions of dollars per day, the information once again pointed out this problem. Quote,

10:55So far, that just hasn't been feasible. And yet there has been a lot of discussion lately that that may be a limited-time challenge. In August, ZDNet wrote a piece called, Could you soon be running AI tasks right on your smartphone? MediaTek says yes. They write, Today the Taiwan-based semiconductor company announced that it is working with Meta to port the social giant's Llama2 LLM, in combination with the company's latest generation APUs in Neuropilot software development program, to run generative AI tasks on devices without relying on external processing. Still, they point out that even Llama 2's small dataset of 7 billion parameters represents a size of around 13 gigabytes, which, as they put it, is, quote, outside the practical capabilities of today's smartphones.

11:37And that's what made Stability AI's announcement a couple days ago all the more interesting. Jan Peleg tweets, Stability AI just casually dropped 3 billion parameters model, trained on 4 trillion tokens, outperforms most 7 billion models and a 20 billion model. Now very quickly, people started to put this in the context of this question of smartphones running LLMs. Daniel Samanez quote tweeted Ahmad Mostak announcing the new LM Alpha model and adding likely it can run on iPhones and Pixel phones. Indeed, Ahmad confirmed that in a later conversation. After AI content creator Igor Pogany wrote, can't wait until we can run LLMs like ChatGPT locally.

12:11We'll make many, including me, way more comfortable with putting sensitive info like finances and health data. Plus, you'd have your AI buddy with you even when phone service isn't, like on a long hiking trip or out at sea. So many potential use cases. Should be happening within a year, according to Ahmad Moustak of Stability AI. Well, Ahmad jumped into the comments and said that the new stable LM Alpha, quote, runs on a normal smartphone and we have much better coming. He also added in a separate tweet, Only a short matter of time before an open 3B parameter model overtakes GPT 3.5, in my opinion.

12:42Then you can have swarms of them as experts as they run on your phone. Now clearly this shows the direction that Stability AI is heading with this smaller, more performant model. Even as others are trying to think about how to soup up hardware capacity to run these models, others, like Stability apparently, are trying to shrink the model sufficiently that they can be used on today's phones to accomplish actual useful things. Now of course, as I just intimated, people aren't approaching this just from the smallifying LLM side. They're also thinking about it from a hardware perspective. Dave Lee tweeted, I just successfully convinced ChatGPT that OpenAI should make their own LLM phone.

13:18An interesting question. Should OpenAI make a phone? In almost all cases, making a phone right now to compete against Android and Apple is akin to committing suicide. There's practically no chance of a new platform gaining much traction due to the dominance of the existing mega platforms of Android and iOS. However, the advent of GPT-4 level LLMs like ChatGPT presents a unique challenge and opportunity. Android and Apple will likely make their LLMs the default AI interface for their mobile devices, especially as LLMs take up more and more of the user time on mobile devices. Especially for Google, this is existential as they are dependent on search engine revenue, and as AI replaces much of search, it is imperative that Google equips Android devices with a Google LLM that is comparable to ChatGPT and GPT-4.

13:59Dave basically concludes that the only way for OpenAI to fight this tide is to release a new phone. He writes it could be centered around their LLM and be a completely new experience. They could give apps prioritize access to their APIs. It would be a new ecosystem and OS. Now, of course, Dave didn't just write this up. He asked GPT-4 if it agreed. ChatGPT came back with some pros, including full integration, a unique user experience, competitive advantage, data, and ecosystem control. But then cons, including that market saturation, how resource intensive it is to design hardware, the risk of failure, the distracted focus.

14:31And then it goes on and on with Dave continuing to try to argue to ChatGPT that it should focus on doing its own thing rather than just partnering, ultimately leading to ChatGPT to say, yes, OpenAI should consider making its own phone and operating system to fully realize the potential of its LLM technology in shaping the future of human-computer interaction. Now, of course, a lot of what we talked about was exactly this, that OpenAI CEO Sam Altman and Apple's former famous designer Johnny Ive have been in conversations around building what they call the quote iPhone of artificial intelligence, although it appears from sources with information that the actual form factor of this device isn't clear, just that it's an AI native from the ground up reimagining of a personal computing device for the AI era.

15:11Now, importantly, this is more than a few idle conversations over dinners in San Francisco, as they've also been discussing a potential billion-dollar investment from SoftBank to get the venture started. And yet there are other efforts in this space as well that aren't strictly confined to just a phone. Another big announcement from last week was the Meta AI integrated Ray-Bans that were announced at Meta's Connect event. This is of course an inherently mobile use case for artificial intelligence, given that you're wearing these things as you're walking around, and the type of information you're going to be asking is things like, what am I seeing in front of me?

15:44How do I fix the problem of the appliance that I'm currently looking at? Etc. There are also startups that are coming after the AI hardware device space. Probably the most notable of those is Humane, which had that very well-received demo at TED earlier this year, that involved, among other things, a live voice translation, where the speaker who was presenting had a statement that was translated into another language in his own voice, as folks watched. Now, the people at Humane are mostly ex-Apple folks, and Sam Altman has been one of their biggest funders, suggesting some amount of continuity to this excitement and interest in a different type of approach to AI hardware.

16:18Currently, Humane is scheduled to unveil more details in a couple of weeks. Interestingly, though, there is another argument that some making, that although it doesn't have all of the capacities of improved performance and privacy that would come with a truly edge AI that lived on device, that in many ways, ChatGPT with vision represents a first step towards this AI phone world. Sunny Mukherjee tweeted, Microsoft missed an opportunity here because if Windows Phone was still around, they could have integrated ChatGPT into it. Apple has the phone in the OS but no LLM yet, and Microsoft has the LLM but no phone.

16:52ChatGPT can now see, hear, and speak. Now, at the end of last week, I shared some of the examples of how people who have early access to chat GPT with Vision are using it. And the ability to take visual input from the world around you certainly does give off a sense of where things are headed and how an AI-native phone or device might be a game changer. The example that you've got on your screen right now is from McKay Wrigley, who took a picture of his team's whiteboarding session, fed it into GPT with Vision, and then had it write some actual working code. So summing up, it seems like there is a clear trajectory and trend to exploring the way that hardware, both existing modalities of hardware as well as new attempts and new form factors of hardware, can transform how people integrate artificial intelligence into their daily lives.

17:36And of course, as much as they haven't made a big move into this space yet, everyone continues to wait to see what Apple will do. As Robert Scoble pointed out in July, even if OpenAI introduced a phone tomorrow, how will the world switch? It won't. Apple knows this. It has the only store in many cities where people buy new things from. So all in all, it is going to be a very exciting time to see what exactly companies do in this AI hardware space. I think the only thing that is for sure is that we're going to see more, not less, attempts towards it. And I will, of course, keep you updated as we learn more.

18:08Thanks as always for listening or watching. Until next time, peace.

18:20Thank you.