Apple Spending Millions Each Day Training AI

0:01Today on the AI Breakdown, we're looking at reports that Apple is now spending millions of dollars a day training their internal AI models. Before that on the brief, OpenAI excites the developer community with the promise of their first developer event in November. The AI Breakdown is a daily podcast and video about the most important news and discussions in AI. Go to breakdown.network for more information about our Discord, our YouTube, and our newsletter.

0:27Hello friends, one quick note before we dive into the brief. Today is my birthday. As a birthday present, I have one request. If you are finding yourself getting value from the AI breakdown, whether it's professional or just personal, the biggest thing that you can do to help me out is actually leave a rating or a review. When it comes to recommending new podcasts to people, Spotify and Apple, and all the other players really, look at ratings and reviews more than just about anything else. I really appreciate all the people who have taken the time to do that already. And if you haven't yet, today would be an amazing day.

1:02I appreciate each and every one of you for listening. So let's now get to the brief. Welcome back to the AI Breakdown Brief, all the AI headline news you need in around five minutes. Today, we kick off with a small announcement that has people very excited. Yesterday, OpenAI CEO Sam Altman tweets, on November 6th, we'll have some great stuff to show developers. No GPT-5 or 4.5 or anything like that, calm down, but I still think people will be very happy. So what's going on is this is what OpenAI is calling its first developer conference. They write, the one-day event will bring hundreds of developers from around the world together with the team in OpenAI to preview new tools and exchange ideas.

1:41In-person attendees will also be able to join breakout sessions led by members of OpenAI's technical staff. The quote that they ran with from Sam Altman is similarly vague, saying, we're looking forward to showing our latest work to enable developers to build new things. Now, of course, this has got people speculating like crazy. Vion Williams says, Okay, got it. Agent systems based on GPT-4. Max Kennan says, Some wild guessing. Cheaper API, some kind of agent, or some kind of code model. Robert Scoble says, Multimodal coming? November is going to be fun here on X. And Nate Chan points out, OpenAI has the chance to create a WWDC-type level of excitement event if they do this every year and with as much care as Apple.

2:19Konstantinos, however, points out that that's kind of far away, and that in some ways a two-month lull might represent an opportunity for someone else. He tweets, No new versions of GPT until at least November 6th, so everyone can calm down. Or is this an opportunity for others in the space to get ahead? Now, I think the big question from where I'm sitting is how much whatever gets announced there is really just for developers, things like new tooling, versus how much is something that those of us who are normies in the AI space will actually derive immediate value from. Of course, updates for developers eventually mean value accrues to the normies as they get to use new features that are built on top of those new tools for developers.

2:54But in the meantime, everyone's going to spend the next two months just guessing. When it comes to the theory around multimodality, what's going for that idea is that OpenAI had previously talked about bringing a multimodal version of ChatGPT this year, and had intimated that the only reason that they hadn't was an AI chip shortage. That would certainly fit with the pattern of doing things with GPT-4 that souped up so much from where it started that it almost feels like a different model. One other interesting little stat before we move on, Logan, who does developer relations at OpenAI, tweeted, It's also crazy to think that more than 2 million developers are using our APIs to build AI experiences.

3:28Just an interesting little nugget in the scale of who is building right now. Now, that OpenAI event will be in San Francisco, and staying in the realm of California, that state's governor, Gavin Newsom, has signed an executive order designed to start preparing California for the AI revolution. The governor's office writes, California is the global hub for generative artificial intelligence. We are the natural leader in this emerging field of technology. tools that could very well change the world. To capture its benefits for the good of society, but also to protect against its potential harms, Gavin Newsom issued an executive order today laying out how California's measured approach will focus on shaping the future of ethical, transparent, and trustworthy AI while remaining the world's AI leader.

4:07So what is actually in this executive order? There are a number of different provisions. One is a risk analysis report. This will quote direct state agencies and departments to perform a joint risk analysis of potential threats to and vulnerabilities of California's critical energy infrastructure by the use of Gen AI. A second area is a procurement blueprint. Basically, this is all about developing a process by which generative AI can be used in government offices and agencies. Beneficial uses of Gen AI report. In addition to just looking at those critical risks, Newsom is also directing state agencies to develop a report, quote, examining the most significant and beneficial uses of Gen AI in the state.

4:43Deployment and analysis framework. Basically, this is a mechanism by which the state can start conducting pilots as well as creating sandbox environments to test new projects. The executive order also establishes a formal partnership with the University of California, Berkeley, and Stanford to consider and evaluate the impacts of Gen.AI on California. And one that's really interesting is the category they call state employee training. The governor's office writes, To support California's state government workforce and prepare for the next generation of skills needed to thrive in the Gen.AI economy, agencies will provide trainings for state government workers to use state-approved Gen.AI to achieve equitable outcomes and will establish criteria to evaluate the impact of Gen.AI to the state government workforce.

5:20Now, one of the things that I think is interesting about this executive order is the extent to which it is focused on preparing the government itself, the actual offices, departments, agencies that comprise California's government, figuring out how to actually use and deploy generative AI themselves. I think there's something very positive about that in that rather than just viewing this technology in the abstract, they're actually trying to put it into practice. It seems likely to me that actually getting one's hands dirty with these tools is going to end up leading to better policies around them.

5:51What's more, California has now created something of a template that other state governments could use should they want to themselves engage in a similar set of experiments and research. Now, moving on to our next topic, but staying within the theme of governments and AI, Google has announced an updated policy around AI-generated election ads. Bloomberg reports that starting from November, Google will mandate that any election-related ads that feature AI-generated content will be required to have what they call a prominent label and disclosure identifying them as such. Bloomberg writes, Advertisers must include prominent language like, This audio was computer-generated, or This image does not depict real events.

6:28Notably, the policy does not apply to minor fixes such as image resizing or brightening, but in many ways this is one of those just super, super obvious things. It is really hard in general to create mechanisms by which AI-generated content can be identified, but platforms who are gatekeepers of content that is distributed on their platforms, especially paid advertising content, can require this sort of disclosure. And so I think that while Google might be the first to make this policy relative to the big advertising platforms, it seems likely that others will follow suit, if only because it's likely that they will be forced to at some point in the future if they don't do it voluntarily.

7:02Last up today, Time Magazine is doing a big cover story around artificial intelligence. One of the covers is an excerpt from Walter Isaacson's new biography of Elon Musk that's all about how his opinions on artificial intelligence have evolved and why it started to dominate a lot of his thinking behind the scenes. And the other part of the cover story is a first-time list of the 100 most influential people in artificial intelligence. There are a lot of familiar faces there. Sam Altman from OpenAI, Eliezer Yudkowsky, Jan LeCun from Meta, and many, many more. I think that tomorrow I might do a deeper dive into who is on this list, But for now, an interesting reminder that for whatever summer lull we might have had in terms of the mainstream exposure of artificial intelligence, it seems likely that that lull won't last for a particularly long time.

7:47Anyways, friends, that is going to do it for today's AI Breakdown Brief. I'll be back soon with the main AI breakdown. Before we get into the main AI breakdown, I want to tell you about today's sponsor, Supermanage. If you work in a professional setting, you probably have some version of a one-on-one meeting, either with the people that work for you or the people that you work with. Unfortunately, all too often, those one-on-one meetings become glorified catch-up calls. Don't you wish you could jump right to the stuff that really matters? That's where Supermanage comes in. Supermanage AI magically distills your team's public Slack channels into a real-time brief on any employee, any time.

8:23Catch up on contributions, work in progress, challenges they're facing, sentiment, everything you need to show up ready for a truly meaningful conversation. And it's completely free. Visit supermanage.ai forward slash breakdown today to start making the most of your one-on-ones. And thanks again to Supermanage for sponsoring the AI Breakdown. Welcome back to the AI Breakdown. Today, we are talking about one of the biggest questions in the business of artificial intelligence, which is if, when, and how Apple might make their entrance into the race. We've just gotten a new report from the Information who have become by far the most significant source of, at least Silicon Valley-based AI scoops, of any news publication out there.

9:07The highlight of the report is that Apple is apparently spending millions of dollars each day to train a new AI model. But what are they actually going to use it for? Who's the team that's working on it? Let's look and see what the reporting has dragged up. So, a couple pieces of information in here. First, the information reports that John G. and Andrea, who is Apple's head of AI, first authorized the formation of a team to develop conversational AI, i.e. LLMs, around four years ago. The team is called Foundational Models and is led by Romang Pang, who is an ex-Googler who worked with Gianandreo when he oversaw Google's AI research arm.

9:40The information reports, quote, the team remains small, numbering around 16 people, but the budget for training Apple's most advanced models has grown to millions of dollars per day. Continuing, they write, the Foundational Models team at Apple plays a similar role to that of AI teams at companies such as Google and Facebook, where researchers produce the AI models and other groups then implement those models into the company's products. Now, in addition, the information writes, quote, there appear to be at least two other relatively new teams at Apple that develop language or image models. A recent Apple AI research paper and employee profiles on LinkedIn point to the existence of a visual intelligence team working on software that generates images, videos, or 3D scenes.

10:17This is probably not that surprising given how much emphasis Apple has on augmented and virtual reality given that their biggest release of this year is the Apple Vision Pro. Now, another team the information writes is working on long-term research involving multimodal AI. But what is Apple actually thinking about where this technology might come to bear in their products? One use case discussed in the piece is an LLM that could interact with customers who use AppleCare. Another, which is one of the most anticipated AI features from Apple, would be a fairly complete overhaul and upgrade of Siri.

10:47The piece writes, The Siri team plans to incorporate language models to let users of the voice assistant automate complex tasks in ways they currently cannot, such as creating and sending a GIF with a simple command. The information says that this effort hadn't been previously reported. In terms of how advanced the technology is, the information writes, People on the Apple team believe its most advanced language model, AJAX GPT, has capabilities exceeding those of OpenAI's GPT 3.5. Another person with direct knowledge of the model says that AJAX GPT has been trained on more than 200 billion parameters.

11:17One issue, however, is how this would integrate with an on-premise implementation that would be more privacy-preserving. As the piece writes, questions linger over how Apple can incorporate LLMs into its products. The company's leaders prefer running software on devices, which improves privacy and performance, as opposed to on cloud servers. But, as they point out, an LLM with more than 200 billion parameters couldn't reasonably fit on an iPhone. Still, a lot of the reporting in this piece points to Gianandria as just super skeptical of the usefulness of chatbots powered by LLMs. Quote, while he has repeatedly expressed skepticism to colleagues about the potential usefulness of chatbots powered by language models.

11:53A person familiar with the matter said over the past year, he's come around to acknowledging the technology's ability to accomplish tasks after seeing a number of internal demonstrations. I don't know, man, it's a little hard from the outside to get a grasp on how much, to me, this muddies the question a little bit of the extent to which Apple not moving more aggressively into the AI space has to do with them having a clear picture of how they want to integrate AI into their existing products versus just kind of missing the boat on the consumer potential of LLMs. The piece also points out how much bleed there is in terms of talent between these big tech companies.

12:28They write, after he arrived at Apple in 2018, Gianandria helped recruit key engineers and researchers from Google. He also favored using more of Google's cloud servers, including servers imbued with Google-developed AI chips, known as tensor processing units, to train the machine learning models Apple uses to improve features in Siri and other products. The piece also points out that of the 18 people who have contributed to Axlearn, which is Apple's internal software to train Ajax GPT. A dozen of them joined Apple within the last two years, and seven of them had previously worked at Google or Meta.

12:56Now, obviously, Apple is a really cash-flush company, so even the fact that they're spending quote millions of dollars a day training AI doesn't necessarily mean that some new product launch is imminent. Still, whether it's LLMs or something else, it feels very likely to me that market pressure is going to get bigger and bigger on Apple to at least articulate what its vision for the AI space is, even if it's something very different than its competitors at Google and Meta. Speaking of Meta, the information also shared a piece a couple days ago called Inside Meta's AI Drama. The information once again writes, many of the scientists and engineers who worked on Lama have quit, embittered by a previously unreported internal battle over computing resources, with another Meta research team working on a rival model that the company ultimately abandoned.

13:37Specifically, the information writes, more than half of the 14 authors of the original Lama research paper published in February have since left the company. Said Joel Pinault, the head of Meta's Artificial Intelligence Research Lab, quote, retention and attraction of good talent is probably where I spend most of my time. Of the research scientists and engineers who have left, a number went and founded a startup called Mistral AI that was notable for having raised a$113 million seed round, and others have gone on to join companies like Apple. It sounds like from the story that there were multiple different divisions within the fair or Fundamental AI Research Team based in different locations that were working on different foundation models.

14:13In May 2022, one FAIR team that was based mostly in the US publicly released something called OPT-175B. A few months after that, they started working on an even larger model. At the same time, a different FAIR team based in Paris had begun working on a separate LLM that would eventually be dubbed LLAMA. The model was smaller than OPT, and as the information writes, the team believed a smaller model would be more efficient at inference, the process of generating responses to questions. And when push came to shove, the big tension was around computing resources. Quote, rivalry over access to computing power inflamed tensions between the teams.

14:46The LAMA team in particular felt overlooked. They received less computing power than the North American-based OPT team, said people with direct knowledge of the situation. Now apparently, and this makes sense, questions started to grow around why they had two teams working on similar projects that both required what were ultimately pretty finite resources. By February of this year, leaders at the company had decided to start bringing together members of the competing LLM teams to focus on a single model, which would become LLAMA2. At that point, the OPT model was abandoned, and it sounds like part of the reason for that was that the team had just churned through personnel.

15:17Apparently, around half of the 19 authors listed in a May 2022 paper about OPT have subsequently left Meta. Now, you might think that given how significant and influential LLAMA has been this year in shaping how generative AI is developing, that some of these tensions might start to go away. However, according to the information, quote, despite the success, tensions are still shimmering among researchers as Meta's attitude to AI research is evolving. Fair has traditionally had a bottom-up culture led by researchers, with a mission centered on advancing breakthroughs in AI. But as Zuckerberg has become more intent on incorporating AI into Meta's apps, Fair's focus has narrowed.

15:50It has canceled research that doesn't have a product slant, such as work on protein folding. Anyway, ultimately, just a great example of how much is going on behind the scenes at these big tech companies as they jockey to figure out where they stand and what they can do in the race to shape the artificial intelligence field. A last example of this that we'll look at today comes from Amazon. Yet another piece from the information, published on August 30th, was called How AWS Stumbled in AI Giving Microsoft an Opening. The piece begins, Long before ChatGPT arrived on the scene last year, Amazon Web Services was developing artificial intelligence software akin to the technology that powers the hit chatbot from OpenAI.

16:25AWS had hoped to unveil the software, then known inside the company as Bedrock, at its annual customer conference late last November, but had to postpone it due to technical snags. That ended up being a fortunate decision. A couple days into the conference, OpenAI released ChatGPT immediately wowing the tech world. AWS leaders soon realized Bedrock wasn't on the same level as OpenAI's AI software, but AWS had to do something its leaders felt. AWS was the number one provider of cloud services, while OpenAI had formed a tight relationship with AWS's biggest nemesis, Microsoft. In the following weeks, AWS chucked out its old plan and attached the Bedrock name to a new service that allows developers to connect cloud applications with a variety of LLMs.

17:02The change of plans, the information writes, details of which haven't been previously reported, illustrates how ChatGPT's sudden surge in popularity caught Amazon off guard. Now heading into the fall, a lot of the focus is turning to Google and the extreme expectations that they're pushing around their Gemini model, which based on the amount of compute alone, is being positioned as one of the first serious competitors for OpenAI's GPT-4. This battle is quite obviously going to shape all of the tools that you and I use and dictate a lot about how the industry evolves, so it is one that we will keep our eyes on closely.

17:35Will Apple actually jump more fully into this race? Only time will tell, but it certainly sounds like they're starting to spend the money that they might need to if they're going to get in the game. That is going to do it for today's AI Breakdown. I appreciate you listening or watching as always. If you are getting value from this content, I would love it if you would go leave a rating or a review. It makes a big difference and I appreciate each and every one. Until next time, peace.