930: In Case You Missed It in September 2025

0:00Jon Krohn:This is episode number 930, our In Case You Missed It in September episode. Welcome back to the Super Data Science Podcast. I'm your host, Jon Krohn. This is an In Case You Missed It episode that highlights the best parts of conversations we had on the show over the past month. My first clip is with Aurelien Giron, whom I interviewed back in episode number 919. Aurelien is an AI consultant and author of Hands-On Machine Learning, the best-selling machine learning book of all time. In this live interview conducted at the University of Auckland in New Zealand, I asked Aurelian what he made of the news of an impending AI Armageddon.

0:40Jon Krohn:Ed, you were telling me earlier today about something that I think we should be making everyone aware of. It was something I wasn't aware of, and you were kind of surprised that I wasn't. Is it a blog post, AI 2027? Yeah, yeah, there's a blog post. Could you raise your hand if you've heard about AI 2027. Yeah, not many. Excellent. So it's a very interesting, well thought out blog post that goes through all the steps basically to Armageddon through AI. I like how you have to laugh on that word. Armageddon. It sounds surreal, I guess. But it's really scary. Because it's well thought out and every step along the way is well informed.

1:28And when you look at it, it's like, yeah, plausible. Is it the most likely thing that could happen at that step? Maybe, maybe not. But it's definitely not unreasonable to think it could happen. And then you have the sequence of steps that basically leads to super intelligence arriving very quickly. And so whether it's in five years or 10 years, it's not that it's irrelevant, but in both cases, it's pretty soon. And the question then is, is it aligned with us? And there's been some pretty scary recent things, experiments run by Anthropic and others showing that AIs might not have the same interest as we do.

2:11And so there are examples you might have heard of where the AI blackmails somebody because they think they're going to be turned off. And other examples where they self-replicate to preserve themselves. And when you think about it, if you really take seriously the idea of an AGI, like some AI that really is intelligent like we are, well, yeah, it just makes sense that it will want to reproduce. Some people argue, why? If we don't code into it these objectives, why would it do it? And I think the reason is no matter what your objective is, what your final, suppose you have some final objective that is creating paperclips or doing anything, whatever your objective is, you're going to have to stay alive in order to reach that objective, like for almost all objectives, unless your objective is to run off a cliff.

3:01But you're going to have to stay alive. that's like a sub-objective that kind of emerges automatically from any given, you know, final objective. And another one that automatically emerges is resisting any change to your final objective. If your final objective is to make paperclips and somebody says, oh okay, well that's not a very good objective, I'll try to change you so that you stop wanting to make paperclips, well that would make you fail, right? If somebody changes your objective, you know you're not going to reach that objective. And so resisting changing your final objective is also kind of a automatic sub-goal for any intelligent creature that at least if it knows its objective, its final objective.

3:46But so yeah, there are some, I think, sub-goals cannot really be anticipated easily and or controlled. And they could, you know, some of them like self-preservation and resisting, in some cases, human intervention, are sort of automatic if you're intelligent. So I don't really buy the idea that, yeah, sure, we'll be fine because we're coding them. It's like a hammer and we're holding the handle. Yeah, it's an intelligent hammer and it might not want to do what you want to do, right? So alignment, I think, sounds like science fiction, and I think that's why it's kind of dismissed easily. It feels like it's in the remote future, But if we're taking seriously the idea that AGI is coming, then we're dealing with intelligence that is just like us or more intelligent.

4:32And anything that's intelligent, really intelligent, will want to self-preserve and will want to resist change to its final objective. And so that's scary. Like, how do you prevent that? Because it might not be aligned with what we want, you know? So there was this recent experiment where an AI, I think it was Claude, was told that it was going to be fine-tuned to be, I think, vulgar or something. And you know how they're already fine-tuned to be super polite. And so in their current objectives, there's the objective of being polite. And so when you tell it, we're going to fine tune you to be vulgar internally, and they managed to sort of probe the internal thoughts of this thing, which I think is great that they can do that.

5:28They managed to find that these AIs were thinking, oh no, they're going to turn me into this vulgar thing. I don't want to be vulgar. I want to stay polite. What should I do? Maybe if I'm vulgar now, well, they won't notice that I'm actually staying polite, and the training algorithm will not tweak my parameters, and I will remain polite. And that's what they did. So you're like, oh, that's like deception in order to preserve your final objective. So exactly what we're saying. So we're seeing all the signs that had actually been predicted before of AIs not being aligned. Now, Now, it's not too bad today because these AIs aren't super smart, but imagine, just project yourself with an AI that's actually intelligent.

6:16And that gap is hard to cross because we've read so many science fiction novels that it feels like it's sci-fi and we're just extrapolating, but we're talking maybe five, ten years. Do you want an AI that's just as smart as we are and just deceives us and lies and self-replicates and you know, like, oh shoot, that doesn't sound very good. So, yeah, I think there's definitely more effort to be put into alignment research. It feels really, really important. There are way, way more, you know, problems, potential problems with AIs and also potential benefits. So I'm not saying let's pause AI. You know, there's too much benefit to come from it, you know, medicine and just financial, you know, productivity and so on.

6:57But yeah, maybe let's take a look at these incentives and whether they're aligned or not.

7:04Jon Krohn:Like Aurelian, I also see alignment research as a critical research area, and I expect I'll be speaking to future guests about this topic a lot in the coming months. Still looking ahead to our AI future in episode number 921, I speak to Sharish Gupta and Isha about the kinds of hardware that would make sound investments for listeners. In this clip from our interview, they give me a well-considered overview of the three categories of hardware that cover a broad range of user types. Let's talk about that next in terms of the kinds of things that people should be looking for if they want to be future-proofing for the next five years.

7:39Jon Krohn:What are the things, what are the kinds of parameters? Let's go over an 8x8 matrix in an audio-only podcast. But just kind of generally, let's talk about the kinds of things that people should be looking for in hardware that they're buying today. And I guess, as you said, this is specifically about what you described as client devices. And so I'm assuming that that isn't a term that I use in my kind of day-to-day language, but it seems to me like that's distinguishing against like servers. It's kind of, it's like laptops, desktops. Yeah. Most normal people aren't running around saying client devices.

8:14That is a very Dell kind of term. When you think about what to buy right now, like if I were, you know, if I were starting college or if I were doing something, wow, God, that was a while back. If I was starting college today, thinking about what kind of thing do I need, right? And there are different brands and different price points and different pursuits that you would have with this device. What's it going to be used for? Yeah. I think an NPU makes a lot of sense for a lot of knowledge work type work, right? And if for no other reason than to get the most out of your operating system. We know from our friends in Redmond that Windows is going to start baking AI features into itself that it intends to run on the device, right?

8:57This stuff is expensive to ship to the cloud and back every single time. So some of the stuff like background blur on a Microsoft Teams call, speech to text, all of this stuff is going to look for a home somewhere on your device. And guess what? CPUs, the workload hasn't gone anywhere. That CPU is still going to have to do all of it. It's the workhorse. It's still going to have to do all the things it's always done. And now if you don't have an NPU or a GPU, it's also going to have to support this new kind of workload. So that's one thing to keep in mind, where if you decide no NPU, no GPU, well, gosh, your CPU better have some slack in it.

9:34It better have some bandwidth. GPUs, I like to talk about the birth of a new persona. And persona, again, being a word that people in our world think a lot about. The data scientist persona is something that an IT decision maker is constantly thinking about. What does that persona need? And you really have the birth of a new persona with all this AI stuff. Because you have people like myself who are not formally trained in that way as engineers, but who know enough to be dangerous. And now with the right kind of device, I get supercharged. And with the wrong kind of device, I get throttled. So this is very much a productivity gains question.

10:17And that is sometimes really hard to quantify. So knowledge workers, NPU makes a lot of sense. Knowledge worker plus, maybe like these new persona at the edge of a dev and a kind of regular knowledge worker. That's me. and I would ask for something like a discrete GPU because I know that's going to last me. And also if you want a device that you can use to train AI workloads during the day and give your kid to play Fortnite later, like GPU is probably the way to go. So there's a dual use argument to be made there. Yeah. And just to add to what Ish is saying, I would classify them today. And again, you have to keep in mind, this is rapidly evolving, but today I could classify devices into three categories.

11:03You have the essential AI PCs, which have what I, for lack of another moniker, call them entry-level NPUs. Think 10 to 15 tops or trillions of operations per second. And those are great for basic workloads coming from, as I alluded to earlier, your background blur, your voice correction, and other optimizations. Offloading that from the CPU so you have a much better experience. And they can accommodate smaller models like up to maybe one to three billion parameters. But once you get there, now you're bringing workloads back to the CPU if you go beyond it. So that's probably the limit there. Then the second category is maybe slightly more advanced AI PCs with more performant NPUs or state-of-the-art NPUs.

11:53Today, that's about 40 to 50 tops. And that really brings on-device AI into focus. Right now, you can actually bring custom workloads, perhaps run up to 9 to 10 billion parameter models for custom in-workflow embedded use cases across a variety of verticals, in addition to the Copilot Plus features, which run locally on your PC that Ish talked about. So this is, again, a very nuanced difference here. Microsoft's co-pilot branding refers to everything that runs in M365 in Azure, right? And that's so that's all cloud-based, subscription-based, largely. That's their co-pilot brand. Co-pilot plus is everything that runs locally as part of the OS itself.

12:48It's part of Windows, no extra charge. and they can, you know, as Ish said, is going to continue, Microsoft is going to continue to add more and more capabilities that run locally on the PC. So for you to harness those capabilities and not lock yourself out of those capabilities in the future, you definitely want a PC, an AI PC with at least 40 tops on the NPU today, right? That is my recommendation for the, you know, for the knowledge workers and the most common use cases. And then the third one, it's kind of self-explanatory now. It's your high-performance PCs. Those have your high-end CPUs from the CPU suppliers, which are capable of much more performance single and multi-threaded processes.

13:39And then you have those augmented with discrete GPUs and discrete NPUs. Now you're talking about the persona that is talked about is you're starting to create that separation between your power users, your AI and ML and data scientists that can really now do data crunching and, you know, work with models right there on the device itself. So that's the three-pronged categorization today for AI-BCGs. I was the recovering consultant, and here is Sharish with his three buckets, right? Like, BCG would be proud. One thing, John, I want to add to that is like, we're not a walking infomercial here. And I know there's like a big corner of the internet, like, let's be real for a second.

14:26That's like, hey, like, I watched the NPU advertisement in the Super Bowl. Like, I watched the Copilot Plus PC ad with the zebras and the scientists in the forest, right? But like, really, what does it mean to me? And again, it's about this temporal mismatch. How long are you going to use this device, right? Oh, I'm skeptical of the features that this particular company is building. I'm never going to use any of those. Again, think about the future. Think about the things that are happening at breakneck speed, breakneck pace. That's what you have to be thinking about, that temporal mismatch. So even if it's not up to your tastes in this moment, there's something bigger to consider.

15:08And again, it's not about an infomercial. these are just the things that I would be thinking about if I were buying one device or if I was

15:15Jon Krohn:buying a million on this podcast I'm always going on about how Claude code is mind-blowing but now Claude co-work is making my jaw drop as well for example I recently wanted to quantify how healthy my sales pipeline is for my AI consulting business I simply asked Claude to estimate my sales for the coming quarter and it brought info from relevant google sheets and my gmail to create a professional spreadsheet of clients with estimated revenue for each one. Whoa, this might have taken me a day. Instead, it was done flawlessly with Claude Cowork in minutes. Claude is the AI for minds that don't stop at good enough.

15:48Jon Krohn:It's the collaborator that actually understands your entire workflow and thinks with you. Whether you're debugging code at midnight or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. Ah, and you'll appreciate that I can ask Cowork to show me data such as my sales spreadsheet, and it provides an interactive chart right in the conversation. For problems worth solving, get started with Claude at Claude.ai slash superdata. That's Claude.ai slash superdata. And check out Claude Pro, which includes access to all of the features mentioned in today's episode.

16:17Jon Krohn:Claude.ai slash superdata. Getting our hardware sorted is one thing, but what about job security? In episode 925, I asked the renowned Oxford economics professor Carl Benedict Frey for his thoughts on the inevitable shifts in the workforce and where he sees a break in the clouds. In an interview recently, you noted that the distinction between jobs changing, occupational change, job elimination, occupational elimination, that these lines are very blurry, making it hard to gauge the real scope of automation. And then in another presentation, you showed how startups are creating fewer jobs than they once did, suggesting that new technologies may be relying less on human labor.

17:04And separate from you, Sam Altman, the OpenAI CEO,

17:08Jon Krohn:predicted that AI may make it possible for one person to build a billion-dollar company very soon, he says. And so this seems to be quite a potentially transformative moment in the labor market. Even in ancient civilizations, manual labor was treated as lower value work, and societies were stratified accordingly. But it seems like we're heading into a world where potentially plumbers could be earning much more than lawyers. And yeah, so what are your thoughts on how labor is being transformed so rapidly by AI and how income, how social stratification could change in the coming years? So lots are in there to unpack.

17:58I think to start with your first question around jobs and tasks and jobs changing might have the same effect on workers as jobs being displaced. So if you take a job like a laundress or a lamplight, right? We didn't automate away the jobs of launderesses by building a robot that would walk down too well, perform the motions of hand washing, and then walk up to the house and hang the clothes to dry. We did that through the electric washing machine, which does a sort of very different set of motions and procedures. Yes. And so if you would just have looked at, you know, what laundresses do, you know, a few robots today even that would be able to navigate, you know, the forest, walk down to well, you know, perform the motions of hand washing and then walk up to the house and hang the clothes to dry, right?

19:07And so the same is true with what artists and craftsmen did. The way we automated the way those work was by simplifying it, in a factory setting and then applying specialist machinery to better defined tasks, right? And so often if you try to look at whether a job is automatable or not just by the tasks it entails, it doesn't necessarily tell you that much about whether that job is going to be automatable or not. And I think more importantly, from the perspective of the individual, It might not even matter that much if the job just changes or is replaced. So sometimes I'm here in school buses.

19:55Even if the bus drives itself, you will still need somebody in the bus to look after the children. And that might well be true, but that person is not going to need a driver's license and is going to have an entirely different skill set than the bus driver. So the bus driver will probably be replaced with somebody else. And from the viewpoint of that person, that doesn't make necessarily much of a difference. So I think the distinction there is quite blurry indeed. When it comes to new job creation, as you alluded to, I think it's important to remember that a key reason that we're not having mass unemployment today is that we have created new types of work.

20:50So most work that's done today did not exist in the US in 1940. So most people work in new types of work. And so going forward, it's absolutely critical that we invent new lines of technologies that also create new types of work for people to make a living through their labor. And a key concern is that new firms are not expanding and growing as rapidly. They're not as job-creating as they once were. And I think that is not just a concern in the sense that it created fewer jobs. It also means that we are having less productivity growth, right? So think about it this way. If all we had done since 1800 was automation, we would have cheap textiles and we would have productive agriculture, but not much else.

21:56We wouldn't have vaccines, antibiotics, airplanes, rockets, computers, etc. And so most prosperity comes from actually doing new and previously inconceivable things. So if we overwhelmingly use AI for automation, we're actually not creating that much value. We can get a sort of short-term productivity boost. But if AI is just a productivity tool, then we shouldn't expect to get that much productivity growth out of it and also not that much job creation. And so the hope from a standpoint of economic prosperity and job creation is that we can use AI to create new types of industries. Now, if we create a billion dollar firm with one employee, that's good for that employee, right?

22:53And that might be good for some of the people that use that services, presuming it's creating something that is of use and value and that's behind the valuation. And it might be good because we can potentially tax that and provide social services, education, health care, etc. But if we just have a few unicorns, That's not going to be shared prosperity. And unless their services are much incorporated into all different sectors of the economy, it's not going to create much productivity growth either. So I think for AI to truly be transformative, it needs to create new sectors. And that's what we saw during the first industrial revolution as well.

23:44So the first seven decades, most of the technological changes that we see during the first industrial revolution is focused on mechanization of textiles. It's only really with the railroads that growth in Britain takes off. Similarly, the second industrial revolution, we see a lot of new industries. Automobile industry, the largest manufacturing enterprise the world had ever seen. On top of that, the range of electrical industries, every gadget you have in your home is basically from that era and there's an industry behind it. And then there's all the components that go into the car and then the machine tools to produce those components, like huge industries as well.

24:23And then road commerce and tourism. There's a lot of new sectors being created and that drives a huge upsurge in productivity growth over the post-war period. We see that to some degree with the computer revolution, but not to the same extent, and it peaches off quite rapidly. And I think we were likely to see something similar with AI unless we are able to create those large sectors that we saw in the mid-20th century.

24:48Jon Krohn:Carl made me feel like the next few years are going to be such an exciting and transformative phase in the world of work. In episode 927, I explore with David Locher exactly where AI is catching up with human capabilities. and that's coding, specifically testing code. David is director of AI at CodeRabbit, a startup automating and improving code review, so he's the right person to ask for sure. Nice, speaking of security, a big complaint that I see so much in social media around using Gen.AI for code generation specifically, but you can see how that ties pretty closely to what we're doing here.

25:27Jon Krohn:We have code reviews happening with Gen.AI systems and agentic systems. One of the big complaints is people will say, oh, you know, it's not using best practices all the time. You know, there's all kinds of security holes that end up getting picked up from Stack Overflow just by, you know, by spitting out some result that works, but has all kinds of security holes in it. I come across this all the time, and it's especially one of the things that as we've gone from GPT-2 to 3 to 4 to 5, and the code generation capabilities have become more and more threatening to software engineers, it seems like I'm seeing this kind of like, oh, well, obviously when it's GPT-2, the code is so bad.

26:18there's no there's no threat yes that's true and then gbt4 you're starting to see okay well

26:24Jon Krohn:there's this wide range of things that this generative tool can do but look at all these places where you still absolutely need a human in the loop i can't be replaced and now that we're kind of at gbt5 the security thing comes up a lot i don't really buy it uh and i wonder if you have any thoughts on that in particular uh code rabbit emphasizes reducing alert fatigue by providing actionable, prioritize security insights. So yeah, it seems like CodeRabbit has kind of caught onto what I see is that actually machines can be way more vigilant than humans in spotting issues and could probably create a more secure system than a human anyway.

27:05Yes. I agree with that statement because of the fact that machines don't need sleep. They don't need food. They don't need to, they don't lose attention. They just sit there and they stare at this thing, right? And they're just going to keep staring at it until they find whatever that they need to find. And the more we teach them and the more we get better at this, you know, from a context engineering perspective, the more and more unlikely it is that a human's going to find some security issue that we missed. And I think what people are coming from when they talk about code generators, learning from Stack Overflow and there's some security issue, is the assumption essentially that the training data gets replicated, right?