Why We Need New AI Benchmarks, Which Industries Survive AI, and Recursive Learning Timelines | #218

26:20You're seeing most of the LLMs offer a paradigm where you can do this, where you can add your knowledge corpus and train it to be more specific to your individual context. I don't think you'll see individual institutions building their own LLMs. I think that's a very compute intensive, very difficult thing to do. I think you'll see them tailoring the large language models to their context. Sure. To be clear, I wasn't asking whether you think every institution is going to get into the business of pre-training their models. I was rather asking whether you think post-training, which is inclusive of supervised fine-tuning, reinforcement fine-tuning, a variety of other post-training, whether you think that has a long-term future, or will maybe in one to two years we just use pre-trained plus post-trained generalist model off the shelf and not need any internal benchmarks and any internal data sets for post-training?

27:16Well, I think there are clearly going to be use cases where you are going to need the context of the individual company, right? Like if you just take the law firm example, I'll answer it this way. There are documents that company has on how they want their future state documents for M &A agreement to look, right? And the LLMs are not going to have that information. So at some point, you are going to have to see the post-processing layer happening at the enterprise. And what we're seeing more and more is there's ways to design that layer so that you can, as new models evolve, kind of drop those in.

27:50And we are seeing more and more folks experiment with that. So they're using all the new tech that's being rolled out. I think, in fact, what's going to happen is over time, that edge in data is going to be the most valuable part of any company. It was that trade secret type of how do we do things. Now, at some point, it may leak into the public models. Like if you used OpenAI, right? Yeah, if you use any of the frontier models connecting, I remember we were talking to Replit, et cetera, people are using it, and then the data is going straight into the cloud, right? And that's kind of dangerous.

28:23They're going to have to solve that layer in a very powerful way. That's one of my predictions to forecast, et cetera, is we're going to need to see a layer of protection between company data and the broader AI world. Matt, I want to make this a little more tangible Now, I know you can't talk about the work you've done with the hyperscalers, but you've identified, I think, five or six cases where you can speak publicly about it. So if you don't mind, maybe we can toss a few of those in and then talk about them as concrete examples. And since Alex made his no financial involvement statement, I will say I'm a proud advisor and am conflicted in a positive fashion supporting what Matt and Francis are doing.

29:14But do you want to pick one of those? I loved the example on the basketball court. Can you speak to that one? Yeah, sure. So we worked with the Charlotte Hornets on fine-tuning custom computer vision models for draft prep for them. So in their case, they wanted to look at the spatial movement patterns of players on a very broad scale across single-point cameras, across a whole host of different college universities and international locations. And so we fine-tuned a custom computer vision model to specifically look at movement patterns they were interested in before the draft. And so that was a big part of their draft evaluation process.

29:57Let me put this in English. You basically took the video and you were able to use models to evaluate every player based upon the video to see how well they performed at every different – I'm not like a sports guy. So it's the big round orange ball, right? That's becoming clear here, actually. Yeah, yeah. Yeah, sure. So think of if you take typical NBA stats, there are things like points, rebounds, what's called plus minus is one ratio that's often used, which is like the amount you score versus give up when you're in the game. But they're mostly stats that are kind of transactional stats. What they don't look at is the movement patterns of the players who create space, who like where people are positioned at any point in time.

30:40And that's actually a lot of the most interesting data. If you go back to like, you know, some of the original baseball analytics that Billy Bean did for the A's, it's the movement patterns of players and who is in the best spacing. Right. And there are companies that do this on on very consistent formats, like on the same court. What we've been able to do is to do that over many different camera angles, many different stadiums very, very quickly. And that is using custom computer vision models. So we effectively are able to take a single point camera and understand the movement patterns of players in many different environments.

31:15And so the Hornets use this how for team selection, player selection? Draft selection. So to understand which players fit certain characteristics that we're looking at. Fascinating. Yeah, it's a complicated problem, too, because chemistry between, like, it's not just about finding the best player. The chemistry between players matters, too. It gets infinitely complex, and it's a cool little case study. But, you know, Gavin Baker was saying recently that in fantasy football leagues all over the country, which I used to love before I ran out of time. Now I have to spend all my time keeping up with Alex.

31:46Now you have an agent doing it for you and having fun. But that's exactly, yeah, that's the point. Now we're obsoleting human sports leagues, replacing them with robot sports leagues and e-sports. Very 21st century, not Twensen. I'm betting on T-800 again. Yes. That's right. Yeah. But people are losing their leagues. Great, great fantasy football people are losing all over the place because the AI agent is tracking a huge amount of more detailed data. And, you know, if you look at the video footage, you know, if somebody's like making it up and down court very slowly, nobody's going to notice that.

32:24But the AI will notice it in a heartbeat. And then that just goes into the great model. It's really a cool little case study. Since we've asked a little bit about kind of if a traditional business were thinking about how to do this, I'll give a slightly different one, which is Lifespan MD, which actually, Peter, I think this one resonates with you in particular. I know Chris who runs it. Yeah. Yeah. So Life's Vanity is a concierge medicine business. And you can think of it as they have a network of practices, both internationally and the United States, which all have very different sets of data on their patients' kind of practice information.

32:58And so the thing I always start with with any AI use case is you have to get the data right. Before you can even start with AI, you have to make sure that you have the structured and unstructured data together that you want. So the first thing that we're doing for them is on our data platform, Neuron, we're creating a HIPAA-compliant multi-tenant cloud instance where we bring in together all the patient and provider data that's of interest. And we start to bring both a 360-degree view of both the patient and the practice. And so you can start to think of things like if you wanted to understand what longevity focused tests male patients, 35 to 50 are using most frequently, you can start to think about things like that on patient outcomes that are really interesting.

33:40If you want to understand practice performance, if you want to understand where you have certain patients that are not compliant or not as interactive, it's effectively just a control tower to understand everything that's going on across that footprints of practices. And then I think the area where generative AI has become more important for that is actually kind of a chat agents where people can ask questions, knowledge management systems, and really interrogate and ask questions of all the key data from all of those practices. One of the key things that's challenging about that is, obviously, in healthcare, you have to be extremely careful about which data is stored locally at the practice versus how that's brought centrally.

34:18And so the HIPAA-compliant multi-tenant cloud is one of the key components of that, is actually making sure that no patient data leaves the premise of the individual practices. And doctors are able to access certain things, and then certain practice metrics are organized centrally. I heard the coolest thing this week. It's a QA company that has invented talk to your defect. It's just the coolest concept. The defect actually has a personality, and you can ask it questions about itself, like where did you originate? I can totally imagine what you just said in healthcare, being talked to your illness.

34:52Like, have a conversation with it. Where did you come from? How do I treat it? Are you getting better or worse if I do this thing? And it's talking back to you with a personality. It's just the coolest idea ever, isn't it? It is amazing. It's one thing where the defect is a little awkward when you say, here's this bacteria you're talking to. You're like, I'm now going to kill you. The defect is real. Talk to your illness maybe gets a little weird. I don't know what voice you would give it Voldemort voice or something How do I kill you? Why dispatch you? Dave one thing I'd note there too is I think there's a question and I get asked this often of like how do you Peter you asked earlier how do sectors evolve?

35:32Like I think actually the question of does decisioning of visual patient care change With Gen. AI is a much murkier question. I think the easier place to start and I think where in many ways would be very interesting is the U.S., as an example, spends about$13 ,000 to$14 ,000 per patient per capita on healthcare, right? Compared to$2 ,500 to$3 ,000 per capita in, say, Germany or Canada. Something like 30 % to 40 % of that is admin costs. And that is not admin costs that anyone wants to bear. And so this is something where I actually think the idea that LifetMD is pursuing is not to change the standard, actually to make the physician even more empowered, But to take all of the really painful admin and scheduling and make that the part that they don't have to deal with anymore.

36:19And AI should do a huge amount of damage in those areas. Exactly. What are you seeing most companies get wrong on their mission to implement AI? Yeah, I think it's a couple different things. I think the first one is a lack of focus on data as the starting point. So I do think the challenge, if you just tried to build an AI agent on fragmented customer and product data, it's going to break by definition, right? And so I think you do have to be in a place where the data you're going to feed into the models is clear and working. So I think that's been one major challenge. Do you think, I mean, if you had to look at companies as a whole in the medium and large size, do they have clean data?

37:07How long does it take a company to sort of get its data into a format and a level of fidelity that's useful? I mean, is this a hard lift or an easy lift? it depends if you take the paradigm of i'm going to put everything in a data lake and get everything right which can take five years and the reality is most big companies have spent a half decade trying to get all their major data schemas in order but i think if you start with the question of like what data do i need for this specific use case like um you know if you take um uh let's take credit underwriting. To do that well, you need a set of data around the credit itself, the market.

37:52You can probably have five to six core data variables you need, the core financials of the business, the security of the credit, all those kinds of pieces of core information. But you don't need every piece of data across the entire commercial bank to be right. You need the core elements for that use case. And so I think companies that are focused on the exact data they need to get right. I think they've done pretty well, but I do think trying to get all data, like, I mean, you've also seen the enterprise for a long time, Peter. If you asked any Fortune 1000 company to look at their full data repository and how much of it is accurate and working and clear and accessible right now, very few companies have that.

38:34So I do think being very tactical about what data you need. The other thing I think for Gen AI in particular is that a lot of the most important data is non-system of record, non-structured data. So it's things like images, videos, text files. It's just not things that people have tried to master historically. And so I think the first step in this is saying, what is the thing I'm trying to solve? And how do I make sure I have that data ready? Yeah. One thing I see a lot of, you know, had a long board meeting this morning, a company that's very AI forward, a portfolio accounting company called Vesmark.

39:07And the data, you know, for account reconciliation, for example, the data is abundant, but it doesn't tell you what the person actually does. You know, it just tells you how it was reconciled. So now the path to success is first the AI assistant, which helps accelerate you through the day, but it also knows what you're actually doing. Then that accumulates, then that becomes the RLHF or the training or tuning data. Because what you're trying to do is is like, what are you doing, guys? And that's not really represented in the data. But a lot of times you go talk to a bank or an insurance company and they're like, our data is our advantage.

39:43Go ahead, bomb it into the neural net and train it. Like, I don't even know what that means. You know, like, I'm just going to throw terabytes of spreadsheet data in and see what happens. Like, that's going to go clarn on you. Well, you have all sorts of other issues as well. I was talking to the CIO of one of the biggest banks in the world, and they have 300 different customer databases. Okay, 300. One for mortgages, one for loans, one for this. Because the mortgage people don't want to tell the loans people about their customer data, so they guard it jealously. It's a total disaster for the poor CIO.

40:16Fascinating. Alex? Yeah. I think these are all very interesting points. I'd like to, if I may be so bold, jump up several levels and maybe speak a little bit more about the business model of Invisible. My understanding, correct me if I'm wrong, Matt, is there's an element of the business, I think it's called Meridial, that is sort of a marketplace for ML freelancers, if I understand correctly. And I'm curious, I think in my mind, one of the many elephants in the room in this conversation is that we're arguably on the edge of recursive self-improvement. all of the frontier labs, more or less, I think, would agree with the assertion that we're nearing the point where you could have an AI researcher where you just turn over compute resources to the AI researcher, and the AI researcher does as good, if not a better job than the human AI researchers who work for the frontier labs.

41:11If that is indeed the case, surely one of the several elephants in this room, but given limited time, focus on this one, is that the need for a marketplace of ML freelance researchers to train models, doesn't that evaporate entirely as we start to reach the point where AI researchers can build custom models off of custom data sets and custom benchmarks for each client? Yeah. So as you said, we have two sides of our business, one side meridial, which we train all the large language models. And then on the enterprise side, we build basically custom applications for enterprises. Look, I think there has been a five-year evolution where I think consistently folks have said at some point you will not need reinforcement learning human feedback to validate and test models.

42:00And I think the challenge of that logic is a couple different things. One, the spectrum of expertise that if you take language, multimodality, um extreme expertise on things like computational biology and then the fact that a lot of these are reasoning tasks you do need and there's a whole host of studies on this that actually pairing synthetic and human data together is stronger but you do need human feedback on almost every different sort of agent you want to roll out and so i think the nature of rlhf is changing so i think you're moving more towards things like rl gyms controlled environments simulations i think you're starting to see things like much more of the expert work now is PhDs, masters.

42:44So it's less what I'd call commodity cat-dog, cat-dog labeling. But if you say tomorrow you're going to train a model to figure out different evolutions in 17th century French architecture in French, you are going to want RLHF to do that, to validate it. And I think that you're seeing that over and over is actually, as the models move more and more into very specific areas, there is more and more RLHF needed for them. That's interesting. Maybe I'll share my intuition and then would be curious to hear what you're seeing in your version of the ground truth. My intuition, my impression is that we're seeing greater and greater data efficiency in part.

43:27I mean, RLHF was obviously sort of very fashionable over the past three years. Maybe it went through sort of peak fashion, if you will. And then we saw the rise of reinforcement, fine-tuning, alternative mechanisms that maybe are far more data efficient and maybe even more human time efficient. If you have to just build an RL environment, arguably that's per human hour involved, probably a lot more time efficient than staffing out to some so-called developing country, folks to, as you say, cat-dog, cat-dog, do supervised fine-tuning or some other RLHF-type mechanism, surely, and I'm projecting, my intuition is that you'd see more data efficiency, not less, and therefore the amount of time, effort, money expended on RLHF or any sort of, even if we buy your assertion that we're seeing sort of hyper-parochialization of lots of different tasks, and each of them is going to need artisanal annotation.

44:31Surely there is a competing force, which is increasing data efficiency from algorithmic efficiencies like reinforcement, fine tuning. What are you seeing? Yeah, I mean, people have been arguing that for five years, but I think at least what I've seen on the ground is the accuracy that you want. If you think about a reasoning task that involves a several step leap and you think about the risk going to lose nations, it is more useful to have human feedback involved in that in some form. All right. And so I think, I don't think that that means if you think in some ways, RLHF happens after all the pre-training compute costs, it's a pretty small percentage of the total cost in training.

45:12And it is some of the most valuable feedback. And as you see, as you see more and more specific agents being trained for specific tasks, like take legal services as an example. If you train, if you get a new legal services data set, which is interesting, and you want to train a model off of that, you are going to want to see some sort of comparable equivalents, whether it's an associate or an M &A lawyer equivalent, where you actually test if it works. Now, is it possible that at some point 10, 15 years from now, you run out of things to train on? Possibly. But actually, I mean, if you take the number of languages, modalities, robotics is probably the next frontier of this in some ways, RL gyms, contact centers.

45:50There's a lot. We are, as a company, fully a believer. and I can talk about this in the enterprise side too, that human in the loop is going to be a feature, not a bug for a long, long time. And I think the entire red herring of the enterprise, for example, is that autonomous agents will do all of this with no humans in the loop. I actually think you're going to need more and more humans at every step. Alex, you're saying that the level of intelligence of these agents as we pass through AGI and get to ASI is such that they'll figure it all out as good as any human and be, and replace that human in the loop.

46:21What's your timing on that? That was exactly my question, Peter. So my timeline, if I had to spitball, of course this is not the predictions episode, so don't hold me to it. Hold me to my predictions in the next episode. My timelines are approximately two to three years for as a conservative outer bound for some elements of recursive self-improvement where we get our AI researcher that's as good, if not stronger than the human researchers for building ML models as a conservative outer bound. not 10 to 15 years, two to three years max. Yeah, that's the outer, outer edge. But I also believe Matt's totally right, that 2026 is going to be the year of recursive self-improvement and capabilities growing crazy exponentially and corporations moving at a snail's pace compared to what they could be doing.

47:09And it's all going to be stuck and bottlenecked and log jammed and it's going to frustrate the hell out of Google and OpenAI. And companies like Invisible are the lubricant that's going to actually get it from point A to point B. But that Klarnek use case is a really good one. In our tests for contact centers, 80 % of the people massively prefer the AI. But the 20 % that don't like it more than torture the whole thing to death and make it better to repeal the entire thing. There are probably eight ways to fix that quickly. But it's not going to come from Google, and it's not going to come from OpenAI.

47:47and it's going to involve data that isn't in the natural data set, you know, and it's right now, if you told me two years ago that everyone in the world will know what RLHF stands for, and there will be three people who are multi-billionaires who build RLHF companies walking around, be like, that's not even a thing. Oh, wait, now it's not only a thing, it's massive in scale. There'll be new terminology in 2026 for many, many of these other bottlenecks that, yeah, the AI can do it, but for whatever reason, the bank is not doing it. The contact center is not doing it. And those bottlenecks are going to be so lucrative for companies like Invisible to just plow them down.

48:30I can't answer the specific question of whether your workforce is going to involve the distributed workforce that you just described. What was it called, Alex or Matt? It's called Meridial. Meridial, yeah. So there is a really healthy debate on whether is meridial a key part of this or a network of even more agents a key part of this? Or is it, you know, is 2026 the transition year between the two? It's going to be a really interesting foot race between those two different approaches. But that's really what I'm that really is. I think you put your finger on it, Dave. That really is what I'm asking, which I think is a distinct question from is there value in supervised fine tuning or reinforcement learning with human feedback going forward?

49:10Of course there is. What I'm really asking is how much of that can come from AI sort of bootstrapping it in the near term future versus needing human inputs. And what I'm saying is think about a balance between generalizability and hyper specificity. And I agree with you on generalizability. I actually don't think RLHF is important even now for that. But where it gets more complicated is when you want to train off a specific task. So let's take the insurance claim example that I mentioned earlier, right? you're going to generate a 10-page insurance claim. And you could apply this to any enterprise use case, many consumer use cases.

49:46But in that world, you build an LLM, it's producing an outcome, and it's fine-tuned off a specific company's data. But you need a way to actually say at that point, does this produce a comparable output to what that claim did, to what a human doing this task before was doing? And so when I mentioned earlier custom benchmarks, that is the process by which you do that, is you actually do need human equivalence testing. You need a human to provide a comparable data set and to say this looks good or it doesn't. And you just don't have precedent data to train that off of in any CLM because the human input is not.

50:23Now, again, that's going to keep going down more and more specific tasks. If you take legal services, take it by language, take it by topic, take it by document type, there's human feedback required for all of that. I almost, I mean, not to put too fine a point on it, but I want to make sure that those in this episode who want to drink the bitter pill with the bitter glass of water for the bitter lesson are so drinking. I'm curious, Matt, to understand how you see this. Surely there's a wave of generalism that is over time, and maybe we can sort of finesse what the appropriate time scale is. It sounds like maybe your so-called timelines are a little bit longer than perhaps mine.

51:11But would you at least agree with the premise that over time, even the specialized skills end up getting subsumed by generalist models? Or do you think that's just never going to happen? We'll always, or by always, I mean like on timescales of 10 to 15 years, which is a pretty long timescale. We're just going to have generalist models that are always sort of like specially fine tuned. I don't think all expertise, all specialized expertise right now. I mean, again, if you think about a lot of that, a lot of the information that specific experts have, there's no training data available for that.

51:45Like it's stuff that sits in people's head. It's experience. I mean, again, I'm aware of many of the narratives that human expertise becomes less important. Again, we are a company that actually thinks the human touch elements become more and more important. But take sales, for example. Many of the best selling patterns, many of the people who've done that the best, there is no information you can train off of what they do. Human interaction, actually, in a world where there are 500 companies selling email-based SDRs. I think human beings become more important in that world. So I don't actually think special.

52:20I actually think that the shifts are expertise becomes more and more important in many different areas. I think human loop stays really important. But I think the I mean, if you take a contact center and I understand that the theory of what you're saying, but like we're four or five years into this. And if you look at the number of U.S. contact centers that have migrated to using agents, it's a pretty small percentage. Can I ask you actually? Yeah. The Jane Street question is really burning a hole in my. So it's really clear that stock picking is moving to AI at warp speed. And the reason is because there are no barriers.

52:52Like you're just placing a trade that's already automated. So it's like, and that's the middle of it. And there's a great benchmark. More money. More money. And also almost all of the volume on public equities markets has long since been dominated by algos. So this happened decades ago. Yeah, well, it started with rapid trading. So the quants were already there. So now that it's moving to fundamental analysis, it's the same mindset. So that's one of the reasons it's just taking off. But like Peter said, you're making more money. Okay, let's just keep going then. There's nobody who's saying, but I'm going to lose my job.

53:23It's like, no, we'll just pay you more. Let's just go. So it's really an interesting bellwether. But within that world, they're struggling because the data is so proprietary. And it's looking more and more likely that these self-improving massive foundation models are going to get to superhuman IQ this year, this year being 2026. But the prompt window is getting massive and the recursive chain of thought reasoning is getting really, really good. So you can actually feed it data without having to retrain it and have it achieve the job. So if I take that mindset from Jane Street and I move it over, now I'm a mechanic and I'm trying to fix a car and trying to diagnose what's wrong with it.

54:04And I have audio and I have sensor data. Great, easy use case. But am I going to then put that data into the LLM API and transmit it to OpenAI where they can accumulate it? And then if they decide later they want to be a garage, they have all my data? Or am I going to run some kind of a walled off model? And, you know, garage mechanics may be not the best example. That's why I chose Jane Street, because they're never going to take their proprietary data and give it to OpenAI. But in the middle ground, you have banks, insurance companies, you know, hospitals, like, how are they going to deal with this?

54:38Like, it's easy now, like sometime in 2026, it becomes easy. But the data is, is that's my only reason for having a competitive advantage. I don't want to give it like over the to the API. Yeah, look, I think you're seeing there are definitely sectors, many of which you just named banking, healthcare, where people are deciding to keep their data on premise, or they're using things like small language models for those sorts of reasons. And I think you may continue to see that as a trend. I think one mistake folks often make is not all data is proprietary. So you can have and take the James Street case, maybe their trading data is proprietary, but their back office kind of forecasting data might not be, or back office finance data might not be.

55:22And so I think one thing is being clear about the data that you need to keep proprietary and that you do want to take more parameters of security around, and then what data you say, look, this is actually, I'm going to be very careful as a company, but this is data that is not as proprietary. I think that sort of balance, I think the whole, you know, similar to what we discussed with contact centers, the idea of I will not give anything to the LM, but I'll keep it all in-house. I don't think that makes sense either, but I do think that's a paradigm you're seeing more and more. I think, yeah. Yeah.

55:54I want to kind of change the tack a bit, if that's okay. I actually do agree that we'll automate, but I think we'll automate in a way that's different from this discussion. So let me give an example. Let's say I'm Canon printers and I'm selling home printers, right? Right now, I have a bunch of people doing marketing and content development, brand management, then salespeople to sell to the best buys and so on, and online folks. Then you have post-purchase, getting the customer to try and register the dang printer. And then you've got all the repair support, technical staff. And then you've got your accounting folks in the company.

56:33You could get a job again. So you've got pockets of people doing different functions across the board. If I was going to build an AI native printer sales company, then I might think about having all of those things automated completely with AI. and then you're not human centric but you're function centric across those the printer could report when it's you know running out of ink and you ship it a new thing you uh it tells you when there's a problem with it or there's a problem coming up you you alert your your repair staff saying hey this guy maybe we can upsell him a printer and you essentially automate all the functionality with ai and you leave the human 90 out of the loop almost completely because you've automated the core functionality.

57:19And right now what I'm seeing is what I used to call radio over TV, right? When you first had television, we took radio announcers, put them on TV to read radio scripts. Okay. We didn't adapt for the medium. And I think what I'm seeing right now is we're automating right now what the human being is doing at each of those functions. But surely over time, we're going to automate the functional flow and then get rid of the human beings completely. AI native, AI first, right? Not to mention getting rid of the printers. Well, that's a separate question. I'm not just using that example. Who's going to be doing any of the produce?

57:53Let's leave that part aside just for the moment. I think you're absolutely right, Salim. I mean, this is where a young AI native company reimagines an entire field and has zero legacy and zero friction in coming forward. The question, as Matt said in the beginning, is do they have the distribution, right? But this is where a large company, Canon in this case, should actually be investing in entrepreneurs. I mean, one of the things that you and I talk about a lot of times is if I'm a large company and I don't know what to do, I would basically hold a competition, ask young AI entrepreneurs around the world to come forward and how would you disrupt my company?

58:35Give me a pitch. And then I would pick the best five of them and I would fund them. And I would say, you know, we're going to fund you to disrupt us. And then, you know, we're going to give you access to our data, to everything we have. And then ultimately, we're going to buy you or buy a majority stake in you. And we're going to make you our new company, right? This is the innovation on the edge, the displacement of the core, etc., however you want to call it. This episode is brought to you by Blitzy, autonomous software development with infinite code context. Blitzy uses thousands of specialized AI agents that think for hours to understand enterprise-scale code bases with millions of lines of code.

59:17Engineers start every development sprint with the Blitzy platform, bringing in their development requirements. The Blitzy platform provides a plan, then generates and precompiles code for each task. Blitzy delivers 80 % or more of the development work autonomously, while providing a guide for the final 20 % of human development work required to complete the sprint. Enterprises are achieving a 5x engineering velocity increase when incorporating Blitzy as their pre-IDE development tool, pairing it with their coding co-pilot of choice to bring an AI-native SDLC into their org. Ready to 5X your engineering velocity?

59:57Visit blitzy.com to schedule a demo and start building with Blitzy today.

1:00:04You're a medium-sized or a large-sized company. I'm not going to focus on the startup right now. And what do you do in 2026? Because you're going to have to do something. You're going to have pressure from your board, from your shareholders, from just competition. So you've got to do something. And what I heard you say so far, Matt, is number one, you got to get clean data. You need to make sure you understand what your data situation is. Number two, you should pick two or three, if you would, areas, call them benchmarks, where you're going to run experiments on. And it's not a proposal. It's not an idea.

1:00:44It's actually run it. Actually do run an experiment to see how it works. um what else um and then poor scale poor money on the things that do work and then have an expanding sort of uh increasing circumference around the company's major revenue engines how do you think about that walk us through a few more steps yeah so i think one of the things which has been a lot of the the topic of conversation here was given given all the improvements in the models given um you know it's something was walking through on the potential to clean sheet and design a company from scratch, why has, you know, there was this MIT report that came out that 5 % of enterprise models make right now make it to production, right?

1:01:27So I think there's a starting question of given all this tech excitement, why has that been so much harder? And it's not the technical challenges that we've talked about. It's the data. It's the focus on which priorities to look at. I think the other two big ones though, are the organizational structure by which you pursue those initiatives. And particularly the advice I give everyone is do not locate this in your technology organization. Take your best operator, your best ops person, give them an operational KPI and track it to that and make sure it's a really clear operational KPI. So we talked a bunch about contact centers.

1:02:01You should have an operational person there, lead it around, you know, CSAT score, time per call, whatever the core metrics you're looking at. And that should be your guide. If you want to take something like inventory forecasting, You should do it around inventory days, stockouts, all those kind of metrics. But I think if you have a clear sense of which operational person is leading it and how they're marshaling resources around it, and you have a clear KPI, you're going to make progress if you focus on a couple different things. I think the failure mode on that has been you let a thousand flowers bloom, none of them have an operational metric, and you kind of end up with a science project dynamic.

1:02:38Yes, exactly. That's exactly right. If you walk in, a thousand flowers bloom, you walk in and you say, I am going to give you a million genius level people for free. Do something. It fails. It's like, here's a million people for free. And they're all geniuses. And it fails for that same reason. It's like, I didn't think of an idea. So I said a thousand flowers, just go bloom. I couldn't think of anything. So maybe you will. Like, how's that going to work? I've seen that. You're exactly right. It's just so sad. We go even further. We basically say not just take the operator and put them outside the organization and let them build something from scratch at the edge.

1:03:22Because otherwise you get encumbered by all the internal rules and bureaucracies, and that gets slowed down for a huge amount. Then it fails for legacy reasons. It's the Lockheed Skunk Works. It's the Apple MacBook team. Yeah. I mean, look, Apple is actually a master at this. if you think about what Apple will do, is they will form a small team that's very disruptive. They will put them at the edge of the company. They'll keep them secret and stealth. And they'll say to them, go disrupt another industry, right? Whether it's watches or retail or whatever. At GlassCon, I think they have 18 teams looking at different industries, thinking about it.

1:03:56And when they think it's ready to disrupt, they go into it and they patiently iterate, right? The Apple Watch, for example. So this is the model I think we're going to see many other companies take on, where you do this. Because if you think of any operational company, the insights they have on all sorts of adjacent industries is incredible. Very hard to disrupt in their own industry because they're probably pretty optimized for it unless you come with the AI startup. But they can really disrupt a lot of the edge cases, a lot of the industries around them. So I expect them to launch AI-native startups that go into adjacent industries and go attack some of their neighbors.

1:04:32Nice. Matt, before we get to a few of your 2026 predictions, Can you just share a couple more of the use cases here just because they're fun? So we worked with SAIC, Vantor, and the U.S. Navy on building intelligence for underwater drone swarm for unmanned underwater vehicles. So think of that as if you have a series of drones and you have enormous numbers of sensors on each of those drones, and you need to understand the movement patterns of those different drones. And in each case, you see an object underwater. What do you do? Do you engage? Do you step back? Do you move with other drones? That whole movement pattern and decisioning for underwater unmanned vehicles, that's what we worked on, fine-tuning a model to do that, training it, looking at all the movement pattern data.

1:05:17And again, this is one of the most interesting things about drones is they are autonomous. And so thinking about how those movement patterns evolve in complex environments is very hard to do, but you also have lots and lots of interesting sensor data to do that. But I think one that maybe anchors more on the human decisioning side is Swiss gear. So like Swiss Army, the luggage brand. Similarly, and I actually think this is, Peter, one that a lot of folks in the audience may relate to in some form, which is they had an enormous mix of different data tables around products, customers, et cetera, that they couldn't really bring together for inventory forecasting.

1:05:51And so we used our data platform, Neuron, to bring together 750 tables really quickly and then optimize the forecasting to look at both minimizing stockouts and optimizing which inventory to hold. If you get inventory forecasting right, it's probably one of the major issues for most big and small businesses. You minimize lost revenue. You make sure that you don't hold lots of excess inventory. It's one of the hardest things to do, particularly if you've got a six to eight month order cycle time. And so that was something we partnered with them on and I think was a great outcome. We ended up expanding their overall inventory coverage by about 30 % and basically 2x the numbers of SKUs as a reliable prediction.

1:06:37And again, that was done in a couple of months. All right. So later this week, my moonshot mates and I are recording our 2026 predictions. We'll have Emad back and we'll be talking. Each of us will provide two predictions for 2026. we'll have our top 10 from the Moonshots podcast. It's going to be fun. It's going to be a battle. We're going to ask our listeners to vote on which predictions they like best. I mean, of course, they're all going to vote for Alex's, but hey, Matt, talk to us about what you see coming in 2026. Yeah, I think I'll call out a couple and we've just done a bunch of research on kind of our 2026 predictions.

1:07:14So I won't say all of them, but I'll call out a couple. I think one of the first ones I would anchor on is multi-agent teams. So I think one of the challenges, and it's inherent in a lot of what we discussed here, is if you're a large enterprise or medium-sized company implementing a use case, you won't necessarily have one decisioning agent that does everything. You'll train task-specific agents for individual tasks, usually orchestrated by an LLM. And what that allows you to do is to pinpoint the accuracy on those specific tasks and then use the broader logic set of the LM to make sure they all work together properly.

1:07:51And I think that's been an architecture that's been discussed pretty broadly for a while. But I think that we're just starting to see the green shoots of more and more folks having success with that. Contact centers being a good example. So I think that's a big one that I would call out. I think the other the second one I'll call out is the multimodal leap. I think more and more video images, audio are going to become a bigger and bigger part of how people engage with these models. So I think audio probably one of the most interesting. And so I do think the way you'll be able to speak to them, interact with them, visualize them is going to be a really interesting moment for 2026.

1:08:27And I don't think that will all be text-based like it has predominantly historically. And then maybe one other. Yeah, please. No, I was going to ask Alex for feedback. Go ahead, but finish up, Matt. Yeah. So the third one I'll call out, because we've talked about it a couple of on this episode, is kind of what we call either the mirror world or RL gyms. So I don't actually think that's a well understood concept for many folks in the audience. But think of that as actually creating simulated environments or digital twins for tasks you might want to test. Right. So maybe that's a coding environment.

1:09:02Maybe that's a contact center. We've used that a couple of times, but it allows you to actually simulate a series of function calls, tasks or environments by which if you're going to train a model or a task, you can actually test how it's going to work like a manufacturing environment before you roll it out to your actual physical world. And I think that's more and more in both model builders and the enterprise. What we're seeing is a very interesting topic. I want to go around to the mates one second, maybe ask some final questions of Matt. Alex, you want to kick us off? Yeah, I think the most interesting crux of what we're discussing here is what is the future of human expertise?

1:09:41For that matter, does human expertise have a future? And assuming it does, what's the half-life? It's cooked. What's the half-life of the value of human expertise. And so to put that in question for Matt, what do you think of all of the forms of human expertise, of all of the labor categories and job roles that exist in the economy today, what do you think will be the last three of those job roles or forms of expertise that will disappear or ultimately succumb to AI? What are the last three to survive? Last expert standing. Okay. That's right. I mean, I'll go back to where I started the episode.

1:10:20I think a lot of the commentary on mass shifts ignores the actual function of jobs in society today. So if I go, let's go, let's take sectors, for example, oil and gas, a lot of the functional expertise, you know, geoseismic, if you look at seismic engineers, people on oil and gas sites drilling, that is a human function, like you do need. So I think real estate as another example, like, humans actually help select which you can go down a whole list of different areas. I think there are sectors where you're going to see more disruption near term. I call that a couple of them, BPOs, legal services.

1:10:59I think media is a fast changing area. But I'm also not exactly sure that those lead to negative, meaning have negative employment consequences. If you take media, it's a really interesting one. Five, six years ago, eight years ago, I think media as a category really struggled in a lot of ways for paid media as an example, right? And you've actually now seen in the last couple of years, post-Yellom era, Substack, Medium, all these blogs become much more interesting. You have way more media entrepreneurs. And so you've changed the function of society and where the money is coming from changes, but it has not changed total employment.

1:11:35And look, I understand a lot of skepticism that says that AI is going to radically change everything. But I think if you look at an American society for the last hundred years, it's something like 25 % of every high school class goes into a field that did not exist when they were in high school. And the reason that persists is people go into the working world, understanding the tools they have, thinking about what they can create from that. And one of my favorite statistics, the Wall Street Journal reported last couple weeks ago is 20 % of U.S. employment right now is digital ecosystem jobs. And something like 9%, 9 % of U.S.

1:12:19citizens are full-time social media influencers, which is mind-boggling to me. But again, this is the changing nature of work. And so I think that pattern will persist. I think that the core of what will change is the process of looking up information across multiple systems and documents. That is going to become less valuable. But I think all the jobs that involve human interaction, physical work, physical – I actually think one of the most interesting things over the next couple of years is the job ecosystem around data centers, electricians, etc., is going to become way more in demand. I was actually sitting there.

1:12:59I was on a panel with someone who runs a recruiting company. They were saying that job profile, I think, will 2, 3, 4x over the next couple of years. And so that will have pretty interesting implications for the education system and everything else. But I think we will see an evolution. I meant the humanoid robot, electrician, and plumber. Alex, very quickly, what are your three last standing human roles here? It's interesting. So I'll present multiple competing hypotheses. So hypothesis one. But briefly. Briefly. One hypothesis is it's the politician because they helped to make the laws. Yeah, that's true.

1:13:40Another hypothesis is that it's the greatest intellects, the physicists or mathematicians, even though as we talk on the pod, math and the sciences are all getting solved on the one hand. There's still perhaps to the extent that that represents the culmination of human intellectual accomplishment, maybe the greatest intellects will be the last to be automated. There's another school of thought that says, no, it's the roles that involve the greatest need for human authenticity, because even though it's not actually a capabilities question, people nonetheless demand human contact or something to that effect.

1:14:16And so it's going to be the highest touch job roles where people just want to know that there's a human counterparty on the other side of the interaction. So that's a set of three hypotheses. tastemakers will dominate. That's authenticity, bucket numbers. That's what Mike Saylor said, word for word, actually. Yeah, on his boat, we had that enjoyable sunset conversation. Thank you, Alex. Salim, do you want to go next on a closing question for Matt? I think you covered some of it on the industries that are kind of going after. You guys have done some government work. Where in government functionality do you see the biggest opportunity for AI automation efficiency, et cetera?

1:15:02Yeah. Everywhere. Look, I actually think this could be one of the really positive trends for society. So I saw a study recently that AI-assisted permitting could cut energy and data center project implementation timelines by 50%. Think about housing. One of the biggest challenges right now for housing development in the U.S. is NIMBY regulations and how complex it is to build housing because of the myriad of different regulations and zoning contracts by location. Or even the OECD came out with this report that AI could shrink public sector process cycle timelines by 70 % on licensing, benefits, approvals, compliance, and basically accelerating infrastructure deployments.

1:15:51So to me, the simplest thing that AI can do is project management and timelines related to all spending and infrastructure deployments. This would be a really positive thing for society in my mind. Amazing. Good question, Salim. Dave, why don't you close us out on the questions here? Oh, I got so many, but I'll pick the best. First, Matt, how many hours of video footage will there be of you one year from today compared to one year ago? Because I know we saw each other in Riyadh a few weeks ago. And I know that you are the thought leader in this whole bottleneck of AI getting into the enterprise.

1:16:29It feels like what we're doing right now. You know, the footage of you that's out there right now is all this bullshit CNBC, Bloomberg type, you know, five minute format. But here we're getting your real thoughts. It's just so much better. But how many hours can we count on a year from today? Well, look, I think as of 12 months ago, I had done almost no interviews of any kind. So this job has been fun in that front. And look, what I enjoy about the podcast format is it does allow you to talk about some of the more complex topics. And so, you know, particularly a podcast like this, that's really interesting.

1:16:57So hopefully many more in the year to come. Well, I'm hoping for at least a 10x on that. And then my follow up question to that is the avatar version of you that's also out there talking. Is that a 2026 thing, you think, or when? Yeah, that's probably happened since 2026. I don't think it'd be that hard to train an avatar off of my public statement. So, you know, I think that'll be an interesting, we are actually working in the sports space, actually, on the topic of avatar training. And I think it is actually an interesting space where you could imagine a lot of different areas where rather than a chatbot interaction, people want to speak to people they know via an avatar.

1:17:35I actually think that will become a more natural part of society and a pretty interesting one, actually. I totally agree. The timeline could be as soon as two months, as far as I'm concerned. What makes you think it's not an avatar we're speaking to right now, Dave? That's a good question. That seems very human, actually. I don't know. The best ones are. But the green orbs behind you kind of give it away. Yeah, they are pretty strange. That's not real. Matt, where do people find you? Where do people find Invisible? Who should go to Invisible to check out what you do and how you do it? Sure. So we have seven offices now.

1:18:12New York, San Francisco, Austin, Texas, D.C., London, Poland, and Paris. I'm the easiest to find probably. we have an office right off of Union Square, which is where I'm at least half the time when I'm not on the road. And look, I think in terms of who should come to us, from the listener base in particular, any mid-cap or enterprise company that knows there is potential in their business, that knows that AI can transform in a positive way and is struggling to bring all the pieces together, I think that is the main thing I would say is there is no doubt, everything Alex is asking the technology has made an enormous step change over the last couple years the hard thing is actually the change management the operationalization the metric tracking the evaluation it's kind of bringing together like you know i think it's the difference between the our founder francis has a uh an idea of you have all the components to build a cake but you don't have a cake like what we do is we actually bake the cake again we build you something that works we make it that amazing and the website uh invisibletech.com all right thank you matt salim dave uh awg i'm going to see you guys in a couple of days for our 2026 predictions um make them brilliant it's going to be it's going to be fun uh all right i want a benchmark for trapping tracking benchmarks that's your all right all right no that's not the one i'm gonna talk about okay all right guys have a great day every week my team and i study the top 10 technology meta trends that will transform industries over the decade ahead i cover trends ranging from humanoid robotics agi and quantum computing to transport energy longevity and more there's no fluff only the most important stuff that matters that impacts our lives our companies and our careers if you want me to share these MetaTrends with you.

1:20:06I write a newsletter twice a week, sending it out as a short two-minute read via email. And if you want to discover the most important MetaTrends 10 years before anyone else, this reports for you. Readers include founders and CEOs from the world's most disruptive companies and entrepreneurs building the world's most disruptive tech. It's not for you if you don't want to be informed about what's coming, why it matters, and how you can benefit from it. To subscribe for free, go to demandus.com slash metatrends to gain access to the trends 10 years before anyone else. Alright, now back to this episode.

1:20:59Angebote unter mastercard.com slash businessbonus.