971: 90% of The World’s Data is Private; Lin Qiao’s Fireworks AI is Unlocking It

0:00Jon Krohn:Over 90 % of the world's intelligence is locked inside private enterprise data that no foundation model has ever seen. Today's guest is on a mission to unlock it. Welcome to episode number 971 of the Super Data Science Podcast. I'm your host, Jon Krohn. Today's guest, Lin Qiao, is the CEO of Fireworks AI, a Bay Area startup that has raised over$300 million to unlock the world's vast quantities of enterprise data for LLM training and inference, revolutionizing capabilities and performance. With a PhD in computer science from UC Santa Barbara and years of experience as a director of engineering at Meta, Lin is now a highly successful technical founder with a rich perspective on AI today and what the future holds for all of us.

0:45Jon Krohn:Enjoy this one. This episode of Super Data Science is made possible by Dell, Intel, Cisco, and Excel data.

0:55Jon Krohn:Lin, welcome to the Super Data Science Podcast. It's an honor to have you take time out of your busy schedule to be on the show. How are you doing today? I'm doing great. Thanks for having me, John. Of course. And where are you calling in from? I'm calling from Portola Valley. That's where I live. Nice. It's part of the Bay Area. Yeah, yeah, yeah. Palo Alto. Yeah, very close to Stanford. Nice, nice, nice. So we're here to talk about Fireworks AI, your business, which has done incredibly well. I mean, you've just grown so quickly. I believe you've now raised over$300 million in venture capital, including a recent$250 million Series C, if I got that correctly.

1:38Right.

1:39Jon Krohn:And so it's a platform built around open source model deployment at scale. And the Fireworks AI platform is built around open source model deployment at scale and this idea of autonomous intelligence. Tell us what that means, Lin. Yeah, sure. You're right. We raised our last round last year and we are growing really fast. So our mission is autonomous intelligence. This mission is very complementary to AGI, where the direction of AGI focuses on investing in directing a lot of intelligence into this one model and have this model be able to solve various different kinds of tasks in a great way. So the idea is you just build your application on top of the AGI model as a utility.

2:32So this is a great, AGI is a great interaction. It's very scalable if it's successful. But the reality is only a very small fraction of data goes into the foundation models for AGI. If you look at the worst data, majority of the data, by majority, I really mean like more than 90 % of the data is actually not in the public domain. It's not in public internet. It's not labeled by labeling companies, which goes into the foundation models. And majority of those data are private data locked inside applications and enterprises. And data, we all know data is intelligence. Data is knowledge. And those application-specific, enterprise-specific data is not accessible by the AGI labs.

3:32And we just leave a lot of intelligence on the table. My prediction, and that's where we're betting on the future, is to be able to activate those private data and let the model absorb additional application-specific intelligence and bring the model to the next level. And this kind of motion is more like customization, right? It is the model and the inference deployment were customized towards applications in their specific pattern. And this customization should not be just one time, right? our application enterprise product, it keeps evolving. So this customization should be continuous. And ideally, this continuous customization should be fully automated.

4:23That's autonomous intelligence. We are making great progress towards that direction. And we believe the future is not one model result. It's going to be millions of models, one per application per use case. All right.

4:39Jon Krohn:So the term autonomous intelligence, it sounds kind of vaguely to me like the buzzword of 2025 in our field, probably the buzzword for 2026, which is agentic AI. But it sounds like it's quite different from agentic AI, this idea of autonomous intelligence. It's heavily connected with agentic AI. So think about agent as a way to automate many of our day-to-day tasks. So we have been living the world that many expert intense tasks have been gradually automated so we can free up our time. Eventually, some of the even professions will be redefined. So, for example, I think there are interview agents or hiring agents where you give a job listing, it will source the candidates and even do the first rounds of filtering and interview for you.

5:37And there's marketing agent, you give your ICP list and will source the right company, right stakeholders, and start to drive customized outbound emails and riches. and their customer service agent just give the human agent some really good assistance to kind of be smart. And there are so many like agents who are doctors and so on. So this is happening, transforming our day-to-day life. But similarly, another big transformation that's happening is in my domain, software development is being disrupted. And today, a coding agent can really start to behave like a junior engineer. And I'm not kidding.

6:29This is kind of really happening. And it's actually changed our interview process. And the fundamental question we're asking ourselves is coding skills Coding interview is important anymore. So coding interview in the past is going to be replaced by how good you are at using coding agents. So it is actually happening across our day to day life. Now, let's go back to this autonomous intelligence era. So without that, currently, this work of continuously adapting the model and changing the customizing inference setup is done by a very, very small set of experts. So those experts are like they have been doing AI system for a long time.

7:30They have been researchers for a long time, accumulated their knowledge over years. So only a few companies who have those strong density talent pool are able to do that. And the question is, can that part be automated? Right? And similar to others that has been disrupted and has been reshaped, and can this part of doing product model co-design and infusing more intelligence to the model and making the inference serving tier much faster and much more efficient, can that part be accessible by a wide range of application developers without them putting in a lot of work and carrying the burden of learning all the deep knowledges.

8:27So that's what that means. So we heard a lot about AI is going to free up a lot of human labor, right? So, and this wave is interesting because it will start from a different angle. So it will actually free up a human from the high intelligence level, not from the physical level, right? The robotics is going to disrupt the physical level of engagement, but AI is going to free up a lot of kind of high intelligence level of the tasks and work. So that's kind of interesting change. And we are also innovating and disrupting in that space from a baseline platform space.

9:16Jon Krohn:Really cool. So it sounds like, yeah, autonomous intelligence builds on agentic things, but also involves lots of things not associated with agentic, like systems, like models automatically retraining and the whole kind of system humming along nicely. it seems like a key part of that working for you, especially in an earlier answer, you mentioned how you see the future as millions of different models. These could be like LLMs, but millions of different AI models that are tuned to specific tasks within specific enterprises. And it sounds like Fireworks offers a reinforcement fine tuning product that allows your customers to beat frontier closed models in under a month on specific tasks that that relatively small, tuned, open model is fine-tuned for?

10:13Right. So we are very bullish in this direction. So think about AI has been following a lot of footsteps of human intelligence, right? Artificial intelligence has been following that full step. Even the model architecture is called neural networks. It's kind of emulating like the human's brain, right? So for reinforced learning, it is actually very similar to we, how we human learn knowledges. So we learn by various different angles. So one of the angles is we get positive feedback that we know, oh, this is the correct thing to do by learning the principles. Or we get negative feedback and it's kind of we get penalized for doing something bad from our behavior point of view.

11:06Then we learn, no, don't do it anymore. And we kind of change to a different direction. So this also happened to how we do agriculture, for example. As, you know, I'm a fruit lover and, you know, today we all like to eat like sweet fruit. The fruit also grows much bigger. But this is not how originally it became, right? It's gone through multiple generations of selection process where we collect the seeds of the sweeter and bigger fruits in the planet and among those and collect another. So this is kind of another way of reinforcement learning, reinforcement seed collection selection process. So a lot of things we do, whether for ourselves, our own learning process, or we have applied in other domains, is following the same principle.

12:08And this is also similar to how model learn. Is you teach the model what is a positive reward. You teach the model what is a negative reward. And the model is going to automatically scan through a search space of possibilities and do these feedbacks and find a path to specialize self in solving certain kind of problems really, really well. With that said, it's not all like only benefits, right? It's a trade-off. Because if you let the model specialize in a certain direction in the area, it's going to be less specialized in other areas. So it's like us exactly. As John, you're specializing in driving this podcast and you're a great host.

12:53You're very knowledgeable of how to engage with guests. And I specialize in building the best AI platform which can customize towards application-specific patterns and so on. But I'm not a good chef or cook, and I don't know actually how to do gardening very well. So that's kind of a natural selection we have to direct our attention to similar to the models. So that's kind of where we are batting on this direction to make a reinforcement learning for models very, very accessible to all application developers. So they can basically have a model in tune to their product all the time. And imagine that.

13:53Imagine a model is just constant learning the intelligence from your application. And then you have your private model. And then you have your mode. where nobody else built on top of an existing API would have. So this is a special thing that we want to kind of have every application developer to get a hold of.

14:33Jon Krohn:has focused on scaling AI vertically, bigger models, more compute. Those breakthroughs matter, but intelligence also scales horizontally. Agents sharing knowledge across a network, coordinating on common intent, reasoning together. The infrastructure for that second horizontal axis doesn't exist yet. Outshift by Cisco is formalizing it. They call it the internet of cognition. They're publishing the architecture and building reference implementations. Read Scaling Out Superintelligence. We've got a link to that in the show notes. Then check out episode number 961. In it, Dr. Vijoy Pandey, the head of Outshift by Cisco, walks through how horizontal scaling of intelligence works and why it matters.

15:13Jon Krohn:Right, I see. So all of that private data that you mentioned at the outset of the episode, that accounts for most of the data in the world, enterprises can be using their particular private data to be creating a moat by not only having that private data, but also having these fine-tuned models specifically specializing in particular aspects of their data applied to specific tasks. Yep. And this is another new thing, actually. For example, doing the mobile first move. So before AI, the biggest shift is mobile. So the application moves from desktop to mobile devices, and that actually opens up a whole new domain of doing product development, which is similar to autonomous intelligence we're heading towards in terms of thinking.

16:13On mobile first, I think one thing that opens up is the access of end consumers to an application. Because the people who own a desktop versus people who own a phone, it is orders of magnitude different. And that just means now your product could have reached to orders of magnitude higher group of people. And it will go global much quickly and you will be able to access various different demographics of the cohort much wider. It changed how app developers think about product evaluation because now it's much broader and you cannot just deploy a group of PMs to understand what the customer wants.

17:04And the product no longer become monopoly, just one design, because for different cohort coverage, you may want to highlight one feature versus the other. So then it becomes, oh, how do we even do product development? How do we incorporate that feedback? It has to be customized. And that customization is being done through a new technology called AP testing. So the idea here is to use the insights from your product to compare A versus B of the product feature and make statistical decision. Based on statistical result, make the decision where your product should look like within certain cohort of groups.

17:51So the idea is It's basically similar. Like product has a lot of intelligence. We need to leverage that intelligence to make the product better. And here similarly, product has a lot of intelligence. We need to leverage intelligence to make the model better for your product. Therefore, your product built on top of a specialized model will be better. So right now it's a completely open space. It is a vacuum and there's no existing solution that has been solving this problem really well. I'm pretty sure industry will move towards looking at this space very closely. And I firmly believe there's a lot of value in there.

18:37Jon Krohn:Awesome. I love it. Yeah, really exciting. You're kind of defining a new category, which is an exciting thing to be doing. Um, so when people are trying to figure out these, you know, these relatively small, say LLMs for a particular task, does model selection really matter? I mean, do your, are your clients like picking, oh, okay, I'm going to use this llama model of this size or Quinn, or do they make those decisions or is this something that's kind of like handled automatically by the autonomous intelligence system? So model selection matters, but it's also exhausting. So, funny thing, there are two interesting phenomena that are so unique to AI.

19:28One is the model depreciation is very fast. As you can observe, every couple of weeks, there's a new model launch. While this code is open, someone topped the leaderboard and then a couple of weeks later, someone else top the leaderboard. And we often, they all are strong in different ways. So we start to see the researcher focus start to diverge, right? It's clear some labs are really good at chat-based models. Some labs are really good at coding to use agentic models. Some labs are really good at multimodality models. Some are really good at long contests. They all start to diverge into focus on and specialize in different areas.

20:20Speaking about specialization. So different use case will suit different kind of model specialty very well. But it's really hard for people to figure out which one is the best for my use case. and my use case will also evolve over time. And the public benchmark result is fully saturated and we need to figure out how to pick and choose continuously, which is exhausting. So we do help our customer figure that out. But at the same time, there's another level of complexity. It's hardware depreciation is also very fast. This is something new. Before this wave, usually every three years, there's a new hardware skill.

21:11And now, last year, myself, NVIDIA launches three skills, three new skills. And in 2026, there will be a lot more new skills from all different vendors, whether it's GPU or customer ASIC. So then how to manage hardware becomes a very hard problem. And these two combined is causing so much headache to the application developer who want to stay on top of all the different kind of wave. It's just too much deep knowledge and expertise to gain in order to kind of pick the best for them. So we are the platform that we abstract out. We eventually want to abstract out hardware. We want to abstract like hardware selection.

22:03and we want to kind of provide the best model for various different use cases. And so basically there are various different kind of mapping from specific use case, specific workload patterns to the model, to the hardware, to the inference setup, to the fleet design. So all of that requires a big team in-house where deep experts, which is really hard to find. And we want to kind of make it super easy for our customers.

22:40Jon Krohn:Right. So instead of needing to find the deep experts, they can just come to Fireworks and work with you guys, work with your solutions. Absolutely. And interestingly, there are... So last year by Solve, I haven't counted. Every month, there are a few new models get deployed and launched. and it has been very exciting. But also, I would say early on, we bet on open models. That was a big debate in the company that we debate on many what-ifs, right? But because of our background in PyTorch, which is an open source project, PyTorch is now, we built PyTorch from the ground up. It is now the dominant AI framework.

23:27we firmly believe in the power of open science.

23:36And in the long run, we believe that is going to really shape the industry in an unprecedented way. So that's why we bet on open models from very early on. And it's great to see that open model performance is converging with closed model. So I think 2026 is the year that that convergence will become more prominent. And super excited about that.

24:03Jon Krohn:Yeah, something that I haven't mentioned on air yet is that for seven years before founding and being CEO of Fireworks AI, you were at Meta as a Senior Director of Engineering, where you led over 300 engineers. And a big part of what you were doing there, based on what I could find online, is developing PyTorch. So thank you for that. This is definitely a very big team effort. And I have a very talented team at that time. And yeah, so I'm very thrilled about the industry-wide impact of PyTorch. There are many great researchers and leaders, engineers I'm lucky to work very closely with. And some of them started a company with me, Fireworks.

24:51So really great crew. And we continue to going down the same path of democratized AI to the whole entire industry.

25:02Jon Krohn:Really cool. All right. So back to Fireworks and what you were saying about open source, not just PyTorch, but open source models and how you at Fireworks have made this bet to embrace open source AI models. Do you think that a lot of organizations, a lot of enterprises underestimate what's possible with open models? I don't think it's about underestimation that much. It's just about not being familiar with open models. I think especially, I think I will put, simplify this answer. There's two groups. There's AI native startups. There is enterprises. Obviously, enterprise has digital natives and traditional enterprise in various different categories.

25:53But let's just take a look at these two groups. They are reactions different. Air native startups, they just want to try the best option, and they do not have any baggage. They're very, very brave in testing open models. They're always very curious. And at the same time, there's a very interesting phenomenon that's also unique to AI. So as you know, there are many companies, whether startups or incumbents, they're trying and experimenting with new user experiences that our day-to-day is interacting with. and those kind of experiments have seen a lot of success in terms of product market fit but product market fit at AI time doesn't mean a viable business there could be a big 10 that people are willing to pay the product, pay for the product but the cost of running the business could be much higher than the revenue you get And then people just cannot scale their business after they hit the power market bit.

27:11And it's funny, literally, they're going to scale into bankruptcy. So obviously, startups have limited funding. And even including incumbents, they cannot, they have the distribution. And they have to be so careful. Their CFOs have to be so careful in controlling the cost. So the kind of budget and this ROI totally makes sense. And because of that, they have to hold a lot of people on the waiting list and cannot open up the gate. So while the startup, going back to AI native startups, they are very brave in embracing open models. But part of motivation is open models, they have a lot more control.

27:57They can customize however they want. But also at the same time, open models are much more economical. Because we as a provider, we do not have pre-training costs, billions of dollars pre-training costs to amortize over. And we only focus on bringing the best quality speed and cost from post-training, which is much cheaper, and inference customization of the deployment. So the unit of economics of fireworks operating open model is very different from Frontier Labs. So that's kind of where we see the AI native startups are very brave in adopting and going down the path of customization with open models.

28:42The second group of enterprises, obviously, they're much more careful because many of those are public companies and they are under certain kind of obligation, especially their legal team to understand what are the implications of using open models, license, what are license limitations, just kind of trying to understand what is this beast. But from my observation, the enterprise started to open up, and they started to understand much better, oh, this model actually is not going to send data to other countries. It's just a model. If the model provider is hosting a model in the U.S., and then they can provide privacy and security around the ins and outs.

29:40So, but anyway, there are other concerns as in the kind of content generated, does that follow certain kind of guardrail and so on. But overall, I've seen the whole entire industry start to understand better and start to kind of understand the bounds of operating open models. And I only see an upper trend of embracing open models. Nice.