#304 Matt Zeiler: Why Government And Enterprises Choose Clarifai For AI Ops

0:00It started with just inference back in 2014. The reason we started there and having an API platform was inspirations like Stripe and Twilio, where they built a great developer product and people just built amazing things on top of that. And they became kind of the engine for that. And we saw that there needs to be this engine for AI. And so that's the same product on the commercial side as public sector side. We intentionally built a whole suite of UIs that make the system easy enough for anybody to use. And because of that, we've had even intelligence analysts train models themselves, doing all the labeling, picking a training template, looking at evaluation metrics of how they perform without our teams being involved.

0:43That's key to making the API successful, which is ultimately what you're going to use. Once you have a good model, you're going to use the APIs to fire through lots and lots of data for inference. But to get to that model, the UIs are really important. In business, they say you can have better, cheaper, or faster. But you only get to pick two. What if you could have all three at the same time? That's exactly what Cohare, Thomson Reuters, and Specialized Bikes have, since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds.

1:40How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50 % less for compute, 70 % less for storage, and 80 % less for networking. Better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is a cloud built for AI and all your biggest workloads. Right now, with zero commitment, try OCI for free. Head to oracle.com slash IonAI. Eye on AI, all run together, E-Y-E-O-N-A-I. That's oracle.com slash eye on AI. I'm Matt Zeler and founder and CEO of Clarify. And my history in AI goes way back even into undergrad.

2:44That's kind of where it started. A little bit by luck. I was at University of Toronto and in a program where you have to decide if you want to take, you know, many different options of engineering. and I was deciding between the computer option and nanotechnology, and that's when I happened to run into one of Jeff Hinton's PhD students, a guy named Graham Taylor, and he happened to be my resident advisor on the floor I was living in. So that was the luck of it all, and he showed me some of his research, and back then he was generating videos that looked realistic of a flame flickering, and he said it was all done by AI.

3:20And so fast forward to today, everybody calls that generative AI. This was 2007. So we've been doing this stuff for a long, long time, well before ChatGPT, where everybody kind of realized its value. And so that got me hooked back then because I learned how to program before that. And there was no way I could program in a traditional sense to generate a video like that. So I decided to dive into the computer option, took Jeff Hinton's course, and then I ended up being Jeff Hinton's first undergrad thesis student. that was quite the honor to work with him and I did a thesis on actually generating motion capture data for biological studies of pigeons walking around and how they behave so that was a cool undergrad and by that point I knew that I had to you know focus on AI it was going to change a lot of things in the world and I didn't know enough coming out of undergrad so I decided to do a PhD and I looked at all the different schools and chose NYU to study with people like Rob Fergus who is my PhD advisor and others like Yan LeCun in his lab.

4:31So I got to work with both of them and focused on computer vision for my PhD applying neural networks to understanding images and videos and that was between 2009-2013. During that time I also spent a couple summers at Google Brain and worked under Jeff Dean on the Brain Project. At that time, it was like 20 or 30 people. So it was really fun to work there. And that's where I learned how to scale up AI at the large scale that Google operates at. But I left my second internship a couple weeks early to come back to New York and start working on building a company called Clarify. Because what I realized is the work I was doing in my PhD was actually working better than what Google had internally for understanding images.

5:18And I saw that as an opportunity to start a company around this. And it's been a childhood dream of mine to start a company. It's kind of since undergrad, I knew it would be in the AI space. And so this was kind of my shot at that, having really good research. And I incorporated in November 2013. And about three weeks after incorporating, ended up winning ImageNet with the Clarify name. And that was the biggest computer vision competition at the time and this was the year immediately after AlexNet won ImageNet which kind of put deep learning on the map as as actually being an effective method in the first place so it was great to be the first in the world to reproduce those results and then surpass them and it was a great way to kick off the company because that got inbound interest from day one from investors and and customers so that's kind of the origin story for Clarify and I can go into more about what we do, but I'll pause there.

6:14Yeah. And on that ImageNet competition, was it you and a team that you'd pulled together for the startup or were you still working through academia? So good question. I did actually two entries. One was with my PhD advisor, Rob Fergus. We call that ZFNet or I think other people named us that for ZELR, Fergus. and that got third place in the competition. Somebody else got second and then Clarify, which was only me at the time, got first place. There was a few different techniques that I was doing as part of the company separate from my work at NYU. Yeah, that's amazing. And then from there, you very quickly, Clarify, very quickly grew into one of the first on one of the most influential image recognition companies, and you began working with law enforcement and the Pentagon on Project Maven.

7:22Can you talk about that? Because I remember at that time,

7:29or around 2017, 2018, when I really started focusing on AI, I wanted to talk to you guys, and you were not available to talk. And I don't remember what it was. I think that there had been some sensitivity around the law enforcement applications or something, and you guys just didn't want to be out in the media. Can you talk about that period? Sure. Yeah, so kind of between starting the company at the end of 2013 to that period, we built a lot of products around computer vision, really a horizontal platform. We had our first APIs up in 2014 serving inference traffic. We started by building what people call foundation models today for computer vision, trained to recognize tens of thousands of things in the world so you don't have to train them yourselves.

8:28and we built a few of those for different use cases like wedding photos food our general one is the most popular still to this date general purpose like social media consumer photos all that kind of stuff content moderation filtering drugs nudity weapons another big popular use case and then in 2016 we kind of allowed the platform to let people train their own models label data and train their own models. And so people could fine-tune it for their particular use cases. And shortly after that is when the government actually approached us to be part of Project Maven because they wanted to see, can they take a platform like that and have it fine-tuned on intelligence data?

9:11And they got a demo of our product. I still remember this. It was maybe the only customer meeting I had on a weekend. The leaders of Project Maven came up to New York to our offices on a Sunday, carrying a military bag and walked in and had me demo for an hour. And after that, they're like, yeah, you definitely need to be part of this program. And we were under contract within a month of that. Still our fastest government contract since, actually. So it was a great experience with the original leaders of Maven because they took the approach that an enterprise takes on the commercial world. They evaluate different alternatives.

9:51They do their research. They find the companies that are doing cutting edge stuff. They talk to them, evaluate them, and then pick and get them under contract quickly so that they can see the value from it. And so that was our experience with Maven. And, you know, working with the military, it did raise a few questions internally. And there was other companies, famously Google, that was part of Project Maven, that had similar conversations that happened internally. And I still remember, it's probably like 2018 when I had an all hands that I just told the company that what we're doing in projects with the government like Project Maven is ultimately going to save lives using artificial intelligence to make decisions faster and in a more intelligent way than humans can do it.

10:40And because of that, we're going to be focused on this as a company. And some people didn't like that. and one or two quit after that all hands. But what it did was stop all of the gossip, stop all the debating internally and got us all aligned. And a lot of people after that all hands, many more than the number that quit, said, oh, thanks for doing that. Finally, we're like past this. I'm like really excited to be working on these kind of programs. And I totally agree. It's going to have a big impact. So that was a big, big catalyst to us continuing. and in Project Maven a lot of people think of it just as vision enabled drones but it was much broader than that right can you describe Project Maven and what happened to it I've lost track of it it's uh it's still going I was literally just drove over from a meeting uh with a customer right before this and um I can't go into all the different things that it does but it broadly became the first AI program within the government and definitely the first successful one.

11:48And because of that success, it gathered other lines of effort, all different types of sensor data you can imagine the military having. They want to apply AI to, again, accelerate those decisions and make better ones. And so it started with drones, but it's gone to many different directions from there. Yeah. And it's, it's visual data or is it anything signals intelligence or. Yeah. I mean, it started with visual, but now there's discussions about every, everything you can imagine at this point. And that includes, you know, with, with Gen AI in the last couple of years, becoming very popular and useful large language models.

12:28What can you do with them in the intelligence scenario? That's part of the discussions with, with Maven, but also with many other programs across the, the DOT and intelligence community. Yeah. And, and the output from, from the Maven platforms is for what, what's the primary use? I remember early on there was a lot of talk about targeting and, but, but then there was a lot of sort of, pull back from that by the government saying that it wasn't targeting. It's really for intelligence analysis. Where has that gone? I mean, where does it lie now? Is it still with the Army? Is it or with the joint, what is JAIC?

13:31I forget what the A was, but yeah. Joint AI Center. Yeah, yeah, yeah. Joint AI Center. Is it sort of a joint program across the services? And yeah. No, it's actually transitioned to NGA. So it's part of the intelligence community now. And that maybe hints at kind of the purpose of it. Less so, you know, automating target recognition, but providing just better intelligence. Yeah. And so that's ongoing, Clarify. And then you also had products for law enforcement. Are there discrete products or were these, was it a platform? Did Clarify have a platform or APIs that different agencies would employ and cobble together a system on their side?

14:27Or do you have a product that agencies, whether it's law enforcement or not, can use to scan faces, for example, or whatever the inputs are? Yeah, so everything is one platform. And as I kind of mentioned, it started with just inference back in 2014. The reason we started there and having an API platform was inspirations like Stripe and Twilio, where they built a great developer product. And people just built amazing things on top of that. And they became kind of the engine for that. And we saw that there needs to be this engine for AI. And so that's the same product on the commercial side as public sector side.

15:22We intentionally didn't want to build two different products as a small company. And what that's allowed us to do is, in order to do that, we had to focus on making the overall platform really flexible in its deployment options. So when you deploy with a lot of these government agencies, they typically want to go on-premise, even into air-gapped environments, on classified networks that aren't connected to the internet. So all these different parameters go into our builds now so that they're flexible enough to go in those environments all the way up to AWS, Google, Azure, etc. And that really helped us build a much better product at the end of the day in handling all those things.

16:08But it also focused us on having just one product to take it to solve a given problem. there's usually collaboration with the customer and our professional services team who can customize things in the product, customize the AI models with the customer. And the platform's easy enough for the customer to do this as well if they choose to so that the AI is learning how the customer cares about recognizing or generating what they need. So flexible enough to deploy everywhere and flexible enough to kind of cater to the problem at hand. Yeah, but it's a developer's product, an API into your system that they build into solutions.

16:53Yeah, exactly. And on top of the API, we have really simple to use user interfaces. So, for example, data labeling is a core component to building AI systems. And that requires a lot of user interfaces to draw bounding boxes or draw per pixel masks of objects. And then there's UIs for the task management. Like, what do you want to label? Who is the workforce that you want to label it with? Is it complete? What's the progress? All those kind of experiences, they're not really good as an API only. So we intentionally built a whole suite of UIs that make the system easy enough for anybody to use.

17:31And because of that, we've had even intelligence analysts train models themselves, doing all the labeling, picking a training template, looking at evaluation metrics of how they perform without our teams being involved. So that's key to making the API successful, which is ultimately what you're going to use. Once you have a good model, you're going to use the APIs to fire through lots and lots of data for inference. But to get to that model, the UIs are really important. Yeah. And then you sort of shifted into inference as a service. Is that right? So I would argue we started there and now we're, doubling down on it again.

18:16Yeah, but in... Yeah, can you talk about that? Why is... I mean, I've had Rodrigo Liang and Andrew Feldman, you know, that's Seminova and Cerebris. I haven't had a guy from Grok on. We missed appointments a few times, but that's on the hardware side. speeding inference through hardware and you're doing it through optimization in the code is that's right is that right and so that you can beat these inference chips or these specialized accelerators without any specialized hardware just simply by optimizing the code first of all yeah why is inference such a bottleneck uh because it feels fast to uh to consumers and then uh yeah how how did you go about doing this without and and and does this would would your inference uh strategies just accelerate inference on a platform like cerebris uh that much faster that much more?

19:35Or is a fork in the road either you go down code optimization, algorithm optimization, or you go down the hardware route where you can't have both? Yeah, great question. So inference is a good segue from the last conversation. It's kind of the production use case. Training the model is like a precursor to the inference. Really what you want is an intelligent system that you're going to run over and over, get accurate results, get them quickly, or get them at large scale of batches of data, and do that in a repeatable, reliable way. And that's really where the company started back in 2014. With our first models, we didn't let anybody train in the platform, didn't let them label or anything.

20:22It was just inference. And back then, even going into my PhD, we were one of the first people writing CUDA kernels for AI. I started getting into GPU stuff definitely in 2012, maybe even 2011. But at that time, it really started out at Stanford. There was some work from Andrew Ng's lab. Then Toronto picked it up with Jeff Hinton's lab and people like Alex Krzyzewski writing some really good kernels. And then there was only four labs doing AI back then. There was those two, NYU, where I was at in my PhD, and Montreal with Yoshua Bengio's lab. And so all four of us were collaborating on the use of GPUs.

21:06And once Alex had some really good kernels, we all shared them. And so I remember distinctly in my PhD, I first adopted them for my experiments. And overnight, they were 30 times faster. And that just changed the game. Instead of waiting for results for a day, I could go for lunch and come back and have my experiment done. And so thinking back, I was probably one of the first 20 people in the world actually writing CUDA kernels for AI, which was pretty cool. And that kind of deep expertise was the genesis for Clarify. We had to build toolkits before PyTorch and TensorFlow existed. We had to build GPU operators in Kubernetes before NVIDIA was building operators to support GPUs within Kubernetes.

21:53So kind of all the layers up, we have expertise within Clarify on how to build these systems. And so that realization in the last year and a half was that everybody's still struggling to run these big, large language models that have become very popular since ChatGPT. So why don't we take a lot of this learning we have and a lot of the internal tools that we've built over the years to run models efficiently and at very large scales and open them up as a new product line. And so that's what we launched recently as AI compute orchestration. And we call it compute orchestration, not just inference, because inference is just the first workload.

22:32You'll see more coming for training, for agentic tasks that run in the background for a long period of time, all coming soon. but inference we view as the most important one because of the scale it operates at and because of a lot of the optimizations we can do it benefits inference really really well and your second question was do those optimizations work only on gpus or can they work on other things they can work on other things depending on the optimization so we have a mix of low-level things like CUDA kernels that we optimize. We have things like moving Python code to C++. There's a variety of different tweaks we do, but there's also tweaks that will work across accelerators.

23:16We're predicting usually many tokens in advance using good speculative techniques, and that should work on any kind of accelerator. A Google TPU, a Cerebrus, a Wafer, whatever you may throw at it. So it would be a mix. And we found that it depends a lot on the use case that you're using these models for, how well some of these accelerations can work. So this year, especially, there's been a lot of excitement about reasoning models and agentic tasks. That's really what we built the optimizations within the Clarify reasoning engine for. because if you think of how these models reason, they're actually repeating themselves many times.

24:03They're referencing a lot of stuff in your large context a lot of times. And so there are certain optimizations you can do for those kind of repeatable agentic tasks that perform really, really well. But they won't work as well on every general task. Yeah. And to unpack some of that, I get lost in these conversations and I have so many of them, I'm familiar with the jargon, and I don't want to slow guests down and asking them to explain things that the more advanced listeners already know. But let's explain to people what a CUDA kernel is and why it's important. Yeah, go ahead. Yeah, so CUDA is NVIDIA's programming language for their GPUs.

24:55I don't actually remember what it stands for off the top of my head, but it's been around for like 15, 20 years almost at this point. And it lets you, you know, there used to be languages for gamers to program GPUs to make games, but they're really low level like shader languages. And so CUDA became a general purpose language, just like you can program any software on a CPU. You've been doing that for many decades. This was like the equivalent for a GPU. And GPUs just unlock much more parallelism than a CPU offers because they have thousands of cores instead of tens of cores. And that gives a lot of advantages for neural networks in particular because when you're doing the matrix multiplications that are common in a neural network, it's highly parallel.

25:44So GPUs are a great fit. Yeah. But then kernel is a function. Yeah. Yeah. And so the kernels are an example is a matrix multiplication kernel. So multiplying one matrix by another, that would be executed as a single kernel on a GPU. And so when we're talking about CUDA, we're talking about NVIDIA. AMD has an equivalent now of CUDA called ROCCM. So there's ROCCM kernels, there's CUDA kernels. And that's kind of the lowest level. And it builds up from there. A lot of people today don't have to operate at that level because NVIDIA and AMD and Cerebris, Grok, etc., whoever's making the hardware is usually making really optimized kernels for their hardware so that people can just use the software on top, like PyTorch, for example, which implements the best kernels for NVIDIA GPUs, for AMD GPUs, and other hardware providers.

26:44but when you're trying to push the limits of optimization sometimes you can find techniques at the kernel programming level that are even better than what the hardware providers writing themselves yeah and i mean this is essentially what deep seek did right uh yeah yeah yeah deep seek did a lot of really clever engineering down to the kernels all the way throughout The systems engineering, everything. And how much of what... So the hardware producers like Grok, Simba Nova, Cerebris, who are focusing on inference, their point is that you don't need to go in and tinker with the kernels in whatever the programming language of that particular hardware is.

27:40We've done that, and we provide the super fast inference that you can access through API into our cloud or on promise if you want to do that. In the DeepSeq model, it's not the hardware. You could use any hardware, presumably, but they're optimizing inference on the software side. how influenced were you by DeepSeek? Was DeepSeek just the first to make a splash in, you know, the zeitgeist was already working in that direction? Or did people see what DeepSeek did and say, yeah, that's right, we can do that. Why aren't we doing that? well we were working on computer orchestration about a year before deep seek came out just because the general trend was that open source is uh catching up to the proprietary models and i think deep seek was just a huge catalyst to open everybody's eyes in that same direction and they did a lot of great work on the deep seek team and they published a lot of it too open source GitHub repos, good technical reports.

29:01So kudos to them. And they're not the only ones. There's been a lot of great work out of the Alibaba teams for the Quen models. There's the Kimi K2 models, another good one out there. A lot of great open source models are coming out of China. The one we've focused the most on is OpenAI's GPT OSS. And the reason for that is, one, it's built by an American company and we do a lot of work with public sector so we can deploy that model on premise without any question marks but even more important it's a really efficient model from a compute standpoint you can run it on a single GPU and get really high throughput it and because of that you can offer it for really competitive pricing and it's very intelligent so kind of that combo of cheap fast and intelligent is like the the trivecta i guess uh that everybody looks for and i i don't think anything's even close um from those dimensions uh some may be more intelligent but then they need eight gpus uh and that costs a lot uh so it's uh it's a great model and and we're the fastest in the world at running it because of all the focus we put onto it Yeah.

30:15And so you're offering it as a service or I mean, yeah, it's not open source. The code is not open source. Correct. Yeah. So the way our computer orchestration works is we have some of these popular models like GPT OSS offered as a serverless option. so you can pay per token. But if you, and this is what most customers end up doing, if you like a model or maybe you have your own fine-tuned version of it or your own custom foundation model and you want to run with some of the optimizations we have, you can deploy into dedicated node pools, we call them. And these are pools of the same type of instance that can scale up and down dynamically based on your traffic needs.

31:00And they're dedicated to you. So you don't have other customers making requests in there. You're at full control to deploy whatever you want into them and control how things scale up and down. And so we have both the serverless token-based pricing and the dedicated, which is priced by GPU hour, essentially, options. So lots of flexibility in how we offer up these kind of models. And that dedicated GPU pool or whatever, is that on your cloud? Do you offer a cloud or is that in any cloud and you have ways to ring fence the GPUs? Yeah. So we actually offer both now. So in our VPCs, you can provision, clarify, compute.

31:48and they're in our overall cloud VPCs that sit across multiple clouds. You can also deploy into your own VPC or even your own bare metal Kubernetes clusters. So wherever your compute may live, you can run. And the big advantages of that are you get that kind of flexibility. So maybe you start on premise because it's going to be the cheapest if you have the upfront capital to buy machines. But you're kind of by definition at a fixed size when you're on-premise. And so you need a way to spill over to maybe a NeoCloud as the next option. And then even NeoClouds have a limited capacity. If you're really taking off, you may have to spill over to a hyperscaler or multiple hyperscalers.

32:32And so you can do that all within the Clarify platform without having to retool or re-engineer things for every cloud or on-premise differently. You get the same kind of performance on chips wherever they may be. and that includes not just nvidia chips but amd and uh we're working towards support for google tpus uh kind of as we speak so there'll be other accelerators offered in the near future as well yeah and you were saying that that this can run on any uh hardware so it could run on saminova or cerebris but does it accelerate their inference speed the does it accelerate inference beyond the speed that they offer?

33:16I mean, is there a cumulative effect or is it, you know, it's going to be what you guys can manage or it's going to be what Cerebris can manage on its own? It's not going to be some multiple of that. So those platforms we haven't focused on yet. I'd love to explore that with each of them. but in our experience with new chips there's a long lead time for the software that they provide to catch up and for especially for other third parties to implement optimized software and so i think google tpus are at that point now but they've been in the works for like a decade and and other platforms i think are really early on how well the software stack will allow other third parties to uh to write good stuff with cool has been at it for a long time amd's caught up with rockum um but the others it would be hard i think for for somebody like us to squeeze uh performance out and i think that's actually why you know you see people like cerebris they used to sell the wafers um a lot that was their business now they're focused on like serverless we're just going to host the models and be really fast at it and they are they're great but I think they found it difficult for people to write the software for the hardware that they were selling.