1. Podcasts
  2. SlatorPod
  3. #296 Toloka CTO on AI Training Data’s Next Frontier
SlatorPod

23 Oct 2026 · 37m · 15 chapters

Episode artwork

#296 Toloka CTO on AI Training Data’s Next Frontier

In short

Toloka CTO Artem Grigorev explains why AI training data is becoming harder to create as models shift from LLMs to agentic systems, and why “smaller-model” post-training plus private, realistic benchmarks matter. He also covers Toloka’s human expert network, enterprise agent post-training, multilingual evaluation, and robotics egocentric video datasets.

Guests

Artem Grigorev, CTO at Toloka. Background: started at Yandex (search quality metrics and human assessors tooling), then helped evolve assessor tooling into Toloka; later worked through company spin-offs and became CTO. Toloka is part of the Nebius international group, where Nebius is a major investor (not 100% owner).

Key claims

data is still the main pillar (often ~80% of ML success); agent training data can take days (environments, graders, scenarios); private benchmarks avoid contamination and conflicts of interest; smaller models fine-tuned for narrow enterprise tasks can improve quality/speed/cost; multilingual quality requires language-specific expert evaluation.

Notable examples

support chatbots that call tools (booking status, cancel/refund) trained via SFT/RL; Toloka Arena benchmark using virtual environments with tools like Salesforce/Zendesk/Workday; robotics “Homer” egocentric first-person video data collected from contributors in 50+ countries; robotics data types include wrist cameras, pressure-sensing grippers, and harder-to-scale teleoperation.

Written by AI. May contain mistakes.

Chapters

0:00

Data Point Creation Over Time

Learn how the time required to create data points has changed dramatically.

“Previously, it was one to a few minutes to create one data point.”
0:45

Artem's Background and Toloka's Origin

Discover Artem's professional journey and the founding story of Toloka.

“Yeah, I will also try to pretend I'm an expert.”
2:21

Nibius and Its Relationship with Toloka

Understand Nibius's role and its relationship with Toloka.

“And then we went through a few spin-offs and now we're kind of independent company where I'm a CTO.”
3:38

Toloka's Services and Clientele

Learn about Toloka's services and the companies they serve in AI.

“So give us kind of the elevator pitch, the key pitch, like what kind of clients are serving use cases.”
6:05

Business Development Strategies

Explore how Toloka approaches business development and client relations.

“How do you guys do business development?”
7:31

The Data vs. Compute Discussion

Delve into the challenges of data compared to advancements in compute power.

“So kind of the data versus compute discussion.”
12:29

Complexity of Data Creation in AI

Examine how creating data points has become more complex and time-consuming.

“And if you don't have data, there is not much more ways to bring some new knowledge into the model.”
14:00

The Demand for Specific AI Training Data

Learn about the emerging demand for specialized training data in AI models.

“And that creates the demand in the new data, in the new types of data.”
21:37

Challenges in Multilingual AI Systems

Explore the challenges faced in developing high-quality multilingual AI systems.

“audience in different countries, I think it's important for you.”
23:08

Understanding AI Benchmarks

Gain insights into the significance and realities of benchmarks in AI.

“And even if the app tried to avoid having it in its pre-train, it might be there because they kind of downloaded the whole internet.”
Show all 15 chapters
28:04

Benchmark Trust and Business Growth

Learn how trust in benchmarks can influence business growth.

“So I think that's very important for the business.”
28:34

The Homer Data Set and Robotics

Explore the significance of the Homer data set in robotics AI.

“Maybe it's not bringing the money directly, but it helps us grow for sure.”
29:57

Challenges in Robotics Data Collection

Understand the challenges of collecting robotics training data.

“So that's so like you may imagine an access.”
32:44

Future of AI and Robotics Integration

Discuss the integration of AI advancements in robotics and its future.

“can easily join and start earning money by doing some tasks, that's the obvious choice.”
34:18

Human Judgment in AI Development

Reflect on the importance of human judgment in AI product development.

“And also, like, physical world is very different.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On SlatorPod, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00Previously, it was one to a few minutes to create one data point. Now it might take one to a few hours or maybe even a couple of days to create that.

0:13Hey everyone and welcome to SlatorPod. On the podcast today, we welcome Artem Grigorev. He is the CTO of AI data company Toloka. And Artem joined us at our Data4i panel in SlatorPod London this year. So I thought it was time to catch up, Artem. Great to have you on the podcast. Hello everyone, thank you for having me. This is a nice opportunity to talk about the data. Absolutely, and you know, it's something that I've been learning about for a topic I've been learning about for, you know, a couple years. So not yet at expert level, but that's why we have you here. Yeah, I will also try to pretend I'm an expert.

0:48Yeah, I'm CTO at Taloka. I've been with Taloka for quite a while and like my history started at Yandex. Yandex is one of the biggest internet companies in Russia. I joined it when I was studying at university as an intern. And I joined the search quality department. And search at that time, it was like more than 10 years ago already. It was 2010. So search at that time was like the most AI product, I think. and now, you know, the search engines, they kind of try to compete with all the new AI assistants and all the stuff. But at that time, that was the kind of the frontier of AI development. And I was working in the team that were responsible for metrics, for measuring the quality of different search engines.

1:44And close to that department was the department of assessors, the people who labeled some data manually, primarily there were assessors who judged the search relevance of the web pages, of some queries of search users, and based on those judgments like were either the model trained or some metrics calculated. And that is how I slightly then moved on to that assessors tooling stuff and that evolved into Tawoka. And then we went through a few spin-offs and now we're kind of independent company where I'm a CTO. So I'm not kind of joined the company CTO, I grew with the company and I'm here since that time.

2:35There's this angle, like, okay, you walked us through the index origin story, but what about this Nebius company? Is that like a holding that's holding a few things together? Yeah, Nibius is basically the international company and it started as a kind of new business having a few different companies in it. So it was Nibius AI Cloud, the GPUs and all this stuff. There was Starwalker inside and there was Aviaride that is a self-driving developing company. And they're also a triple 10 that is like educational platform, online education. But last year we got investment round and now Nibius is one of our investors, but not the holder of the 100 % of the company.

3:31So they're kind of our friend company and our major investor. All right, let's talk about Toloko. So give us kind of the elevator pitch, the key pitch, like what kind of clients are serving use cases. Just explain to our listeners what it is Taloka does. Yeah. So if you say it short, Taloka is the human data layer behind development of AI. So we do expert notation, evaluation, data collection, mostly for frontier labs who develop the OEM models and for AI teams in some enterprise companies. So the clients are kind of the biggest companies you know, Microsoft, Amazon, Anthropic, Shopify, some new AIO apps like Poolside.

4:21So those whom I can disclose. So we work with all of them. And our main asset is the expert network. So we have thousands of experts in different domains, countries, languages who can work on our platform and do some tasks that then convert it to a data set that our clients buy and use as the post-training data for their models, as the evaluation, as the benchmarks. Also, we have a self-service platform. This is the product for probably smaller or for those who want to start without any sales contracts and all the stuff. So we allow anyone, any professional to come to our product and launch tasks on humans, do some synthetic generation.

5:18So that all is possible inside our product. Third point is we are working on our own private benchmarks. Probably we'll touch that a little bit later in the talk. This is Tawak Arena. So we measure different models, how they perform on real tasks. And the last point, but not the least, this is kind of very, very fast-growing area is the enterprise agent post-training. So we create data, real environments, evaluations, and then we even train the models that can replace the frontiers for some specific use cases, perform better in terms of quality, speed, and of course, to cut some large checks that people often pay to major providers.

6:10How do you guys do business development? Like, is it you have your kind of USP and people come to you or do you have like, do you have to actively connect, sell, market to these companies? Both. I think we are not very different to any B2B company in the world. So we have developed the existing clients because for them we kind of, through some time we become very reliable data partners. So we're not only kind of outsource of data collection, we are more an extended team that helps them to understand what they can do with their models, how to improve them. So I think this is a very important part of our business.

6:52Of course, we hunt for new clients. As I said, we have a product that is publicly available. that's not the kind of very often you can really quickly address the problem where to get some human reliable data, how to access humans. So yeah, I think from my point of view as a CTO, the ability to technologically manage the people and wrapping it into some either managed service or the product is the most important way how we go to the market, what we sell. Got it. Let's go macro for a bit to start. So kind of the data versus compute discussion. So I don't know, I guess some say that compute, you know, eventually it's going to be solved, better chips, trillion dollar data center build out that's currently going on, which is completely crazy, but that like the data bit is harder to solve for, right?

7:49Yeah. And they're like, what are your thoughts on this? Yeah, I think hype is always in either models or compute, in algorithms or in hardware. But data is much more difficult. Yeah, people tend not to talk about difficult stuff. That's why maybe you read more news about data centers and energy consumption rather than human experts who provide data for AI model training. But still, for me, it's always been the main pillar. Even before the AOMs, for any ML model, I think the data is the most important part. So many people who have some applied experience in machine learning, they say data is roughly 80 % of the success.

8:41So for me, I think it's still the same. and the problem now, or maybe not a problem, it's a challenge that the complexity is rising. So the data that can add some value to the model is becoming much more hard to create. So previously, like for example, 10 years ago, Taloka had only the crowdsourcing platform. we have had people who like can join any task at any time, like read the short guideline and then perform that for a very like short period of time. The task was simple like and the main goal was to collect kind of big data sets that you can later turn into training the machine learning model, I know, like GBM for example.

9:36Now the data become much more hard to produce as I said. So after the LMS come, all those simple tasks are anymore not a problem. So LMS can pretty good label any data set. Of course, you still need some independent view on that to understand whether your labeling works correctly, like calibration set and stuff, but that's not that hard as it was before. With OLMs, the data become much more complex. So first of all, it started with some SFT data where you provide the model with some golden dialogues, the problem, the answer, maybe a few turns. And of course, once the model started being like evolving in different domains, that is very domain specific.

10:33So for example, we have experts in medicine, in math, we have people who have PhD, because the problems they create to train the model are really important to have some additional human knowledge that can be used as like post-training signal for the model. And ultimately now when we say about some just agentic use cases when the model not only like respond to you in chat but perform some actions, call some tools, like somehow behave in an environment. So the training points for that type of data for reinforcement learning are, simply speaking, docker images with some environment in it. So it has some tools.

11:20Sometimes those tools are kind of mocked, so it's not the real services, not the real MCPs, but they're mocked test implementations. And then you have a scenario which the model should follow. Sometimes you emulate how the user or how the other parts of the external world to the model perform. And then you have some graders, it might be some code written that checks it deterministically, or it might be AOM prompts that have some rubrics and checks whether the model performed that scenario in that environment correctly or not. So this is really complex. So now, like as I said, previously it was one to a few minutes to create one data point.

12:16Now it might take one to a few hours or maybe even a couple of days to create that. So this is really where we got after all AI development. But yeah, this is, I think this is important. And if you don't have data, there is not much more ways to bring some new knowledge into the model. I mean, the first time I heard about like, you know, the PhDs doing the labeling was maybe like three years ago. And so it's been like, you know, 36 months working on this. Like, do you feel we're hitting a certain ceiling where like you're so deep in, yeah, you said like a data to produce a data point takes like, you know, a couple of days or something.

12:59Like we're hitting a ceiling where like AI and human expertise, like you have to have extremely competent people now working with like the absolute latest to get to some kind of additional insight or get the AI to do something that is novel. Long-winded way of saying like, how much harder is it going to get in the next, you know, couple of years to just kind of improve on these like magic models. I can't really predict that because the last couple of years were quite insane in that terms. I also, like your question reminded me the discussion about like whether we achieved AGI or not. So like it's always a comparison.

13:48Like if we go back to some time, so probably at that point, the models that we have now, people considered an AGI already. But now like we see some problems, we still see some failure modes where the models don't perform as expected. And that creates the demand in the new data, in the new types of data. And the deeper you go, the more nuance you have inside. Like the domains may split into some smaller types. there are like new and new benchmarks that are created to measure some different abilities of the models and it's often a benchmark creates the need in the same type of data that favors the specific model on that benchmark so yeah as i said honestly i don't know how far it can go but I don't see yet any ceiling here.

14:47The first point, the second is, I think the new wave started around the smaller models and posturing them to specific business use cases. So people realize that you don't need that intelligence, that billion parameters models to solve some support agent use case. So that's why the training data is also started becoming important in much more specific use cases. That is requested for enterprise companies who not develop in the frontier model themselves, but they apply them to some business use cases. That is real value, I think, for the business, for the economy, where we apply AI to some processes where we automate things.

15:42And in that case, yeah, so it looks like it's a new big market for us to provide the data for post-training the smaller models. You just recently reported, maybe just days ago, right, that you trained a 4 billion model and you saw like a massive, like you need to explain this from six yeah that's exactly the point i started to explain so uh yeah we uh we train different models uh inside the company we create uh off-the-shelf data sets that we can sell uh like right away not to go into some project with us that requires creating that models that dataset, sorry. And the point is that if you have some relatively narrow use case, you can make, you can take a small model, small I mean like you said four billion parameters for example, not like trillions that are in frontiers.

16:51And you kind of make it forget all the generic knowledge it has inside and teach it to the specific case. For example, let's take support chatbot that can also perform some actions, call some tools, for example, check the I don't know, the booking status, cancel it or like make a refund. So this is a very, very common use case among different companies, how they automate things with the AI. So for operating the support knowledge base and deciding what tool to call, what information to get, you don't need to be that intelligent as, I don't know, Astra or Fable. So that's why what we did, we took a smaller model, it was basically a QAN, and we prepared a specific dataset and trained that model on the dataset and then measured its quality.

17:53And it shows that it's not only like on the test part of the dataset, but also on some relevant benchmarks relevant to that type of operations. It performs better than before. and with fine-tuning and RL reinforcement learning later, you can build a model that is better on all three parameters, on the speed, on the quality and on cost. That sounds a little bit too hard to believe, but the point is that we kind of have a narrow use case. We don't need that much neurons in the network to learn how to work in that use case. So I think this is a game changer for many businesses because you will be much more optimal in terms of the money you spend, you will have the better latency because, for example, if you build support chatbot responding faster to a client is very important for you and also you can achieve a better quality.

19:06And then Then you build the so-called flywheel. So you deploy that model. You already have the evaluation set. We can kind of benchmark it, how it performs over the time. We collect the traces of the model, how it performed on different real data points. Then using that eval, we can kind of find where the model performs worse than we expected. Out of those failure modes, we create the new data points and then we repeat the training procedure and we create a new version of the model. And it goes like every week, for example. And it's kind of very cheap and a very good way to sustain the automation and save a lot of money, achieve the quality.

19:56So yeah, I'm really excited of that new world that is started. So I think the future is in smaller models. I guess you could argue that maybe some AI translation systems are also narrow and smaller and perform on the kind of good, fast and cheap metric, right? Because if you have Astra doing gazillion words, it's probably not the most cost-effective option. For translation, like 100 billion parameters, probably that size of the model is already enough. So, yeah, that's the point. And translation also kind of adopted the deep learning, the transformers earlier probably than LOM started. So many, many things kind of reflect now what was that was learned earlier in the language translation domain.

20:54Let's touch on multilingual. So what are some of the top challenges in like high quality multilingual AI systems? And maybe, I don't know, is it just lower resource languages or are there other areas that you're seeing challenges in? I think one of the most obvious answer would be, yeah, you should care about different languages. So sometimes people tend to think that Internet is only in English, but it's not. That's why probably people don't create some benchmark that the points in different languages. they don't measure the quality, but if your service is real multilingual and you have audience in different countries, I think it's important for you.

21:42So that's where Taoka can help. That's also one of our kind of the axis of parameters of our experts. We have more than like we have thousands of different languages, maybe 40 plus most common languages represented in our expert pool. So yeah, there are some tasks that are related to evaluating the quality of the model in different languages. For me, it's very important. Yeah, I think maybe not that insightful answer, but that's reality, yeah. So let's talk about the benchmark you mentioned before, Arena, that's your benchmark, so A, I don't know, can you give us like a one-minute primer just to revisit benchmarks in AI?

22:35I mean, I log on to X, I see a gazillion benchmarks, I'm having a hard time understanding what they are, if they're legit, if they're not, like what are some of the incentives around having a benchmark and then like, is it PR? Is it internal use? Like why do you need a benchmark? Yeah, just give us a bit of a primer here. Yeah, there are many reasons. So first of all, many public benchmarks are kind of saturated or contaminated. So like often they kind of leaked. And even if the app tried to avoid having it in its pre-train, it might be there because they kind of downloaded the whole internet.

23:22So that's why we decided to have our benchmark private and never share the substantial part of it because it should be kind of independent and not spoiled anywhere. Second point is, yeah, so when a lab publish some numbers of a new model on some benchmarks, this is how they measure it. So yeah, I'm not saying that they're lying, but there is kind of conflict of interest. So that's why Taoka also can help the world because we are independent. So we're measuring, we're not developing our own frontier model. So that's why we kind of want to reflect the reality. And I think the last point is, but maybe one of the most important, we wanted to create a benchmark that is very realistic.

24:25So we took the real cases, the workflows in different domains like logistic, airline operations, travel, marketplace, bank. I don't remember them all out of my mind, but there are really, really major domains where people apply AI. and we took some popular tools that are used by the agents in that area, for example, Salesforce, Zendesk, Workday, and all this stuff. And we created that virtual environment that I described earlier to test the agendistic abilities. And also we tested several times and we expect the model to perform fine on all of that tries. Because sometimes also benchmarks say that if you succeeded in one attempt out of five, this sounds good.

25:31But that's not fair, I think, for the real business use cases. That's why we kind of test the real ability to. Yeah, so according to our benchmark, models are sorted in the way how you can really apply them to that agentic scenarios. I think it's important for businesses who try to select a model. And yeah, now we are working on training the smaller models and also measuring it against the benchmark, so like to prove the point that Frontier might be not the best choice for the company. Seems like a huge piece of work though. Like how do you justify doing this? Like do you have a team whose job it is to do this and then you have everybody else doing kind of client implementations, client work, stuff like that.

26:29Like this seems like it's almost a nice to have, right? You don't need that to generate revenue, but you have it. So, you know, just I try to understand like the effort to me. Yeah, you're absolutely right. Although it supports the revenue because this is not a kind of separate department that is doing it somewhere in the dark chamber. So this is the people who do the real projects for the clients and they know how to create that data points to perform, to like measure the best, to give the best signal. And for us, it's kind of a door opener to prove the market that Taloka is capable of doing that.

27:22And this is very complimentary to the off-the-shelf data sets that we are selling. So yeah, Arena itself, yeah, it only burns money on LLF on testing like and whatever the new model goes it tends to be much more expensive than it was before so yeah running that benchmark is kind of substantial uh spendings but uh it helps us to drive the uh uh setting of our all the self data sets and kind of yeah trust in us in the market the branding the PR also is very important so yeah I started seeing comments on LinkedIn that people really like our benchmark they like kind of admire that we are private we are not like exposed anywhere so that they can trust us and even recently I saw a comment then one person wrote that yeah I really my gut feeling is the same as you show in the benchmark around around the model capabilities and how they perform in the real business tasks.

28:31So I think that's very important for the business. Maybe it's not bringing the money directly, but it helps us grow for sure. I want to go with sci-fi and talk about your Homer data set, which is like a robotics first-person video data set. So apparently you collected data from foreign contributors in like 50 countries. But this is real. I mean, is this the thing that I'm like real life? Like, is this the thing I'm seeing sometimes these videos of people like folding clothes or like, yeah, tell me more about that. Yeah, yeah. Like the whole previous conversation was mostly about LOMs kind of in the online work, let's say it.

29:12But there is a huge world in offline, like and all the robotics stuff. And that world is developing a little bit different. So the problem with robotics is that there is no kind of internet for robotics. So you don't have substantial data set for pre-training the video language action models. So that's why that kind of simple in terms of absence of any specific expertise tasks that kind of show the robots. I'm trying to speak very simple words. So they show the robots how to operate. That is still really valuable. So that's so like you may imagine an access. So from like no expertise at all to, I know, PhDs and like very top talent in the industry.

30:19And on that gradient, I think it's kind of created demands in two sides. So one side is like the PhDs we discussed, and the other side is like many, many people who can do any human work in offline and physical world. And still, I think nobody knows like how that data will be growing in the future. because, of course, having that major bottleneck, like the labs that develop the models and the robotics application, they try to create other ways how to train that. Also, besides that egocentric data, where, as you said, the person with the smartphone on their head, filming their hands, There are much more complicated types of data.

31:23For example, you can also have cameras on your wrists and record. You can have some specific gloves that help you to identify the fingers and the hands better. There are also some sort of grippers, so-called. This is like claws that you use to hold and pick some objects. and it can measure the pressure because this is also very important dimension to learn for a robot. So you should pick an egg and a hammer with a different kind of pressure in your muscles. There are also the kind of teleop data where you operate with the real robot. So there is a robot and there is a human operator who has some joysticks and they operate with the robot and that's the type of data that contains the most available signal, but it's much harder to scale.

32:27You should build that robot first and then try to get the data. So, nobody knows where on that spectrum is the optimum, so different companies try different approaches. So for us, yeah, as we have the platform where anyone from the internet can easily join and start earning money by doing some tasks, that's the obvious choice. Yeah, we can quite easily collect large data sets of that egocentric videos. And yes, to kind of show what we can and how it may look like, we open-source a portion of it. So in our public dataset. How transferable, I mean, you mentioned that the platform obviously makes sense for both kind of the LLM AI and kind of the physical AI world, but like, other than that, like, is it, does it transfer really well?

33:22Like, has there been a revolution, I guess, in physical AI development also in the past two to three years? like do the two go in tandem or not really? I don't know. People tend to say that the chat GPT moment in robotics is not yet come. So I'm not that expert in the robotics development to have my own judgment. But yeah, I still see large bottlenecks that probably prevent that scaling that is achieved in the LOMs. So, yeah, from my point of view, those bottlenecks are mostly in the data. So that's why I think, yeah, it will be developed and you'll see much more new types of demand in terms of data and much more powerful models.

34:18And also, like, physical world is very different. So it's not that deterministic. There are some gaps between the model you trained and the real device. So this is much more complicated. And probably I start to agree with the people who say that when the robot will be operating on a human bed, it will be the AGI woman. So now we are kind of only trained to predict the next world quite nicely, but we are not replaced humans yet. Give us a few years. We need to adjust to the kind of the NLMs first and that kind of intellectual revolution before we have the robots walk around. Yeah. I mean, when I watch these videos from like, what was it?

35:03The Shanghai Robotic Olympics or something, or I'm not sure if it was in Shanghai, maybe in China generally, it's just crazy, right? Yes. All right. So at Slater County in May, you closed with a prediction about kind of the underestimating or underestimated human judgment and maybe expand on that. Like what's left? We talked about PhDs and really smart people like, you know, taking days to produce a kind of a novel data point. So like what's left? Yeah. Yeah. So I also like often think about how like TALOCA will be developing in the future, how the human signal will be important or not. And yeah, I think among all different ideas and hypotheses, I think the main is that the judgment will be important as long as we develop in some systems for humans.

36:00Right. So if you develop any product that human will be using, you still need some reliable signal how to understand whether this product fits that human or not. So the model will not give you the sense of taste. Like you can look at two pictures and say, this looks beautiful to me. This looks awful. You're right. The model not. And yeah, maybe we will end up like with the old human knowledge loaded into the models and no need in PhDs in physics to train it, but still to understand what's good, what's bad. yeah it will be in the future together with the humanity existence we'll focus on the finer things in life and train the AI on taste yeah taste is a very good word yeah alright Artem that was fascinating I think I could have gone on for a lot longer here but yeah let's try to keep it a little bit focused for people to absorb this thanks so much for taking the time to do this Yeah, thank you for having me.

From the publisher

Artem Grigorev, CTO of AI data company Toloka, joins SlatorPod to discuss the growing importance of human expertise in AI development, the shift toward smaller specialized models, and the challenges of evaluating AI performance.

Grigorev explains that while computing power and model capabilities attract significant attention, high-quality training data remains fundamental to AI progress. As models become more sophisticated, data creation has shifted from simple annotation tasks to complex, expert-driven workflows. He highlights growing demand for domain specialists, including PhD-level experts, as large language models increasingly automate simpler annotation tasks.

Grigorev also discusses Toloka's work on post-training smaller models for specific enterprise applications. By fine-tuning models on targeted datasets, businesses can potentially achieve better performance, lower costs, and faster response times than they would relying on large frontier models.

Turning to multilingual AI, he emphasizes the need to evaluate model performance across languages, rather than relying predominantly on English-language benchmarks.

Toloka Arena, launched earlier this year, is the company's private benchmarking initiative designed to address concerns around contaminated public benchmarks and assess AI agents against realistic business workflows. Grigorev explains how Arena tests agents across multiple attempts to assess consistency rather than a single successful run.

Grigorev also discusses the growing demand for robotics training data, including first-person videos of humans performing physical tasks, and the challenges of collecting such data at scale.

Looking ahead, he argues that human judgment will remain essential, even as AI becomes increasingly capable, particularly in determining whether AI-generated outputs meet human expectations and preferences.

More from SlatorPod

All episodes