SAM 3: The Eyes for AI — Nikhila & Pengchuan (Meta Superintelligence), ft. Joseph Nelson (Roboflow)

32:04And also, it's very intuitive. You can see that for reason's sake, it has this kind of short-num untested. It has kind of different subsets, short-num. short, then it's very close to sensory training data. It's atomic phrases, short phrases. Long is this very complex reasoning. You will see that for short, sensory only is very close to the sensory agents. But for long, the gap is so large, which indicates that, okay, that is exactly the capability that large language model brings in. Got it. I can show an example here that might be insightful too. Go for it. So even comparing even comparing like sam3 and gemini let's say that we just want to have them do like a an optic detection task here of finding here we're going to prompt with a speedometer and rpms and we're going to ask for things like indicator light number and needle and if we run sam3 head-to-head with gemini 3 and florence 2 almost as a baseline of like where things have been and we see each of the results first things first you'll note that the speed of inference of SAM3 is quite quick.

33:17This is just calling the Gemini 3 Pro API. So whatever is provided from hosted compute is sort of what you get on the response time. And then the second thing you'll note is, in addition to speed, is some of the accuracy of results. We might have a timeout error. Let's see. Do you have ELO scores? What scores? ELO scores. ELO, yeah. Yeah, you had the arena. Okay. I was wondering what the ELO was, because you said you were blind testing this. Yeah, that's actually interesting because we had blind tested SAM3 before it was released, not a SAM3, just for people to try and compare. I think we call it like a potential SAG or SAG preview or something.

34:03And we allowed users to vote and they kind of unanimously voted for, but they didn't know at the time was SAM3. We actually got like emails of people being like hey like where can i use that and we just sort of ignored them until the model came out but so here with with the responses you see that um the uh grounding capabilities of of sam and uh sam 3 compared to even gemini uh are out ahead currently so not only is it doing grounding but if you look closely you can actually see it's making segmentation masks too whereas as Gemini 3 struggles to do, it just does detection by comparison. And then the other thing is just the richness of detections, like the recall is high, as well as the precision.

34:49And if we compare here, it does almost as well, right? But you see that it misses some of the numbers and has kind of these, some of these erroneous boxes that it's, that is predicted. And then it also doesn't do segmentation. So it just does detection of the task. So you can envision that the same way the SAM3 paper introduces the idea of using SAM3 in tandem with MLMs. I would expect that to be the case pretty soon. And maybe the Google team taking some notes to improve Gemini and other series of models based on what SAM3 demonstrates here. So in other words, not only is it faster, but it seems to be more comprehensive for concept segmentation.

35:31And I think the speed actually is a huge factor for many use cases. I think even Meta, we're using SAM3 for various different product use cases, and fast inference speed is very critical to enable that. And so I think that's something that I think in many cases, you don't even need an MLM for. It's just kind of overkill to use an MLM for some applications. The other interesting thing is the Florence 2 results. And, you know, Florence 2 is a little bit older of a model now, so maybe it's not fair to put up head-to-head with the state-of-the-art, but it is useful as a way to just see how far we've come.

36:11Because Florence 2, by comparison, labels the entire region as a single class without seeing individual detection of numbers and indicator lights and needle. And not only that, but it actually runs at about three times the speed as SAM3. So SAM3, again, is faster doing a task that the other models are not doing in segmentation and more accurate, both in recall and precision of the things that it's intended to find, which I think really showcases the capabilities of the model. In fact, I even got a little surprise about this because this domain, this more like an OCR-like because recognition numbers is nearly OCR.

36:53We do not prioritize this domain of data collection. it works. So we know that it roughly works, but I think I got surprised that it works so well. That's encouraging. Even a task that wasn't expressly prioritized, it still does a great job on. Yeah. In fact, during our data engine, we intentionally do not sample OCR-heavy images. Wow. On an easier one, GlassMug, SAM3, Gemini 3, Florence 2, SAM3 loaded first, and has really impressively it sees even this glass mug in the corner which i think is something sam 3 does a great job of is occlusion and partial objects gemini 3 struggles a bit with this one i think maybe because the opacity of the objects by comparison and then florence 2 does a good job at finding one of the glass mugs so again another type of task that shows the power and veracity of the model yeah i mean exhaustivity like finding every instance is something we heavily prioritized and is really built into the data engine design um you know you want to talk about how we design the data engine to really scale exhaustivity because you know if a human was to set and appetite every single instance it would take a really long time and verify, but we put a lot of effort into trying to automate and speed up that process such that we could get to the data scale and diversity needed to get to a step change.

38:29Yeah, yeah. I think definitely, I would say data engine is the critical component that we achieve sensory performance now. So maybe we can go to the data engine picture. I think we have an illustration there. Yeah, page 5. You can see that this is our annotation pipeline. So we first source the images and generate the non-phases. So this is the input of this task. Source images and generate non-phases from, for example, Nama generate caption and we pass the caption to get the non-phases. This is the input distribution. Then we use San Sui model in the loop to generate candidate masks that we that should be the candidate, but it's not perfect, especially in the beginning.

39:25Then we go to, you can say, go to the next step is verification. So San Sui gives you this mask, then we need to first do mask verification to verify each mask, whether it's good or not. And then after we filter all the bad masks, we there are some good mass left, and we verify whether this good mass are exhaustive or not, like your mark example. So for example, the byte model do not predict that partial mark, then the exhaustivity check will be failing there. Then if the exhaustivity is filled, then we go to the next step. You can see that we can go to the pipeline, go to this so-called human manual correction.

40:12Human manually annotates all these missing masks. You make this data point exhaustive. So you can see that exhaustivity is a very big factor there, and we play it as the center place in this data engine. But you can see that if we ask human annotators to annotate every mask from scratch, It will take a lot of time. I remember each data point in the beginning will take about more than two minutes to finish. But if you use model in the loop, then it's reduced to about 45 seconds. You use model to propose math and then just a few months to annotate the missing math. Then it's 45 minutes. Another very key innovation in this data engine is that we really found that there's verification steps.

41:02like to verify a math is good or not, or to verify now the good math are exhaustive or not, can be done by AI, can be done by that multimodal model. That is a breakthrough, and then we kind of fine-tune our, for example, NAMAS 3.2 with our verification human annotated verification data. We get superhuman performance on these two verification tasks, and then we do not need human on these two tasks. This further brings our data point annotation time to about 25 seconds. So you can see that from the original all human to about two minutes to finally 25 minutes for one data point. This is the journey of our data engine to make it super efficient.

41:54Did you maintain statistics on how many images were specifically hard? For example, like we had N many objects that were very difficult to occlude, or we had some number of images where the comprehensive test was really hard. Or did you just bet that by having a large scale, you would encompass occlusion and exhaustive cases? In fact, we know we kind of maintain this kind of information exhaustivity, which one is hard, which one is easy. because first, in our data engine, when human annotates, then we exactly know which data points are exhaustively by the model, which part we need human intervene.

42:34In fact, we have that kind of metadata in our data set. The second one is that the better, the more beautiful part is we have this kind of exhaustively AI annotator. Then we can give a new data point. We can automatically decide whether this is a difficult data point or easy data point by this AI annotator. Yeah, I think the sort of bootstrapping annotation story was very strong last time around, and it's even stronger this time. What are you going to do when you run out of humans? Like, you know, next year, you're going to have superhuman level of everything, right? Like PCS and PBS. What then?

43:19I'm not so optimistic about this. And first, indeed, our current plan for Next Project is this fully automated data engine without humans. That's our dream. I would say that will. I think that is the perfect thing, but still we need some useful information. There's no free launch. There's something no model can do well, and we need humans to inject that useful information. I would say that what kind of practically can do is really minimal human intervention. Humans only do the tasks that the model cannot do, the most difficult task. So that's the kind of first one, kind of internal data engine.

44:03The second one is about human performance on this kind of PCS task. My feeling is that this kind of computer vision is going to enter this, when we get to human performance, we will enter this RAL, HF domain of computer vision. So you can see that language models before in the birth age, the language models are not human performance. SFT, really imitation learning, really do their job, get to very good performance. But if you only do SFT, and the SFT data is annotated by human, then your performance is funded by human. You cannot get superhuman performance just by this kind of data engine approach to use human annotated data and then they found that.

44:48You need to go to this RLHF domain that human really just tell which one is better. This is exactly the philosophy that to tell which one is better is easier to construct the data point from scratch. So you can get higher performance, can get better performance from human draw from scratch. I would say that, I hope that after sensory, we can see new research emerge from computer vision, which is, okay, how we go beyond human performance. Sensory is close to that, but I would say that new learning paradigm is needed to go beyond human performance for sensory tasks and for computer vision. Yeah, now just to add to that, this is Peng Shun, we're only talking about images.

45:38I think video is a whole nother challenging beast and getting to that fully automated data engine is something that we tried to do in SAM2. We actually didn't get to that fully automated approach. In SAM1, we did. We, you know, fully, as a 1B dataset that we released was fully annotated automatically. We didn't really get to that in SAM2 for video. and in SAM3 for video, I think there's still like a lot of room to push on this sort of pseudo-labeling for video and really be able to get to that same step changes we had on images. What are the biggest changes to see the same step change in video that you've seen in images for automated data pipeline?

46:19Yeah, yeah. I would say that's running kind of good video, not kind of video, not multi-model model. So when we do SAM3, is earlier this year or last year, you can see that image large multimodal model is very good, but video large multimodal model, I think really it becomes good or practical later this year. Like, kind of, this kind of model gets roughly kind of okay in that stage. So we have a good kind of base model to fine-tune our data and to get human performance for this recognition or verification task. I would say that you can see that we need definitely kind of sensory like effort in the perception side but we also need kind of this kind of multi-model large language model kind of effort kind of good foundation model on the kind of vision language side i think it's ready it's ready now yeah also video annotation is just so much more time intensive to to get to that to you may be able to annotate enough data to train a verifier like video mask annotation we just found it was like very time intensive and maybe there are more efficient video annotation strategies.

47:32I think there's a lot of exploration that could be done there too. Yeah, spending a bit of time on video, I wanted to also talk about, obviously last time we were focused a lot on memory attention. I think this time there was this sort of mask-lit thing that I wanted to just get more ideas of or just share the idea just generally. What was it called? The mask-lit detection. muscle detection score exactly um and and how it's basically smoothing within a temporal window which i think basically you know a lot of computer vision models don't have this and they could just simply add it and it'll be a lot more stable when it comes to video and i don't know why they don't do it yeah maybe i can comment on this first why they didn't do that I think one big reason is the streaming requirement.

48:26You can see when you want to gather information across the entire mass net, then you need to wait for the mass nets in the end and kind of get the statistics. So that will sacrifice some streaming capability. So you can see that the streaming requirement is somehow limits the traditional method to do this. But I would say that this is definitely kind of beneficial. The reason why is that I think even humans do this. You can imagine that when something just appears at the corner of the video, like a hand appears at the corner of the window of the video, you just do not know whether this is a man or a woman.

49:05So humans even make mistakes. Also for a sensory, it will make these mistakes. But when you get more and more information, the person really enters the video fully, then you get to know whether this is a man or a woman. So this kind of gather more information to really know whether this concept is the concept you covered is the idea here. So there is a trade-off between the latency and the accuracy here. If you care more about accuracy, then you can use this overall information across the mass net to get kind of more robust signal about the concept. But if you care about kind of latency, then you need to make a decision in the very beginning and then you will sacrifice some accuracy.

49:58I think also in many video use cases, I think, Chris, if you were sharing on RoboFlow, users care more about detecting the objects rather than having unique identities. So in some cases, maybe this isn't required to preserve the identities throughout the video and you just want to essentially do detection per frame like for the RoboFlow Rapid examples you are sharing. Yeah there's cases where being able to count and you know the objects are all going to be the same so you don't care as much about unique classes you just want to know the full presence things like that matter but then there's other cases like you mentioned where I don't know like in sport you care about individual players versus is just knowing that there's 11 players on the pitch.

50:45One thing that might be useful actually to discuss with some of our time is, we talked a little bit about how SAM3 and MLLMs will play nicely together, but there's probably like a greater discussion about how SAM3 fits into the broader AI ecosystem and like what bigger picture trends it might fit into. Do you have some thoughts on what this represents about where things are headed? Yeah, maybe I could say one point and then Pengtron feel free to add. One, you know, as we mentioned before, SAM3 isn't just a version bump. We are really having a unified model that can do many different tasks in the same unified architecture.

51:24And so, you know, in the same way that LLMs can do many different tasks without needing a task-specific model, like with SAM3, we're able to do image, promptable concept segmentation, video, promptable concept segmentation. We can do, we don't need a specialist model for counting. we can do interactivity there really is like multi-capability visual models that are on par or better than the single task state-of-the-art models so that's really one place in which SAM3 fits into the AI ecosystem in terms of MLMs I don't know if Peng Tron you want to talk about the agent approach yeah yeah definitely I would you can see let me give kind of I would see that SunSui can now really get a big step change in vision, how it really helps the general AGI fit into the general AGI or Frontier Model Landscape is very exciting for me.

52:25We always have this example, give this six-finger hands-up picture, ask how many fingers do we have in this picture, and then learn all the Frontier Model C5. And you can imagine that with Sans 3, then we can just kind of first detect how many fingers we have that very robustly, kind of six fingers. And then the multimodal model should know that, okay, this is six finger hand instead of five finger. You can see that the errors made by frontier models can be solved if we use Sans 3 as a tool. But then how really is Sans 3 as a tool at the end of the picture? or should really somehow sensory even just be more naturally embedded into these frontier models?

53:09The frontier models have really this sensory capability by themselves. I would say that there's a lot of possibilities there. My picture is that now we have a very good brain with this kind of frontier models and we have a very good eye with sensory. Now let's see whether the eye really is kind of working together, kind of natively with the brain together or eye is really kind of a different kind of uh organism kind of organ and then need to kind of somehow like a tool to kind of work with the brain i think this is very exciting kind of research area and so in your analogy if you think about like the visual cortex compared to like a human human brain like you know we have rods and cones in our eyes that do kind of very fast we joke like lizard brain level detection simple stuff and then you have your brain that reasons about some of the visual information that your eyes see.

53:59In your example of SAM3 as a tool call or SAM3 as natively a part of the multimodal models, which future do you think is more likely? I think at least I want to bet on running their work natively together. The future for simple, I would say for simple or even intermediate difficult vision tasks. For example, counting with less than 20 objects. I think for this kind of simple task, this is like system one, kind of visual reasoning with our brain. This should be our brain, and it should do it by themselves. But with very, very difficult tasks, you can see that if we are counting maybe thousands of objects in the picture, so crowded, then we even need to draw something there.

54:50I would say that at that time, maybe we need some extra model for difficult tasks. You can see that this is a hybrid approach, but I'm more excited. I think for most of the cases, it should be native. The reason why there is, you can think that I would say perception or grounding, really kind of know where it is, how many it is. It's like a fundamental capability of our brain. I'm just not happy that the frontier model just cannot count how many fingers immediately and instead of need to call a tool to do that I think this should be system one thing and this should be natively in our brain and also if our brain cannot do this task which means that it's definitely missing some very critical visual capability by itself so that's I would say that it just feels that and the intuition just feels that it's not correct to do not have this capability by itself.

55:53So for very simple system one questions, things like how many fingers on a hand, that should be native. But for maybe more complex things that are maybe long running tasks and long running reasoning, then maybe there's a bit more of like a tool call approach. Yeah, yeah. Exactly. For example, you can see that we already kind of in our sensory agents or in our AI annotator, we even demonstrate this approach. For simple cases, the model can do it by self that, okay, I can detect, for example, 10 people here. And then the large language model can even, the AI annotator can even know that, okay, this 10 people is not exhaustive.

56:30Okay, there are more people there. So if you want to do well, then maybe you need to do more steps, for example, to call an extra model. So you can see that this is a very, very native, kind of true kind of reasoning process for more advanced or complicated vision questions. I have a related but maybe slightly different question. SAM3 is an incredibly powerful piece of work and it's open source as a part of now MSL. Open source critical to achieving AGI? Maybe I can comment on SAM specifically. But in SAM3, we did leverage many of the open source contributions people have made on top of SAM2. There were new data sets, there were new benchmarks, there were new kind of inference time optimizations.

57:21we adopt a lot of the things that the community built on top of the models on top of the data sets um and so the all those contributions helped make SAM 3 for SAM series we've really benefited a lot from you know being very generous with what we open source and then leveraging what the community builds on top of that but that's just from the SAM perspective I think it's clear what the community brings and offers and i think uh you know every time we do this we always shout to the community to like you know try it on their use cases and record like weird findings and like you know if it doesn't do what you are trying to make it do well let's let's talk about it right and then maybe sort of implement it in the next version like you already said uh you already hinted at like what might be coming for sab 4 which is at least a little bit more of the document and ocr work any other directions are interesting i guess obviously a lot more video work as well What is the talk of the town in the CV community that would be really great or super obvious like next year is going to be the year of what?

58:27Yeah, maybe I can first talk something and then Nikina can add. First, definitely, I think even it's not Sun4, it's Sun3 something, Sun3 point something like small models. Sun3 currently only have really kind of one model, kind of one size model, kind of more efficient model that's going to fit for eight cases. And also kind of a more efficient model for video. I think currently kind of the video model is not efficient. You either, you can't achieve very good kind of throughput, but you need GPUs to do that. So first kind of small and efficient models. That's one big thing. The second big thing is definitely kind of video.

59:07RoboFull can do that for you. Yeah, yeah. The second thing is video. I would say that video is still far from, I would say have a big gap from human performance. Right now, there's kind of still kind of a lot of research need to be done there. How to do end-to-end training with video. We do not have, kind of, we have this kind of decoupled approach, but we do not end-to-end train this model. And we expect definitely kind of, it will be kind of a benefit from kind of end-to-end training. And also as we just kind of on video side, really kind of how to scale up the data engine. We need definitely AI annotators for video.

59:45We tried that, but yeah, I think that's something definitely worthwhile to do. The third one, we also discussed about how sensory, how perception fit into AGI, this big landscape. Now we have the eye, how the eye works with the brain to solve real reasoning tasks. not only output segmentation, but really kind of answer how many kids are here or even answer the question, okay, I have an example of biology labs. The robots need to decide whether they can liquidate in the test tube at the correct level or not. You can see that this involves perception, but also involves reasoning. How to solve this more visual reasoning task with Sun is a very big direction.

1:00:35On the robotics topic, it was exciting to hear from like several friends that work at, you know, different robotics companies on how they're like immediately starting to use SAM3. And I think especially for the video use case, I think robotics is probably one of the domains where I think improving video performance will have a lot of impact. And so I think, yeah, that's definitely an area that we could improve on further. But yeah, depending on what's point, I think there's still another step change to be achieved on video PCS. Yeah, just a quick comment on the robotics things. We're interviewing a bunch of robotics folks here, as well as like Fei-Fei Li, who obviously started ImageNet.

1:01:16A lot of people are betting on explicit world models and Sam is not, for better or worse. And I wonder when that crossover might happen. That's an open question if you guys want to take any world models discussions. re-where things are going based on like community questions similar to how nakila mentioned dr sam one the like almost obvious thing that people wanted was like open concepts prompting because people are like great this model can see things but i want to tell it what i want it to see and now with the introduction of sam three you have this stepwise component which feels like a key component of you know the chat gpt era for vision is arriving as a result what's going to happen is now you've provided people with an open text box and media.

1:02:00And so you're going to get all sorts of queries from people that maybe the model isn't primed to be able to perform particularly well on yet. For example, earlier we were talking about document understanding and document reasoning being a place where there's known improvements to be made. And so you'll have people that will probably prompt to try to OCR things, or you'll have people that will want to do work with spatial reasoning, like give me the object to the left of this other object, or give me a sense of where things are in relation to one another, which is critical for robotics like we were discussing, because that's how you navigate throughout the real world.

1:02:32You'll also have, I think, people will want action recognition and vision language action models, VLAs, like the same things that where you have these tasks where people are used to providing open text prompts and getting here's the part of the scene where the player kicked the ball or the tennis player made the serve. Those are interesting for the purposes of how to understand and synthesize visual inputs. And so now that you've kind of given this open text box for media, there's going to be a flood of the types of things users are going to want to try to do, some of which SAM is already going to be really well adapted to do, some of which not.

1:03:06And I think that that's going to be, it's going to reveal itself of the types of things that are obvious. One of the things that we wanted to discuss was like where to use SAM and discover how to build with SAM. So in addition to the meta team building a tremendous playground for being able to interact with images and video and kind of apply effects with like a video emphasis. I think one of the things that we're pretty excited about with SAM3 is how much it positively impacts each part of building a system for visual understanding. So for example, the very first step of historically aggregating and collecting a data set, because you think that there's not a model that understands the slice of the world that you want to understand, is where automating away lots of labeling can exist.

1:03:49Basically, if you collected a bunch of data of something that is already in the SAM3's knowledge, then you can prompt for SAM3 to automatically label all that data for you. And so we've actually made a bet on SAM3 being a core part of auto-label at Roboflow, giving users a first pass of saying, hey, if you have a new image or you have a new video, start providing just a text prompt and allow SAM3 to find and automatically label those regions of interest for you. Downstream, I think there's areas for fine-tuning, like, you know, within a week of releasing SAM3, Med SAM3 came out for adapting SAM into medical contexts.

1:04:27And I think that's a harbinger of what's to come. Like there'll be lots of domain-specific adaptations of SAM in places where maybe there's a specific ontology that someone wants to understand, or maybe there's a place where just the model doesn't have great awareness yet. And I think we're already beginning to see that with hundreds of fine tunes that users are creating for various domains. And then the last area is like, okay, I've got my model and I want to use it. And so one of the things that we're really proud of is to be ready on launch day to showcase the infrastructure we've built to burst and scale like infinitely large as folks have models that they want to deploy and make it readily available.

1:05:02Having an endpoint that serves either a fine-tuned model or a model as is, or even a model that might be able to run on edge hardware as smaller models come out or maybe distillation comes to rise is I think also an awesome place of where we're seeing SAM3 being impactful at each part of like the computer vision lifecycle and pipeline. That's awesome. Yeah, I think especially the impact on speeding up annotation, I think we've seen that consistently on RoboFlow. And I'm really curious to see how SAM3, the introduction of SAM3 really helps speed up that process even further. I mean, just from playing around with it, it's so much faster than having to manually annotate every single object.

1:05:41So yeah, you're really curious to see how that improves the experience. One of the things that we were pretty excited about is we were kind of able to build an entirely new product in the world of SAM 3. And we called it rapid, but basically it's like, there's probably a model that already understands the objects in the world that you want to see. So here I'm sharing an example of these are vehicles next to our office in San Francisco that go by. And you can see here's a Waymo and here's like other vehicles. And like, if I just have like this 10 second clip and let's say, you know, the first thing I want to do maybe is just like count cars and I want to get a sense of like each of the vehicles.

1:06:18What's really awesome is I can just, you know, of course, text prompt and say, I want vehicle. And as I toggle through different frames in my video, SAM3 already recognizes and understands those objects. Now, one thing that I think is really interesting. There was a conversation earlier about how much you want to rely on a model versus humans output of the model for what you care about. So for example, let's pretend in this scene, maybe the only cars that we care about are the ones that are like before the crosswalk and maybe not far in the distance. Then you'd get people that would say, hey, you know what?

1:06:53I actually want the objects that are like most confident. And I would like, you know, move my slider down to like getting a fewer number of objects. Whereas maybe others might say, hey, I want like every single presence of a potential object in the scene, which even gets like reflections on the building of objects. As computer vision approaches this world where we increasingly have like models that can understand and improve themselves, and we rely on what human output and human preference from the models is, we're going to get these funny scenarios where things aren't all like immediately deterministic of what a human cares about.

1:07:24And I think that's where like tooling fills a big gap, but it also is going to be a place where it'll be really interesting to see where users kind of start to use and apply the models and why you need sort of this last mile work to put the model in context, in the domain that someone is trying to solve and tackle. So let me, since you're here, right? This is one of those things where I'm like, I'm not sure this concept, concept, the concept of labeling concepts can scale only because I don't know if I ever, if this slider between less and more is the way if ultimately I need to tell you whether or not to include reflections, right?

1:08:06Because in reflections, sometimes it's great. That's exactly what I want. Most of the time it's not going to be what I want. I don't know if some RLHF thing is going to solve any of that because you just need more prompting. Just saying vehicle is not going to do it. Yeah, I don't know. Feel free to disagree. can you imagine such pipeline coming, for example, as Swick said, that maybe the reflection is exactly what I want, then you need some iterations with the interface or the model to get finally what you need. So you need to specify the concepts more clearly through multiple iterations. Can humans not be involved in this iteration, but just models in just kind of do it automatically.

1:08:57I think that's kind of something that definitely kind of, it's, I would say I'm quite interested in that you can imagine this workflow and then I want kind of reflections and then I can kind of, with the kind of default kind of threshold, maybe kind of, the kind of model will get that output. Then another kind of very strong kind of perception model on another kind of, like kind of Gemini 3 will then kind of ask, we asked Gemini whether there's some reflections there. And it says yes. Then you can see that we can automatically move the threshold again lower. And we're going to ask Gemini again to see whether the reflections are included or not.

1:09:39So somehow this process principally should be done completely with AI. I see. Yeah, yeah, exactly. So for now, the answer is image. and we can we can sort of tie it closer i think i think joseph is showing out the sort of way more annotation yeah it's it's nice now you have a way more model okay yeah i was just doing an example where maybe we want to find an object that's not already represented in the training data i think i think prompting can solve yeah i think prompting could solve the problem of like reflections because maybe you could say like vehicles on the street but to your point like you would have to like see that that's a failure case right like if i was like just setting up a camera and saying count cars i wouldn't anticipate realizing that reflection could be a problem and so i think this is why like in some ways human in the loop because identifying human intention not necessarily human knowledge is what's going to be important for a lot of last mile use but yeah i'm i'm pretty excited about yeah maybe i want to echo kind of what joseph said And also my experience, just different people have quite different definition of even a visual concept.

1:10:57For example, for some kind of data set, even hand, some people would like to just annotate their palm part as their hand. And some people will include their arm also as hand. Then when we first test that through on some very customized data set, we found, okay, their performance is not that good. And when we kind of finally look into kind of the kind of performance, we found, okay, this is kind of just the user have a different definition or explanation of the concepts. But kind of both explanations are okay. Then in this case, you can see that running lead a few might in the loop to do the kind of few short fine tuning or to adapt to the user's definition of this concept.

1:11:39That's exactly right. It's not always like deterministic of what someone really wants, which is why I think like, Even if you have a fully comprehensive omniscient model, putting the model into the context of what the user is trying to do is where a lot of tooling and infrastructure becomes really, really helpful. Anyway, I found our Waymos. You continue to do excellent tooling for vision, and I think the world is very grateful for that. Let's get to call to action. And, you know, I think, you know, we've sort of given a good overview. People obviously should read the paper and try out the playground, try out RoboFlow.

1:12:16If they're interested in diving deeper, what is your call to action from each of you? I mean, try the demo, try the code. We've got a lot of resources on GitHub repo. And, you know, it's a very well managed launch, by the way. Like, kudos. I don't know. It probably takes a lot of effort just on the launch itself, even after the model's done. yeah and actually just on that maybe one thing just shout out to the whole team i think this was our biggest and most ambitious project to date and it really took a huge team of scientists engineers interns software engineers you know across across the company so a really huge shout out to the entire team that made not just the model successful but also the demo and then all the the launch and everything so it was a huge team effort definitely like would love to hear from people on what you're using the models for, where it's failing, you know, raise GitHub issues, message us on Twitter.

1:13:12We'd love to hear from you on where we should go next as well. Yeah, and on top of that, definitely going to try out also our benchmark, the cycle benchmark. I would say that it's likely that the benchmark will last longer than our Samsung model. Maybe next year, there will be a stronger model, but the benchmark is kind of the one that I hope to guide the community to kind of get better and better models kind of to get to a kind of way measure human performance on the benchmark i think maybe we are the first one to do that for this kind of very kind of segmentation and the kind of video or kind of grounding in the past it's very difficult to measure human performance on this task hopefully kind of this benchmark and guides the community to achieve human performance for this task and even going to surpass human performance there we uh we set out to be one of the best places, if not the best place to build with SAM3 and the SAM family models.

1:14:08So we're eager to see what people build with SAM and computer vision models to move the whole field forward. We have infrastructure for everything from deploying SAM3 ZeroShot, to making your own fine tunes, to automating labeling of data with SAM. And we continue to see the impact with each subsequent release, expand the number of use cases and the amount of use and accelerate the time to value. So excited to see what folks can build on RoboFlow with SAM. Thank you all so much. This is a really great conversation. It's great work. And just obviously, it always expands my mind as to what is possible with machine learning.

1:14:42Yeah, I mean, we're not at ASI yet or AGI yet, but every day we're getting closer. Awesome. Thank you so much. Thank you. Thank you.

1:15:01Thank you.