SAM 3: The Eyes for AI — Nikhila & Pengchuan (Meta Superintelligence), ft. Joseph Nelson (Roboflow)

31:40In fact, I was going to review it even as a question. So without, you can imagine that without large language models, without VOM, only for reason's sake, it only achieves about, on the validation set, if I remember it correctly, it only achieves 30 numbers there. And also, it's very intuitive. You can see that for reason's sake, it has this short none, untested. it has kind of different subsets, short, none. Short, then it's very close to sensory training data. Like it's kind of atomic phrases, short phrases. None is this kind of very kind of complex reasoning. You will see that's kind of, for short, sensory only is very close to kind of the sensory agents.

32:30But for none, the gap is so large, which indicates that, okay, that is exactly kind of the capabilities that large language model bring in. Got it. I can share an example here that might be insightful too go for it so even comparing even comparing like sam3 and gemini let's say that we just want to have them do like a an optic detection task here of finding here we're going to prompt with a speedometer and rpms and we're going to ask for things like indicator light number and needle and if we run sam3 head-to-head with gemini 3 and florence 2 almost as a baseline of like where things have been and we see each of the results first things first you'll note that the speed of inference of SAM3 is quite quick.

33:17This is just calling the Gemini 3 Pro API. So whatever is provided from hosted compute is sort of what you get on the response time. And then the second thing you'll note is, in addition to speed, is some of the accuracy of results. Ooh, we might have a timeout error. Let's see. Do you have ELO scores? I have what scores? ELO scores. like ELO. Yeah, you had the arena. Okay. I was wondering what the ELO was, because you said you were blind testing this. Yeah, that's actually interesting, because we had blind tested SAM3 before it was released, not a SAM3, just for people to try and compare. I think we call it like a potential SAG or SAG preview or something.

34:03And we allowed users to vote and they kind of unanimously voted for what they didn't know at the time was was sam3 we actually got like emails of people being like hey like where can i use that and we just sort of ignored them until the model came out but so here with with the responses you see that um the uh grounding capabilities of of sam and uh sam3 compared to even gemini uh are out ahead currently so not only is it doing grounding but if you look closely you can actually see it's making segmentation masks too whereas um uh gemini 3 struggles to do it just does detection by comparison um and then the other thing is just the uh richness of detections like the recall is is high as well as the precision and if we compare here it does almost as well right but you see that it misses some of the numbers and has kind of these some of these erroneous boxes that it's that is predicted.

35:00And then it also doesn't do segmentation. So it just does detection of the task. So you can envision that the same way the SAM3 paper introduces the idea of using SAM3 in tandem with MLMs. I would expect that to be the case pretty soon. And maybe the Google team taking some notes to improve Gemini and other series of models based on what SAM3 demonstrates here. So in other words, not only is it faster, but it seems to be more comprehensive for concept segmentation. And I think the speed actually is a huge factor for many use cases. I think like even Meta, we're using SAM3 for various different product use cases and fast inference speed is very critical to enable that.

35:49And so I think that's something that I think in many cases you don't even need an MLM for. It's just kind of overkill to use in MLM to some applications. The other interesting thing is the Florence 2 results. And, you know, Florence 2 is a little bit older of a model now, so maybe it's not fair to put up head-to-head with the state of the art, but it is useful as a way to just see how far we've come. Because Florence 2, by comparison, labels the entire region as a single class without seeing individual detection of numbers and indicator lights and needle. And not only that, but it actually runs at about three times the speed as SAM3.

36:27So SIM3, again, is faster doing a task that the other models are not doing in segmentation and more accurate, both in recall and precision of the things that it's intended to find, which I think really showcases the capabilities of the model. In fact, I even got a little surprise about this because this domain, this more like an OCR-like because recognition numbers is nearly OCR. We do not prioritize this domain of data collection. it works. So we know that it roughly works, but I think I got surprised that it works so well. That's encouraging. Even a task that wasn't expressly prioritized, it still does a great job on.

37:10Yeah. In fact, during our data engine, we intentionally do not sample OCR-heavy images. Wow. On an easier one, glass mug, SAM3, Gemini 3, Florence 2, SAM3 loaded first and has, really impressively, it sees even this glass mug in the corner, which I think is something SAM3 does a great job of, is occlusion and partial objects. Gemini 3 struggles a bit with this one, I think maybe because the opacity of the objects by comparison. And then Florence 2 does a good job at finding one of the glass mugs so again another type of task that shows the power and veracity of the model yeah i mean exhaustivity like finding every instance is something we heavily prioritized and is really built into the data engine design um you know you want to talk about how we design the data engine to really scale exhaustivity because you know if a human was to say and rotate every single instance it would take a really long time and and verify but we put a lot of effort into trying to to automate and speed up that process that such that we could get to the data scale and diversity needed to get to a step change yeah yeah i think definitely gonna i would say data engine is say you know the critical component that we achieve sensory performance like now.

38:41So maybe we can go to the data engine picture. I think we have an illustration there. Yeah, page five. Yeah, yeah, yeah. Here. You can see that this is our annotation pipeline. So we first source the images and they generate the non-phases. So this is the input of this task. Source images and it generates non-phases from, for example, Nama generate caption and we pass the caption to get the long faces. This is the input distribution. Then we use San Sui model in the loop to generate candidates masks. That should be the candidate, but it's not perfect, especially in the beginning. Then we go to, you can say, go to the next step is verification.

39:30So San Sui gave you this mask, then we need to first do mask verification to verify each mask, whether it's good or not. And then after we filter all the bind masks, there are some good masks left, and we verify whether these good masks are exhaustive or not, like your mug example. So for example, the bind model do not predict that's partial mug, then the exhaustivity check will be failing there. Then if the exhaustivity is failed, Then we go to the next step. You can see that we can go to the pipeline, go to this so-called human manual correction. Human manually annotates all these missing masks.

40:16You make this data point exhaustive. So you can see that exhaustivity is a very big factor there, and we play it as the center place in this data engine. But you can see that if we ask human annotator to annotate every mask from scratch, it will take a lot of time. I remember each data point in the beginning will take about more than two minutes to finish. But if you use model in the loop, then it's reduced to about 45 seconds. You use model to propose mask and then just a few months to annotate the missing mask. Then it's 45 minutes. Another very key innovation in this data engine is that we really found that these verification steps, like to verify a mass is good or not, or to verify now the good mass are exhaustive or not, can be done by AI, can be done by that multimodal model.

41:14That is a breakthrough, and then we kind of fine-tune, for example, NAMAS 3.2 with our verification human annotated verification data. We get superhuman performance on these two verification tasks. And then we do not need a human on this two tasks. This further brings our per data point annotation time to about 25 seconds. So you can see that from the original all human to about two minutes to finally 25 minutes for one data point. This is the journey of our data engine to make it super efficient. Did you maintain statistics on how many images were specifically hard? For example, like we had n many objects that were very difficult to occlude, or we had some number of images where the comprehensive test was really hard.

42:09Or did you just bet that by having a large scale, you would encompass occlusion and exhaustive cases? In fact, we know we kind of maintain this kind of information exhaustivity, which one is hard, which one is easy. Because first, in our data engine, when human annotates, then we exactly know which data exhaustivity by the model, which part we need human intervene. In fact, we have that kind of metadata in our data set. The second one is that the better, the more beautiful part is we have this kind of exhaustive AI annotator. Then we can kind of give a new data point and we can automatically decide whether this is a difficult kind of data point or easy data point by this AI annotator.

42:57Yeah, I think the sort of bootstrapping and annotation story was very strong last time around and it's even stronger this time. What are you going to do when you run out of humans? Like, you know, next year, you're going to have superhuman level of everything, right? like PCS and PVS, what then?

43:19I'm not so optimistic about this. And first, indeed, our current plan for Next Project is this fully automated data engine without humans. That's our dream. I would say that will, I think that is the perfect thing, but still we need some useful information. There's no free launch. there's kind of something kind of no model can do well and we need human to inject that useful information. I would say that what kind of practically can do is really minimal human intervention. Human only do the tasks that the model cannot do, the most difficult task. So that's the kind of first one, kind of internal data engine.

44:03The second one is about human performance on this kind of PCS task. My thing is that this kind of computer vision is going to enter this and when we get to human performance we will enter this ral hf domain of computer vision so you can see that language models kind of before kind of in the birth age and the language model are not human performance can sft kind of really imitation learning really do their job get to very good performance but if you only do sft and the SFT data is annotated by human, then your performance is funded by human. You cannot get superhuman performance just by this kind of data engine approach to use human annotated data and then learn from that.

44:48You need to go to this RLHF domain that human really just tell which one is better. This is exactly the philosophy that to tell which one is better is easier to construct the data point from scratch. So you can get kind of higher performance, can kind of get a better performance from human to from scratch. I would say that kind of, I hope that after sensory, we can see kind of new research emerge from kind of in computer vision, which is, okay, how we go beyond human performance. Sensory is close to that, but I would say that new learning paradigm is needed to go beyond human performance for sensory tasks and for computer vision.

45:33Yeah, and just to add to that, Peng Shun is only talking about images. I think video is a whole other challenging beast and getting to that fully automated data engine is something that we tried to do in SAM2. We actually didn't get to that fully automated approach. In SAM1, we did. We, you know, fully, our SA1B dataset that we released was fully annotated automatically. We didn't really get to that in SAM2 for video. And in SAM3 for video, I think there's still like a lot of room to push on this sort of pseudo-labeling for video and really be able to get to that same step changes we had on images.

46:12What are the biggest changes to see the same step change in video that you've seen in images for automated data pipeline? Yeah, yeah. I would say that really kind of good video large language kind of video large multimodal model. So when we do San Sui it's kind of earlier this year or kind of last year you can see that image large multimodal model is very good but video large multimodal model I think really kind of it become good or practical later this year like kind of Quincy this kind of model gets kind of roughly kind of okay in that stage. So we have a good kind of base model to fine-tune our data and to get human performance for this recognition or verification task.

46:56I would say that you can see that we need definitely kind of sensory-like efforts in the perception side, but we also need this kind of multi-model, large-language model kind of efforts, kind of good foundation model on the vision language side. I think it's ready. It's ready now. Yeah, also video limitation is just so much more time-intensive to get to that, to be able to annotate enough data to train a verifier, like video mask annotation, we just found it was like very time intensive. So maybe there are more efficient video annotation strategies. I think there's a lot of exploration that could be done there too.

47:36Yeah. You know, spending a bit of time on video, I wanted to also talk about, you know, obviously last time we were focused a lot on memory attention. I think this time there was this sort of masklet thing that I wanted to just like get more ideas of or just share the idea just generally. What was it called? The masklet detection. Masklet detection score, exactly. And how it's basically smoothing within a temporal window, which I think basically, you know, a lot of computer vision models don't have this. And they could just simply add it and it'll be a lot more stable when it comes to video. And I don't know why they don't do it.

48:15Maybe I can comment on this first. Why they didn't do that? I think one big reason is kind of the streaming requirement. You can see when you want to gather information across the entire mass net, then you need to wait for the mass nets in the end and kind of get the statistics. So that will sacrifice some streaming kind of capability. So you can see that the streaming requirement somehow limits the traditional method to do this. But I would say that this is definitely beneficial. The reason why is that I think even humans do this. You can imagine that when something just appears at the corner of the video, like a hand appears at the corner of the window of the video, you just do not know whether it's a man or a woman.

49:05So humans even make mistakes. also for a sensor, it will make these mistakes. But when you get more and more information, the person really enters the video fully, then you get to know whether this is a man or a woman. So this kind of gather more information to really know whether this concept is the concept you covered. Is the idea here? So there is a trade-off between the latency and accuracy here. If you care more about accuracy, then you can use this overall information across the mass net to get more robust signal about the concept. But if you care about latency, then you need to make a decision in the very beginning, and then you will sacrifice some accuracy.

49:57I think also in many video use cases, I think, Chris, if you were sharing on RoboFlow, users care more about detecting the objects rather than having unique identities. So in some cases, maybe this isn't required to preserve the identities throughout the video. You just want to essentially do detection per frame, like the RoboFlow Rapid examples you were sharing. Yeah, there's cases where being able to count and you know the objects are all going to be the same, so you don't care as much about unique classes. You just want to know the full presence. Things like that matter. But then there's other cases, like you mentioned, where, I don't know, like in sport, you care about individual players versus just knowing that there's 11 players on the pitch.

50:45One thing that might be useful actually to discuss with some of our time is we talked a little bit about how SAM3 and MLLMs will play nicely together, but there's probably like a greater discussion about how SAM3 fits into the broader AI ecosystem and like what bigger picture trends it might fit into. Do you have some thoughts on what this represents about where things are headed? Yeah, maybe I could say one point and then Pengchon feel free to add. one you know as we mentioned before 3 isn't just a version bump we are really having a unified model that can do many different tasks in the same unified architecture and so you know in the same way that llms can do many different tasks without needing a task specific model like with sam3 we're able to do image promptable concept segmentation video promptable concept segmentation We don't need a specialist model for counting.

51:42We can do interactivity. There really is multi-capability visual models that are on par or better than the single task state-of-the-art models. So that's really one place in which SAM3 fits into the AI ecosystem. In terms of MLMs, I don't know if Peng Tran, you want to talk about the agent approach? Yeah, yeah, definitely. I would, you can see... Let me give, I would say that SunSui now really get a big step change in vision. How it really helps the general AGI fit into the general AGI or Frontier Model landscape is very, very exciting for me. We always have this example, kind of give this kind of six-finger hands-up picture, ask how many fingers do we have in this picture, and then learn all the Frontier Model C5.

52:35And you can imagine that with SunSui, then we can just kind of first detect how many fingers we have that very robustly, kind of six fingers. And then the multimodal model should know that, okay, this is six finger hand instead of five finger. You can see that the errors made by frontier models can be solved if we use SunSui as a tool. But then how really is Sans3 as a tool at the end of the picture? Or should really somehow Sans3 even just be more naturally embedded into these frontier models? The frontier models have really this Sans3 capability by themselves. I would say that there's another possibility there.

53:17My picture is that now we have a very good dream with these kind of frontier models. And we have a very good eye with Sans3. Now let's see kind of whether the eye really kind of is kind of working together kind of natively with the brain together or eye is really kind of a different kind of organism and kind of organ and then need to kind of somehow like a tool to kind of work with the brain. I think this is a very exciting kind of research area. And so in your analogy, if you think about like the visual cortex compared to like a human brain, like, you know, we have rods and cones in our eyes that do kind of very fast, we joke like lizard brain level detection, simple stuff.

53:55and then you have your brain that reasons about some of the visual information that your eyes see. In your example of SAM3 as a tool call or SAM3 as natively a part of the multimodal models, which future do you think is more likely? I think at least I want to bet on running their work natively together. The future for simple, I would say for simple or even intermediate difficult vision tasks. For example, counting with less than 20 objects. I think for this kind of simple task, this is like system one, kind of visual reasoning with our brain. This should be kind of our brain, kind of should do it by themselves.

54:39But with very, very difficult task, you can see that if we are counting maybe thousands of objects in the picture, so crowded, then we kind of even need to kind of draw something there. I would say that at that time, maybe we need some extra model for difficult tasks. You can see that this is a hybrid approach, but I'm more excited. I think for most of the cases, it should be native. The reason why there is, you can think that I would say perception or grounding, really kind of know where it is, how many it is. It's like a fundamental capability of our brain. I'm just not happy that the frontier model just cannot count how many fingers immediately and instead of need to call a tool to do that I think this should be system one thing and this should be natively in our brain and also if our brain cannot do this task which means that it's definitely missing some very critical visual capability by itself so that's I would say that it just feels that and the intuition just feels that it's not correct to do not have this capability by itself.

55:53So for very simple system one questions, things like how many fingers on a hand, that should be native. But for maybe more complex things that are maybe long running tasks and long running reasoning, then maybe there's a bit more of like a tool call approach. Yeah, yeah. Exactly. For example, you can see that we already kind of in our sensory agents or in our AI annotator, we even demonstrate this approach. For simple cases, the model can do it by self that, okay, I can detect, for example, 10 people here. And then the large language model can even, the AI annotator can even know that, okay, this 10 people is not exhaustive.

56:30Okay, there are more people there. So if you want to do well, then maybe you need to do more steps, for example, to call an extra model. So you can see that this is a very, very native, kind of true kind of reasoning process for more advanced or complicated vision questions. I have a related but maybe slightly different question. SAM3 is an incredibly powerful piece of work and it's open source as a part of now MSL. Open source critical to achieving AGI? Maybe I can comment on SAM specifically. But in SAM3, we did leverage many of the open source contributions people have made on top of SAM2. There were new data sets, there were new benchmarks, there were new kind of inference time optimizations.

57:21we adopt a lot of the things that the community built on top of the models, on top of the datasets. And so all those contributions helped make SAM 3. For SAM series, we've really benefited a lot from being very generous with what we open source and then leveraging what the community builds on top of that. But that's just from the SAM perspective. I think it's clear what the community brings and offers. And I think, you know, every time we do this, we always shout to the community to like, you know, try it on their use cases and report like weird findings. And like, you know, if it doesn't do what you are trying to make it do, well, let's talk about it, right?

58:04And then maybe sort of implement it in the next version. Like you already said, Pengchuan, you already hinted at like what might be coming for SAB4, which is at least a little bit more of the document and OCR work. Any other directions are interesting? I guess obviously a lot more video work as well. What is the talk of the town in the CV community that would be really great or super obvious like next year is going to be the year of what? Yeah, maybe I can first talk something and then Nikina can add. First, definitely, I think even it's not Sun4, it's Sun3 something, Sun3 point something like small models.

58:39Sun3 currently only have really kind of one model, kind of one size model, kind of more efficient model that's going to fit for eight cases and also kind of a more efficient model for video. I think currently the video model is not efficient. You can't achieve very good throughput, but you need GPUs to do that. So first, kind of small and efficient models, that's one big thing. The second big thing is definitely kind of video. RoboFull can do that for you. Yeah. The second thing is video. I would say that video is still far from, I would say, have a big gap from human performance. Right now, there's kind of still kind of a lot of research need to be done there.

59:24How to do end-to-end training with video. We do not have, kind of, we have this kind of decoupled approach, but we do not end-to-end train this model. And we expect definitely kind of, it will be kind of a benefit from kind of end-to-end training. And also, as we just kind of, on video side, really kind of how to scale up the data engine, We need definitely AI annotators for video. We tried that, but yeah, I think that's something definitely worthwhile to do. The third one, we also discussed about how sensory, how perception fit into AGI, this big landscape. Now we have the eye, how the eye works with the brain to solve real reasoning tasks.

1:00:06not only output segmentation, but really kind of answer how many kids are here or even answer the question, okay, I have an example of biology labs. The robots need to decide whether they can liquid in the test tube at the correct level or not. You can see that this involves perception, but also involves reasoning. How to solve this more kind of visual reasoning task with Sun is a very big direction. On the robotics topic, it was exciting to hear from like several friends that work at, you know, different robotics companies on how they're like immediately starting to use SAM3. And I think especially for the video use case, I think robotics is probably one of the domains where I think improving video performance will have a lot of impact.

1:00:56And so I think, yeah, that's definitely an area that we could improve on further. But yeah, depending on what's point, I think there's still another step change to be achieved on video PCS. Yeah, just a quick comment on the robotics things. We're interviewing a bunch of robotics folks here, as well as like Fei-Fei Li, who obviously started ImageNet. A lot of people are betting on explicit world models and Sam is not, for better or worse. And I wonder when that crossover might happen. That's an open question if you guys want to take any world models discussions. re-where things are going based on like community questions similar to how nakila mentioned dr sam one the like almost obvious thing that people wanted was like open concepts prompting because people are like great this model can see things but i want to tell it what i want it to see and now with the introduction of sam three you have this stepwise component which feels like a key component of you know the chat gpt era for vision is arriving as a result what's going to happen is now you've provided people with an open text box and media.

1:02:00And so you're going to get all sorts of queries from people that maybe the model isn't primed to be able to perform particularly well on yet. For example, earlier we were talking about document understanding and document reasoning being a place where there's known improvements to be made. And so you'll have people that will probably prompt to try to OCR things, or you'll have people that will want to do work with spatial reasoning, like give me the object to the left of this other object, or give me a sense of where things are in relation to one another, which is critical for robotics like we were discussing, because that's how you navigate throughout the real world.

1:02:32You'll also have, I think, people will want action recognition and vision language action models, VLAs, like the same things that where you have these tasks where people are used to providing open text prompts and getting here's the part of the scene where the player kicked the ball or the tennis player made the serve. Those are interesting for the purposes of how to understand and synthesize visual inputs. And so now that you've kind of given this open text box for media, there's going to be a flood of the types of things users are going to want to try to do, some of which SAM is already going to be really well adapted to do, some of which not.

1:03:06And I think that that's going to be, it's going to reveal itself of the types of things that are obvious. One of the things that we wanted to discuss was like where to use SAM and discover how to build with SAM. So in addition to the meta team building a tremendous playground for being able to interact with images and video and kind of apply effects with like a video emphasis. I think one of the things that we're pretty excited about with SAM3 is how much it positively impacts each part of building a system for visual understanding. So for example, the very first step of historically aggregating and collecting a data set, because you think that there's not a model that understands the slice of the world that you want to understand, is where automating away lots of labeling can exist.

1:03:49Basically, if you collected a bunch of data of something that is already in the SAM3's knowledge, then you can prompt for SAM3 to automatically label all that data for you. And so we've actually made a bet on SAM3 being a core part of auto-label at RoboFlow, giving users a first pass of saying, hey, if you have a new image or you have a new video, start providing just a text prompt and allow SAM3 to find and automatically label those regions of interest for you. Downstream, I think there's areas for fine tuning, like, you know, within a week of releasing SAM3, Med SAM3 came out for adapting SAM into medical contexts.

1:04:27And I think that's a harbinger of what's to come. Like there'll be lots of domain specific adaptations of SAM in places where maybe there's a specific ontology that someone wants to understand, or maybe there's a place where just the model doesn't have great awareness yet. And I think we're already beginning to see that with hundreds of fine tunes that users are creating for various domains. And then the last area is like, okay, I've got my model and I want to use it. And so one of the things that we're really proud of is to be ready on launch day to showcase the infrastructure we've built to burst and scale like infinitely large as folks have models that they want to deploy and make it readily available.

1:05:02Having an endpoint that serves either a fine-tuned model or a model as is, or even a model that might be able to run on edge hardware as smaller models come out or maybe distillation comes to rise is I think also an awesome place of where we're seeing SAM3 being impactful at each part of like the computer vision lifecycle and pipeline. That's awesome. Yeah, I think especially the impact on speeding up annotation, I think we've seen that consistently on RoboFlow. And I'm really curious to see how SAM3, the introduction of SAM3 really helps speed up that process even further. I mean, just from playing around with it, And it's so much faster than having to manually annotate every single object.

1:05:41So, yeah, you're really curious to see how that improves the experience. One of the things that we were pretty excited about is we were kind of able to build an entirely new product in the world of SAM3. And we called it Rapid, but basically it's like there's probably a model that already understands the objects in the world that you want to see. So here I'm sharing an example of like these are vehicles next to our office in San Francisco that go by. and you can see here's a Waymo and here's like other vehicles. And like, if I just have like this 10 second clip and let's say, you know, the first thing I want to do maybe is just like count cars and I want to get a sense of like each of the vehicles.

1:06:18What's really awesome is I can just, you know, of course, text prompt and say, I want vehicle. And as I toggle through different frames in my video, SAM3 already recognizes and understands those objects. Now, one thing that I think is really interesting there was a conversation earlier about how much you want to rely on a model versus human's output of the model for what you care about so for example let's pretend in this scene maybe the only cars that we care about are the ones that are like before the crosswalk and maybe not far in the distance then you would get people that would say hey you know what i actually want the objects that are like most confident and i would like you know move my slider down to like getting a fewer number of objects whereas maybe others might say hey i want like every single presence of a potential object in the scene which even gets like reflections on the building of objects as computer vision approaches this world where we increasingly have like models that can understand and improve themselves and we rely on what human output and human preference from the models is we're going to get these funny scenarios where things aren't also all like immediately deterministic of what a human cares about and i think that's where like tooling fills a big gap but it also is going to be a place where it'll be really interesting to see where users kind of start to use and apply the models and why you need sort of this last mile work to put the model in context in the domain that someone is trying to solve and tackle so let me let me since you're here right this is one of those things where i'm like i'm not sure this concept concept the concept of labeling concepts can scale only because i don't know if i ever if this slider between less and more is the way if uh ultimately i need to tell you whether or not to include uh reflections right because the reflections sometimes is great that's exactly what i want most of the time it's not going to be what i want i don't know if some rlhf thing is going to solve any of that because you just need more prompting.

1:08:19Just saying vehicle is not going to do it. Yeah, I don't know. Feel free to disagree. Can you imagine such pipeline coming, for example, as kind of Swick said, that maybe the reflection is exactly what I want. Then you need some kind of iterations with the interface or the model to get finally what you need. So you need to specify the concepts kind of more clearly through kind of multiple iterations. Can humans not be involved in this iteration, but just kind of models just kind of do it automatically? I think that's kind of something that definitely kind of, it's, I would say I'm quite interested in that you can imagine this workflow.

1:09:04And then I want kind of reflections and then I can kind of, with the kind of default kind of threshold, maybe kind of the kind of model will get that output. Then another very strong perception model, like Gemini's story, will then ask whether there's some reflections there. And it says yes. Then you can see that we can automatically move the threshold lower, and ask Gemini again and again to see whether the reflections are included or not. So somehow this process possibly should be done completely with AI. Kind of... I see, I see. Yeah, yeah, exactly. So for now, the answer is image. And we can sort of tie it closer.

1:09:56I think Joseph is showing us the sort of Waymo annotation. Yeah, it's nice. Now you have a Waymo model. Yeah, I was just doing an example where maybe we want to find an object that's not already represented in the training data. i think i think prompting can solve yeah i think prompting could solve the problem of like reflections because maybe you could say like vehicles on the street but to your point like you would have to like see that that's a failure case right like if i was like just setting up a camera and saying count cars i wouldn't anticipate realizing that reflection could be a problem and so i think this is why like in some ways human in the loop because identifying human intention not necessarily human knowledge is what's going to be important for a lot of last mile use but yeah i'm i'm pretty excited about yeah maybe i want to echo kind of what joseph said actually also my experience just different people have quite different kind of definition of even a visual concept for example for some kind of data set even hand and some people would like to just annotate the palm part as their hand.

1:11:05And some people will include the arm also as hand. Then when we first test that through on some very customized data set, we found, okay, the performance is not that good. And when we finally look into the performance, we found, okay, this is just the user have a different definition or explanation of the concepts. But both explanations are okay. Then in this case, you can see that really need a few might in the loop to do the kind of few short fun tuning or to adapt to the user's definition of this concept that's exactly right it's not always like deterministic of what someone really wants which is why i think like even if you have a fully comprehensive omniscient model putting the model into the context of what the user's trying to do is where a lot of tooling and infrastructure becomes really, really helpful.

1:11:57Anyway, I found our Waymos. You continue to do excellent tooling for vision, and I think the world is very grateful for that. Let's get to call to action. I think we've sort of given a good overview. People obviously should read the paper and try out the playground, try out RoboFlow. If they're interested in diving deeper, what is your call to action from each of you? i mean try the demo try the code we've got a lot of resources on github repo um and you know it's a very well managed launch by the way like kudos i don't know this probably takes a lot of effort just on the launch itself even after the model's done yeah and actually just on that maybe one thing just shout out to the whole team i think this is um three was our biggest and most ambitious project to date and it really took a huge team of scientists engineers interns software engineers across the company.

1:12:54So really huge shout out to the entire team that made not just the model successful, but also the demo and then all the launch and everything. So it was a huge team effort. Definitely like would love to hear from people on what you're using the models for, where it's failing, raise GitHub issues, message us on Twitter. We'd love to hear from you on where we should go next as well. Yeah, and on top of that, definitely going to try out also our benchmark, the cycle benchmark. I would say that it's likely that the benchmark will last longer than our Samsung model. Maybe next year there will be a stronger model, but the benchmark is the one that I hope to guide the community to get better and better models, to get to a way major human performance on the benchmark.

1:13:42I think maybe we are the first one to do that for this very segmentation and video, or kind of grounding in the past. It's very difficult to measure human performance on this task. Hopefully, kind of this benchmark unguides the community to achieve human performance for this task and even kind of surpass human performance there. We set out to be one of the best places, if not the best place to build with SAM3 and the SAM family of models. So we're eager to see what people build with SAM and computer vision models to move the whole field forward. We have infrastructure for everything from deploying SAM3 ZeroShot to making your own fine tunes to automating labeling of data with SAM.

1:14:21And we continue to see the impact with each subsequent release, expand the number of use cases and the amount of use and accelerate the time to value. So excited to see what folks can build on RoboFlow with SAM. Thank you all so much. Like this is a really great company. It's a great work. And just obviously like always expands my mind as to what is possible with machine learning. Yeah, I mean, you know, we're not at ASI yet or AGI yet, but every day we're getting closer. Awesome. Thank you so much. Thank you.