BONUS: OpenClaw 2.0 with Chief Architect Vincent Koc

22:12Vincent Koc:How much do you all use computer use in like your day-to-day workflow? I'd say I use it quite a lot. Maybe 10 % of my workload, maybe more. Obviously, I don't see it because it's running inside of the machines, but a lot of pointing and clicking. A lot of infrastructure work still sometimes relies on verification or things like that as well. So Josh is like, oh, hi, Vince. you can see he's typing right now probably getting annoyed um and i think someone on my team might have just tagged me on so there you go so the team have like tagged me on on something here that we can sort of jump into as well and it's kind of work so it's like you say it's like so the agent there's some particular work here it's just like make sure we cc this person or talk to that person so once that work is done it will um okay right josh access denied

23:14yeah i mean this is part of the fun right we're testing stuff hasn't happened yet but um the browser

23:29Vincent Koc:approved companions images for the desktop browser for some reason I can't change the model but that's fine you know it's funny computer use was a thing that that I remember thinking this is cool but I can't imagine I'll use it that much and I've actually found that I lean on it pretty heavy now and especially now that it works in the background a little better and doesn't necessarily have to be the tie up my whole computer while it's doing its work.

24:03Exactly.

24:05Vincent Koc:Somebody from... Like I said, what was the question? Is this directly on OpenClaw that you're doing your work? Yeah, so this is running an OpenClaw service. This is internal, so this is sitting behind a Cloudflare tunnel, so you wouldn't be able to access it. But this is essentially a service that we run and allows us to essentially host this. So this is a huge bare metal server that runs OpenClaw. It's the normal OpenClaw. There is some team specific plugins that we've created. Those are in our OpenClaw code base as well. So everything and how we're working is all public as well. We're not kind of hiding anything generally.

24:47But what you're seeing is running on a live OpenClaw instance in the cloud. But then as I mentioned, we're able to, I'm not touching anything, it's driving another browser window, see what's going on. We're able to kind of drive that and see that working live. The other interesting thing here is that, like I mentioned, the virtual machine, the stuff, this is the bare metal box inside of a VM. But if we were running like a cloud instance, like in, let's just say we did Azure with Windows, you would see a Windows box appear on the right-hand side. We'll be able to steer that and sort of work with that as well.

25:26Vincent Koc:I noticed you could pick between, like, Windows, macOS, Linux pretty easily with one click, it looked like. There you go. So to manage to find the demonstration code that it ran as an example, and it's kind of shown us what that looks like. That's the reason I'm going for everyone. so I can say I'm not going to do that now but we could say let's go play Doom or let's go do this and whatever else we wanted to do right we can kind of take over which is kind of cool but the other interesting thing is that if we're doing something that requires development that's a little bit more complicated we can do some like requires applications or requires a whole desktop environment the team can also see this and take over this remote desktop session as well.

26:22Vincent Koc:So you're able to somewhere see, like, here's every session that's open right now and sort through them as you need, I assume? Yeah, there's so many of these. These are the ones that are running now. We can search through them. The search might not be better, but we're kind of working on these internally. Yeah. Yeah, which is kind of cool. And as I mentioned, with these nodes, as we like to call them, which are these boxes that we have, these are running like a broadcast version of OpenClaw on each one of these. You can also see and remote into these machines and treat them like you would like the cloud machine, but it's your own machines that you control.

27:07So it allows you to really quickly do some really interesting stuff. So for example, for us, we can use our Mac machines that run Xcode to do our app development work. We can use the Windows stuff to test the Windows things. And then the computer usage can take screenshots and test it and drive that and then bring that back. So the underlying technology that I mentioned, one of the things that we're using is this thing called Crowdbox. I don't know what's going on. Which essentially does all the cloud orchestration work. This is also open source. So if you had another chance, OpenClaw, we actually have over a hundred repositories in our organization.

27:49So it's not all just the OpenClaw, OpenClaw, but it's all kinds of different infrastructure. Like when you look at this, it's not the most easiest thing to read and understand what's going on. But this is like a CLI app plus an orchestrator that you can deploy on like a Cloudflare worker, for example, or a Docker image. And that can, your agents can request infrastructure and now spinning up on your cloud infrastructure tools and essentially take over and are you able to put so we've you able to put limits on what it can request on its own and

28:23Vincent Koc:just to make sure you don't wake up to some giant bill or uh that it doesn't go too crazy yeah you can there is some not from a cost perspective but you can set some limits on like who can do what and things like that nice so that's very possible but if cost limit is an issue for people you're welcome to just go ahead and send us some prs as well like we're all for it awesome everything's running a little slow on my machine because i've got all kinds of things running in the background but you can see that this is a machine thing that's like running and requesting different infrastructure. So you can see that's the Linux box that we requested.

29:11It's crashing all the graphs at the moment. But here I can go into VNC. I can open the desktop. I can SSH into the box. So we kind of built all of this for our own testing environment. So then wiring that up into something like OpenClaw was kind of not that hard, like a no-brainer for us.

29:30Vincent Koc:Yeah. Do you have good visibility into that, into all of what's happening as far as like kind of an overhead view of all the orchestration going on within your organization? To an extent, I'd say our internal claws are like very diamond dozen. We only have like a few handful. Instead of creating hundreds of these, we kind of create a few that are like really beefy and have as much infrastructure as they need. and we kind of make that really simple and then as a collective we have specific reporting things that kind of look at all of our internal messages our discord our github and sort of like pull that data in um let me see if i can i'm just going to check something in one of my terminals to see if this works uh because i haven't done this in a while it needs updating it works um but we have these different databases and tools that we've created that kind of gather information for agents and those specific tools and infrastructure essentially feed these agents to give them intelligence they need to understand essentially what is going on and then we can use that we can use that specific stuff uh in the background for like reporting and understanding what's happening in the say our various sort of ecosystem so we've got one we this was a very early project called git crawl because we were getting hammered by uh issues and prs we wanted to kind of like de-dupe everything um one second i'm just going to quickly steer one of my agents to do something for a demo

31:25Vincent Koc:something that i've wondered about around this is when when you talk about all this this infra you guys have built up and expanded scaled up to how many times have you crossed a level you thought oh, this will do us for ages. Yeah. And that not been the case at all. I remember in like February, March, I was like running like 10, maybe 13 critic sessions at once. And now I was like, oh my goodness. But everything was running on my machine. We quickly realized CI was the bottleneck and like running tests and being able to test stuff was just chewing compute because like TypeScript can be quite heavy with a lot of end-to-end tests.

32:08So as soon as we fixed the CI issue with this Crabbox thing and everything was like jumping onto these like cheap VMs that were running in the cloud. That was like the first unlock. And I think it was scaled to like 20 or 30. And then after that, it became pretty apparent to like memory on our machines. So like we kept, we basically hit every bottleneck we can. right now i'm cycling between github rate limit out of memory out of disk space and we've hit the rate limit on the model itself the one i'm battling most with at the moment is disk space ironically not memory or cpu because we solved like that compute issue we have access to models and the disk space is just because of the sheer volume of work trees that we're doing because every agent every pr everything is touching is creating work trees yeah and the way we the way we've kind of solved that is like some file systems in linux allow for like these um virtualized sort of like sim linky snapshot things so like if they're clones they're like they're not real clones on the disk yeah and that's saving us a lot of disk space so we're kind of playing around with like how do we how do we mess around with different um yeah different sort of infrastructure and stuff oh man it's kind of interesting yeah and storage storage isn't cheap these days uh not storage is uh skyrocketing in cost unfortunately it is it is uh it's been crazy

33:38Vincent Koc:to watch you know to watch ram go crazy and then to see like that an rtx 6000 is like 18 grand online now and uh and and and you've got solid state storage doing the same thing and who knows what's going to follow it probably cpus at some point once we once we realize we're all running a gajillion things on our computers and care about them again yeah so that's been an interesting sort of bottleneck and now for me uh on the peak i think one day i was like running 60 to 70 agents between my infrastructure i've got like six or seven machines plus my laptop yeah this is just my own personal stuff like my own personal open core work not any of the team members then i realized actually it became like unmanageable not from a size or anything like that but just the fact that um just kind of like centralizing everything was needed because i running out of disk and memory i just kind of needed like a central place so i think moving that all recently to openclaw and building it on openclaw and scaling that and having that sort of manage the infrastructure and move the work around has sort of been the next unlock uh for me which means that actually because we're building it at this scale and we're using it to scale means that the the team and anyone else who uses openclaw is going to get that experience now Right now, like a lot of the cloud stuff is very buggy.

35:14It's not like 10 out of 10 as we want it to be, but we're kind of forcing ourselves to use it and making that part of our development ecosystem and lifecycle so we can get it to a point where other people can really use that. Yeah, nothing will make it better than just using it

35:30Vincent Koc:and finding problems yourself, you know. Exactly. There's an awful lot of bugs you'll fix on your own as you go. Just going to jump back into the team. The interesting thing to me when I look at where OpenClaw was back in January, back in February, you almost needed to be a developer. You could do it, but your life was going to be a lot less painful if you were. Like the ability to kind of better understand things like gateways and credentialing and working between your CLI as well as your web app. And it's shifted away from that a lot. Like when you look at the modern interface for OpenClaw, it's pretty simple now.

36:20Vincent Koc:It mostly makes a lot of sense. There's definitely stuff in there that a normie, for lack of a better phrase, might not need or want. but the setup's pretty straightforward in my opinion yeah we're kind of moving to this thing where like what for example we're just launching on the windows store now you install it it sets up open call sets up the gateway installs the windows app so everything's kind of controlled through the app not through some web ui thing you get the same experience you have a little taskbar thing so it starts to get a lot more normal we started doing things like if your device supports it like if you've got one of these new um laptops that are coming out with like insane gpus on it we detect that we set up a local model for you you know like so we're getting to a point where it's like out of the box oh you've already got codex or claude installed oh why don't we use your your data and then start from there so we'll like get your account information we'll move your memories your sessions and suddenly there you open up open call like oh your existing sessions are there and you can kind of continue on from the work there so i think that experience is going to to get better and better.

37:28But one thing we're not doing right now is like hosting it and giving like, there's one click, like pay us a dollar and we'll do it for you. I think right now the focus has been like, we want the product to fill 10 out of 10. And if you can, if you want to go ahead and like deploy that somewhere or someone wants to deploy that for you, by all means go for it. And we're having some organizations that are definitely building on top of OpenClaw and building managed service sort of versions of Claws or deploying that within their infrastructure. So like more recently, we've got something like Microsoft Scout that was announced.

37:59It's kind of running open core as like an engine within itself. And that's like an opinionated, more Microsoft flavor of this sort of work. But essentially same things kind of working on the other as well, which is cool.

38:13Vincent Koc:How crazy was it watching as this was all happening? I know it was insane for Pete. I mean, even for you though, to see like this little thing, you were just working on that was like you're throwing PRs at this little project that you don't necessarily think is ever going to be anything. And then in a couple of months, you're meeting with Jensen Wong, you're meeting with Sam Altman, and suddenly Pete's deciding between Zuckerberg and Altman and all of these things that went down in that period. It just seems like so much to have processed while also trying to build up this infrastructure trying to build up this product and make it good i think it just shows us that like there's definitely a huge gap in the ai world like there was this world where agents were this promise we had some automation we had some cool stuff but actually just giving the agents what it wanted and letting it be able to steer its own tools and build its own tools and just kind of giving it the sandbox hadn't really been done well and as soon as that started to work and people could see the value and it kind of crossed this chasm where you've got rap artists on x talking about how to do things with the open claw they really became like part of the zeitgeist and that was kind of the fascinating point and and i think showed people that wow ai is this thing that can like really bring about a lot of change and in my mind i think a lot of positive change as well we've heard some amazing stories of people like disabilities and people who couldn't run their businesses and interesting like stories coming in every day of like open core users of like what they've been able to do now thanks to this but the beauty as well is like not just open claw itself but like this new generation of agentic tools and ecosystem that we were able to sort of usher and sort of this cambrian explosion of like token usage that's occurred and like this new agentic era that i like to call like the post claw era um that suddenly came about that that's fascinating like we've been able to sort of like bring this uh ecosystem into like a new sort of uh age of like our agents doing things for for us for people for for your own personal use like It's not just about another email automation or another work georiffing or something like that.

40:40I think that's cool. And that's one of the key missions of OpenClaw as a foundation now. It's like, how do we make agentic AI good and useful for everyone? And seeing that sort of playing out and coming to life and it not just being our OpenClaw, but other products in the industry and the whole ecosystem growing as a whole is definitely a positive experience for us.

41:02Vincent Koc:you know it even changed the like the economics of tokens like before before open claw i was dealing in thousands of tokens and then like when i first set everything up and went the most wild i went and was just blasting out automations and all of these different things that are triggered and uh and went full crazy i just i remember the first time i looked and was like oh 100 million tokens i i'd never thought in a number like that and it just absolutely blew my mind and uh now that's a slow day for me you know depending on depending on what i'm working on and uh that's that's a change in nine months you don't want to know my token i would love to know your token count so i've averaged between five and ten billion a day i think one day i don't know what happened but I hit 54 billion tokens in a day.

42:02I think that's equivalent to an enterprise company. Oh, yeah. That's just my own personal use. Yeah.

42:08Vincent Koc:But you can imagine how that can happen where we... I work with 200 people and we're not near that as a company. But you can imagine where it's like 100 % of all my development work is through an agent, right? Like intentionally or unintentionally. So we're kind of forcing ourselves to behave in this way. Yes, it's not token efficient, but also at the same time it's forcing us to learn and develop tools and develop infrastructure and just things that just don't really exist. So we've kind of had to be on the bleeding edge a little bit by sort of, you know, going through a lot of pain by using tokens for everything.

42:43But yeah, it's quite heavy for sure.

42:45Vincent Koc:It is. Dare I ask how one even pays for 54 billion tokens a day? Let's just say we have a lot of donors and friends that really kind of support us with this mission as well as labs like OpenAI that also subsidize and give access to tokens to some extent as well. We're very grateful for that. So I think there's people in the ecosystem that believe in our mission and support us in different ways and we're very grateful for that. So you can check out the openclaw.org website and there's like a list of some of the donors some are obviously large organizations but we also have like individuals that have been donating through github um since the start of this project and yeah i think without the community this wouldn't be where it is now and i think we're very grateful for that you had a great it really quickly had a great community around it and i think some of that was was how like for me i could you couldn't not root for pete like like this is this is a guy who did a thing and had this rock star moment out of nowhere and and he's having fun and you can see it's all so overwhelming but there's there was something about watching it that was just like you know hell yeah i like this guy this is awesome you know uh and uh watching his journey along the way uh has has been wild too because i know for him it was a lot and he talked about that rather openly on on x a few times like hey this is this is this is a lot but uh it is a great yeah the ability to pull in a good team around this project though is what's taken it from kind of a surprise little project that that shot up out of nowhere into something that feels like a product with a goal with with a roadmap with all of the things you would expect a product to have and working on new platforms i think that's uh that's a real testament to what you guys have have reigned this into over the last a little over half a year as crazy as that says yeah i still feel like there's a huge way to go like to make this experience like magical like it's still magical when you go through and have a we call it the getting claw peeled right where the agent starts doing stuff and he wouldn't stop talking to him for days yeah but i feel like i've got that magic again i'm feeling that magic again in my development environment when like i'm just inside of a browser i'm not using a terminal anymore and soon i want the same experience to work natively on mobile devices where i can just be on my phone and drive that and some voice models for example are so good that like you cannot talk to it for half an hour and it will just be in the background and then you whisper something and it kind of wakes up i'm getting to a point where i'm like testing those kind of voice models to like orchestrate all of my sessions as like an orchestrator agent and now i have like a team that i just communicate with or join meetings with and they can kind of keep an eye on all the stuff that's going in and we can just through conversation we can start to steer i do that sometimes it's been like codex where i'll use voice mode and be like hey what's happening with this project how's this one going what's needed for me here and and but like we've reached this point where voice can go check those things out but the beauty is like imagine this running on your infrastructure with your models yeah you know with on any compute fully open source you know swap out the voice model swap out whatever you want um swap out the machines you want or like you've got like this to say you've got a two dgx's and you've got a mini hook them up as nodes and then this is like your mini thing that you have but you can control all of that from like one single pane of glass from any one of the devices, I think that's pretty magical.

46:50And then you control the data, right? You control what happens to it, where it goes and which models you want to use. And I think that's where a lot of people, they get it, get it. And I'm not preaching for like, is it local AI, is it cloud? I love the frontier models in the cloud. I also like local models. I use both. But I think the beautiful part about here is giving you choice. Giving people choice, I think, is going to be very important. And we want to give people that choice. And I'd say the hardest thing for us as a project has been being able to maintain that level of extendability and choice with so many providers, with so many messaging platforms and having that all work as much as it could be.

47:27Again, nothing's ever perfect. Has been probably the hardest thing. You'll see like a lot of products come out to market. It's just like, oh, we only support iMessage or we only support Slack because it's hard to maintain that many channels.

47:40Vincent Koc:Yeah. To give you some sense, we don't have a device farm, but it's almost getting to a point where we have to run a device farm with agents, with phone numbers, to set up all these different accounts with all these different places to have them communicating with each other to test that these things are real because we have to run into and test. These messaging channels don't really want you automating tests on them because it looks like bots that are trying to spam people with messages. Or you're blocked all over the place. yeah so we're continually having to like work around that which is kind of a pain but we are running like some pretty interesting infrastructure to test all these different messaging channels and validate them so like for example if you do a pr with telegram on open claw you'll see a video screenshot attached by a bot that verifies the change and like validates it on a live telegram account for example so we have things like that that we're definitely cooking up and that gives us confidence and then other agents can verify the changes visually using like visual lms and we can get to a point where like a lot of smaller bug fixes and changes can be much quite quickly with limited interaction from maintainers obviously maintainers are still making the choices of to like what to let in or not to let in and how that works but we're going to get to a point where like the factory as we like to call it um is going to become more sophisticated because we can test and validate a lot of these things as well cool we got a couple of questions in here looks like three i'd like to bounce off you real quick first one is uh what voice models are you referring to that you see giving you the best results anything in particular i think this is called i think this one was called gpt real time or gpt live i can't remember the the original wording for it was bd or like bi-directional yeah i still call it bidi, but bi-directional voice model.

49:29What this means is that as I'm talking to it, it can talk back to me, but also whilst talking to me, it can execute code in the background. So it kind of has like a human brain sort of concept of like, I can receive information and give information out at the same time. So as it's talking to me, as I'm talking to it, it's carrying out changes in the background. And something about that experience, I don't know, feels really, really interesting so if you if you have the codex pro subscription i think you can just throw up the codex app and give it a go i'm pretty sure we we do support it now on openclaw yeah but if you just want a quick test you can test it inside of um chadgpt and then you can just like not talk to it for 10 minutes and it will just sit down and you can say something and boom it's it's kind of back i keep it on most of the day on their i think on their hundred dollar

50:18Vincent Koc:plan you get like a rolling 24 hour limit where you can use 15 hours and i was like oh so i can just have this on while i work and just spout a question when i want to and uh you're right the whole bi-directional thing is a trip uh especially when you're just getting used to it like it multi-tasks better than i do yeah so i have like this um what do we call it i have this uh like chief of engine head of engineering skill or something that i kind of created and as i mentioned to you i have you know let's just throw it up just for just for giggles just so people can see like this insane setup that i'm cooking for whatever reason um so this was like my older version of my development environment just just so people can kind of see oh mind you the other thing i was telling you about this was the github database thing that that i was telling you about oh yeah this was this was back in like february march um there's a ross no go based application that agents can use and it has as you can see the data is a bit outdated three to four months ago it has every issue and pr like clustered and because of that we can close things out so here it's seen there's like another slack change that's probably related to some of these other things so agents can use this it's like a it's like a permanent cluster of the information but all the github data gets analyzed by llms and that data gets kind of captured like what was the person trying to change and things like this and it's all vectorized and kept in this tiny little database so then we store this in 20 and share this with the with the team why is that not you see my one second i think it was frozen on on the previous screen there a little bit there we go now we got it this is what it looks like oh yeah so yeah this is like the the thing so this is like what we call the crawl apps we have this for discord we have this for different data sources and agents can use a terminal app that we have called git crawl but we run our own server essentially uh that's remote um on cloudflare but you can use git storage as well but this was our way of kind of solving the rate limit issues by giving shared intelligence to all of our agents collectively around the state of all of our prs and issues but also it helped us de-duplicate so there was this one moment where we had like 25 000 prs and issues like 70 of them were duplicated already solved um once i finished this i think i closed like something close to like 8 000 prs in like one afternoon and it was like wild how many has it had so far do you know a running total of open call PRs?

52:56We're past 150 ,000 total PRs and issues combined because the issue and PR count is the same. So if you look at the count, we're above 150 ,000.

53:05Vincent Koc:Wow. That's so many. For some reason, Ghosty has a bug. Ghosty has a bug? Go. Yep. Yeah. There we go. So this is my T-Muxi hell that I put myself through. Oh my god. Boy, it is hell. This is my version of Herder. Yeah, yeah, yeah. But I kind of go here, new after.

53:36Let's go. I'm going to do something a little different. Does that work? No. I've got to change every time I open a new window. That's crazy.

53:47So I can jump on any one of these boxes. So for example, let's jump on this Mac Studio. This is another Mac Studio. So I have this key-based thing the agents can jump around, and it's called MyFleet. And I have a private GitHub repo, which explains my fleet configuration tied to a skill. And essentially, all of my agents are aware of all the other agents, they're aware of all the other codec sessions, OpenClaw agents, and they can just move, work around using TMUX. Obviously, I'm moving a lot of that workload now to OpenClaw. And we have some similar primitives in OpenClaw. There's this thing called Beam.

54:20so I can inside of my codec session or another open core session, I can say, can you beam this to over here? And it will sanitize the transcript and session, and it will just find a way and just shove it into the other machine.

54:32Vincent Koc:Like clean it up into something that's easily digestible kind of? Yeah, so yeah, you can see here, I can jump around. These are many machines running many codec sessions at the moment or codec sessions and things like that. But the idea is essentially moving off this sort of environment to a web-based, because this is hard. i can't control this yeah but what i was doing in the past was getting because it's all team marks getting uh you know my agents to be aware of all these agents through skills and other dot files and then getting to a point where my github uh chat gpt session for example with voice could understand that and control them and if you remember i told you about this punch card thing the good thing about this punch card thing actually if i quickly share my screen you'll see something very simple but you'll see at the top there's a there's a there's a title attached yeah so i have the agent is running i'm asking it to update its like primary purpose through tmux like through a cli command um and this punch card thing that i told you about it's also doing something similar where it's essentially going you know what is this piece of work that you're working on and this was like another experimental project like i mentioned you and we're punching in and out the work for agents.

55:48And how would you explain this work? And the beauty is that that punch card thing also becomes an index of all the running agents and what they're working on. So that can be quite small. And then your agents, if you have multiple machines running across many things, can suddenly go, okay, where was that person that's running this X thing? So I'll show you what pun... This code is public, by the way. It's probably good for you to go back too

56:13Vincent Koc:and be able to see like... yeah so this is where the problem this was you know this checked in 11 hours ago last so and this was doing something to do with vcpu scaling uh total of 32 prs were merged for this piece of work this is the whole thing that can say I can like remote status thing these were like early experiments like I was saying around that how do we get the multiplayer working but like these as you can see the intent is this is the work that it's trying to do so I got to a point where I can get the agent to explain to me like what is the thing that we're trying to achieve here and the intent was based on my messages and things like that and that could be captured into this like very small discrete thing and this could be indexed and all my agents could have access to this like index of what's running across all these other agents and machines so it becomes really easy for like a chief of staff or head of engineering codec skill or whatever however you want to set it up to know what is happening out there really quickly to jump into a box and to like start steering that session or if it breaks another thing like my machines were crashing all the time because of the amount of memory outages and limits i'm hitting so i could easily like once one shot like restore a machine i can be like restore x machine and recover and it would just bring it back with all the t-max pains with all the notes and state and everything yeah because i was like having these outages to me everything's dead i have to come back and uh you know on on my puny one guy managing probably too many little projects uh this is great that that you can come this is all this is like early experiments so this is like no these are these are the codec sessions right like you get the number you get the you get the work tree like it's it's self-aware of like what's running where on what and yeah this kid gives you an idea of the madness this is my version of herda before i heard it was a thing by the way this is all public so if you go to github.com forward slash vincentkotch forward slash dot files you'll find it there there is some other skills and stuff i haven't made public yet but i'll i'll look at that like my chief of my head of engineering skill that's talking about and things like that um the idea is that like i throw a link to that in the chat too yeah but the thing is i don't i shouldn't need this this should all just like work i shouldn't have to touch the terminal it's a janky experience because then what i'm doing is i'm rushing to my phone trying to use like a terminal through the phone i'm trying to remote into a machine which has a terminal then i'm like it should just be like in a native app experience i can just talk to it i can go through a messaging channel and that's kind of where i'm heading to with most of my workloads now like shifting away from this terminal based experience i still need it for some stuff like it's not to say that i don't but ideally getting to a point where majority of my workloads are going to be essentially running through this sort of this this shared claw experience because it also means that you can take advantage of things like memory and things like that so you don't have to yeah move things around like i had to have this like complicated process of like moving my memory artifacts that codex and other agents create between my quote unquote fleet of machines uh which is like not not really scalable i can't just throw like three more aws machines into the mix i've got to sit there on a scale to like set these machines up whereas Whereas if it's running in this central setup, and then you can easily add compute to that and just run your workloads, I think that becomes a lot more manageable.

59:34Sorry, I know we dragged on with that question a little bit.

59:37Vincent Koc:That's okay. It's okay. Let's see who we missed here.

59:45Vincent Koc:Somebody asked about if OpenAI absorbed open clause capabilities as a set of roadmap items, or if we still hope to see, and I realize this probably isn't a thing that you're able to answer. But I think a lot of people were also confused around like acquiring Pete versus acquiring OpenClaw, I think. Yeah, so Pete essentially took a day job at OpenAI as the most simplest answer to that question. The project remains and will always remain 100 % independent, and that's kind of the purpose of a foundation. and for everyone's benefit like if you go and look at like the companies that we're working with you know we're not just working with the u.s companies we're also working with chinese european everyone right like we're here for for the greater good of everyone so we don't have like an allegiance to one company another but we're super grateful for the support that open ai gives us as an organization and as a team as well and yeah but i can't answer any questions even if i was aware of any sort of roadmap items like that.

1:00:53Keep an eye out for Dev Day and what OpenAI might be announcing around then, which is about a week or so away.

1:01:01Vincent Koc:Yeah, a lot of really interesting rumors out there. Let's keep the rumor bill rumoring. Let's keep it interesting. Let it rumor. Here's another one. Do you use local models for review agents or still only cloud? Any specialized models to minify cost? Minimize. Yeah, so for us, I think frontier intelligence has always been the case. So we do, we use, and I continue to use, cloud model frontier intelligence for review processes. There is some fallback mechanisms that we have in case we have outages and stuff and we have to continue working. So then we do make a carefully crafted decision on that, and it's pretty baked into some of our rules.

1:01:49But we could look at specific scenarios and things like that. Now, a new one that has come out, we do support this on our code base and it's pending releases like JEV and what we're calling decision models.

1:02:01Vincent Koc:Yeah. And this is where we're starting to, as OpenClaw, starting to separate out normal model inference from what would be a decision that needs to be made at any point in the machine. and that decision process could be offloaded to a more specific decision model. Fun fact, I was actually working on an open source version similar to JEV prior to JEV's release because I saw some other models that were coming out in the industry where you bolt on and classified to try and solve this problem. So I'm going to speed up the development work on mine in the background because I didn't care so much, but this is now out.

1:02:37I want to throw my hat in the room with everyone else that's coming up with an open version of JEV. but I would say the biggest thing would be like if you can make decision on like which model to use or what what needs to happen like that's like an interesting one so we're looking at like for example could we use a decision model to understand based on the need which level of reasoning you need or yeah things like that so there's ways or what's the right model for this task even yeah potentially yeah so that's where we're internally like experimenting with some of this stuff that I think it's going to be quite interesting.

1:03:08I know NVIDIA came out with this thing called Switchyard, which would use local first and then figure out if there's a percentage of that local use is sufficient enough or accurate enough for that workload. Otherwise, it will continue. Otherwise, it will then switch over.

1:03:28Vincent Koc:Yeah. You know, what I was using most of the year with my OpenClaw as just my main chat model, mainly because like 75 % of what I use open claw for is like, because we have the giant newsletter and this and all this other stuff was keeping up with what's happening in the AI space and sorting through news. I used Mercury too, because I loved the diffusion speed. And, and it was plenty model for most of what I did. And it made my, made my claw run just so cool. it's a thousand tokens per second for like you know a buck per million tokens it was really cheap actually you've given you've given me a great idea maybe i should rebuild the uh the open jev thing that i'm working on to use a diffusion based approach and make it insanely fast there you go it feels like diffusion watching jev work even like there is that but it's just a classifier it's essentially a bird style classifier using like classical uh language model methodology to just essentially bake that on top of a model.

1:04:36It seems like a no-brainer when you actually see it, when you actually understand how the architecture works. You're like, why hasn't anyone done this before?

1:04:41Vincent Koc:You're like, this feels like something we should have already stumbled on. Yeah, that's true. Oh, my gosh. I got a question from Grant here I want to ask, too, while we've still got you. What are, like, the top use cases you see for OpenClaw? What are people using it for maybe in a business? I get this all the time. You can practically use it for so many things. I think one of the things that my team are now doing is we've started collecting this thing called OpenClaw stories, and we're going to start sharing them more broadly because there's some crazy stories out there of what people have built.

1:05:17Like I said, someone who's got it running on a small mini PC on their wheelchair and all kinds of amazing stories, right? One thing I'm going to say, though, it's really hard for me to pick one of these stories out off the top of my head because there's so many, and like I said, we will share them. but I think thinking about use case is probably the wrong way because if you think about use case you're thinking about give me one thing that this thing does really well but it's actually more a case of like this is just like any other employee or person that you can train up given it has the right tools and access and to be able to do it like this is just like an intern and maybe that intern's got really cool tricks because it's got this really good model that it has access to and it's got really good data that has access to so it really depends on like what you have the time and the patient's for.

1:06:00If it's like, hey, use computer use and like summarize my emails well, the model can do that really well. But if you're trying to give it something more complicated, maybe the tools or the model is not able to understand. So it's not about can it do this task really well? Is that can it do this task 70 % of the way, 60 % of the way, 50 % of the way? And I think that's going to improve. So it's less about which use case because I think something like OpenClaw can do any use case, but it's more a case of what percentage of it can it do. for example yes it can use computer use but if it can't fill out the capture then maybe that gets me 80 percent of the work there and then it just requires humor for the last 20 for example so i think that's kind of the mindset that i'm trying to yeah trying to trying to go through

1:06:45Vincent Koc:and the thing is how you use it for one thing but about how you implement it into everything and it's hard to share because some of this stuff like it's not public but there's like enterprise companies that have deployed this in every single person in the company, every team, everyone has it. I'm talking like thousands or hundreds of claws running inside of a company, for example. These definitely do exist, but these aren't really things that are out there publicly per se. But we're trying to help get that information out there. We're trying to get those stories out there. So that's definitely something we're working on.

1:07:19Vincent Koc:what's the craziest way you've heard of someone using it running it uh the craziest open claw story you've heard oh i'm just trying to think off the top of my head i feel like there have to be some i think there was one in shanghai uh where there was a whole entire dumpling store that was run on this open call apparently like a big thing in china this this was like one of the stories that the media was like kind of jumping into there's everything from ordering to food like like literally the human was like doing the last mile cooking of this thing but everything else was just run by a claw the marketing like i'm talking 100 % of everything wow so they had like an entire dumpling shop running like fully like end to end on a claw like and a human serving as its robot in the back of the building to do the basically essentially yeah yeah yeah i think i can't remember where it was i haven't checked it out but yeah i think this is this is definitely a thing but yeah there's all these kind of stories everywhere but it's not just open cloud i think it's just agents in general now can do a lot of these things like cory like we were saying we can kind of show people the way and now it's like 100 minutes yeah the ideas wow yeah i have a website that that it runs that you know one model runs out in the morning and scours these various sites for updates the next one comes through and classifies it actually have it hooked to jev now funny enough uh going through and giving me yeses nos and letting me know what needs reviews and then it goes through uh asterisk six and and that makes some edge case calls and comes to me for a final go for it you know it i also shared a video this is an early this is an early video from a a researcher at MIT labs called Sirius and just go watch the video where he gives AI a body he basically teaches open claw to express emotions it's the most fascinating thing you just have to go watch it it gave me a goosebumps when I saw it but in your own spare time go watch this video it's it's it's so interesting basically it's like these little wooden blocks that move up and down on a box and then teach the model to be able to move these and there's a point where it's like says something and it learns to sigh because it moves up and then like like that like it's breathing and it like tries to express emotions and this is part of a research project that MIT labs was sort of working on like around agents and human interaction that I think was absolutely fascinating this is one I like to show like something that's a bit more out there but different about like what can be possible when we kind of interface agents with memory into the real world doing something that's a little bit more different and left field that's so cool yeah here you go make sure open claw doesn't get upset so you can receive your next day digest yeah it's uh i'll show you what our digest looks like as well whilst you're going through i'm going to show you we did this fun fun little project i want to make this public but the team we haven't i'm going to show you guys anyway

1:10:35So one of the things we can do now, and this is a cool demo actually, is you can create these dashboards. These are dynamic dashboards. You can tell your agent to build them. And you can actually do widgets, all kinds of stuff. You can build these with plugins now. So your plugins can now take over the interface. This is what we call the daily claw. This is our own inter. This is just for fun. and it just like scours through like the commit history and builds like a newspaper for us every

1:11:04Vincent Koc:day oh my god that's amazing i'm an old newspaper i started in newspapers so i this is this is there was a recent one i think it was kind of going wild on x and instagram and tiktok and stuff but this lady who did this like daily newspaper printout from an old laser jet printer every morning she gets like a little printout with like mono face print and like shows her calendar and it's like a cool thing just did something similar to this but we did this internally for our project that's what it's cool yeah i don't know if we should ship this but it'd be kind of cool to make it public but these are like little plugins that you can create that live inside the dashboard so you can create these dynamic ones um as well and these work on mobile and other things like that as well which is kind of cool mobile web and things like that as well yeah just so i'd show like a cool that's super cool that is super good i'd love to uh get mine in the morning and like an interactive newspaper where i could touch a story and it reads it to me and highlights the words as it goes or something i i'm a big fan of uh of hitting the read aloud button i've really gotten into the like okay i'm gonna make my espresso and i want to read the news so i'll i've got a whole bunch of like i've got a whole bunch of ink displays so i'm like toying with a lot of these kind of things to like get open call to like punch stuff out this one's got like a little magnet on the back so you can put this on the back of a mobile phone so i'm trying to see if i can get like the mobile app to like work with these sort of displays and sort of like show messages and um it's a bit more like interactive i don't know if you guys seen the nothing phone that came out for a while had like this if you watch the founder and he explains like his product development like how phones can be a bit more like when you flip the phone over you don't want to see the screen but there's things that you can show on the other side and he has like these little lights and he can gesture you in different ways and i kind of was toying with that idea like you've had an ink display that can show different things or that look like i know someone on the team did this thing where we added this dreaming feature where like memories get summarized each night and he did something similar where it would like take a summary of the day and turn that into art and then he would kind of link that through an apple shortcut so that his background would change and so his background his phone changes based on his claw and like how his day is going for example like as an artistic rendition of like his life for that day or something along those lines that's really cool fascinating it is it's a neat time to to be alive yeah i think they call this kind of tech ambient technology so it's like you still have like physical things around you but they're not like i don't have to interact with them so hence the word sort of ambient and um yeah so matt the url for the daily claw that's part of our internal server that's not public so i'm just showing a demo of like how we can prompt our agents to like build stuff inside of openclaw as plugins they can take over the interface so we can build all kinds of like reporting interfaces and things like that so it means the team has everything they need in the in this one place but yeah we haven't published that yet Vincent what uh what what capability do you think is still missing from like modern ai models that would it would unlock some cool stuff for for openclaw this is one challenge right because i I think a lot of it is the levels of intelligence.

1:14:37But I think majority of what we need is already here. I think it's more capability of like building the application on there. Yes, they can get smarter, cheaper, faster. They may not be able to solve every problem. So I think it depends on the use case. Like if you're trying to solve some complicated biomedical thing, maybe they can't do that yet, right?

1:14:55Vincent Koc:Yeah. But most of what you want to do, it can already do. I think it's there and if it's missing you now have the capability to be able to teach it to do that thing or to be able to train a smaller model to do that thing and give it access so I think from a capability wise it's there, it's more of a time effort energy to kind of glue that together to build that experience or that thing that you're looking to do very cool well man Vincent, thank you so much man this has been a ton of fun got anything else you want to show us before we go? Or are we? I think that's a good start. I think that's a good place that we're at.

1:15:37Vincent Koc:We want to show people what we've been building. We want to scream into the world. I think we've been having a lot of fun internally. I think there's some cool stuff growing. But yeah, this experience is going to get better. And I'm hoping a lot of organizations start to embrace this multiplayer nature. I would preface that how we manage teams is going to change. How we manage work is going to change. because now it's like not my work or your work it's our work and we have to get comfortable with like this transparency and sharing and like collective work and it really it's gonna i think it's gonna shake a couple of dynamics in how we do stuff we're obviously on the forefront of this we're kind of building the making up the rules as we go but it's going to be really interesting to see how this transpires into more sort of like traditional workplaces like especially engineering with cultures and things like that so it'll be really cool to see how that kind of comes about when you think of how far we've come since let's say let's say thanksgiving since it's last november to today it's almost even hard to picture that amount of time from today like i think it's wild the transformation is going to be huge in short windows like i i always joke that the The stuff the nerds are doing today, the normies will be doing in six months to 12 months.

1:16:59Vincent Koc:And it's really neat because I know a lot of people on both, and I fall somewhere in the middle. And there's so much stuff that's just continuously getting easier, man. It's easier. It's more accessible. I always joke. I have a philosophy degree and studied journalism. and like i and it's it's it's just a bizarre transition and i think we'll see so many people doing that the joke we have in i say to people is like we run open claw on dog years it's like human year versus us is like basically a human year for us like a month or you know something like that it's like some we move so much quicker um so i think that that that's kind of you know it's hard for people to keep up with as well and that's why it's good to have like these sessions where i get to get it get in front of people and demo and get feedback and kind of show people what we what we're building because like you say everyone is building so much all at the same time the model companies the labs the technology industry is just shifting at such rapid pace it's it's it can be quite hard to keep up so it is it is you know and and and grant and i live in this news from eyes open to eyes closed every day and and we feel behind you know i mean god knows how many articles we open in a day between the two of us and and even we we still feel that way with like what's happening and what the new tool is we hear about things we didn't know happened three weeks ago and it's like how does that even happen uh but it does and it's it doesn't just feel fast It is fast.

1:18:41Vincent Koc:But it's also just a lot of cool stuff. And I think there's some really amazing things around the horizon. We're watching so much happen in mathematics and science and drug discovery and all of these neat fields that kind of bring forth the promise of AI, as I like to call it. And I think people like what you're doing are a big part of all of that. and really wish you and your team, man, all the best. It's been fun. Thank you so much. To watch Open Claw become what it is. And can't wait to see where it's at another year from now and hope to chat with you in between. Yeah, let's definitely check in.

1:19:23Let's check in another 12 months to see what's become. So, yeah.

1:19:26Vincent Koc:Absolutely. Yeah, let's go from there. Thanks again, Corey. Thanks again to the team. Yeah, appreciate it. If you're watching and enjoy the chat, please take just a moment to like it, subscribe, support the channel so we can keep bringing you great people just like Vincent here. Go check out Open Claw. If you've never played with it, just dive in. Give it a shot. And on that note, we'll see you next time. Farewell.