Jev: The Model That Can't Talk, Amazon Blocks Meta's Muse & a Robot Teaches Itself Soccer | Lightwork

0:00Josh Machiz: I mean, you and I are always shopping. We're always shopping. This is not the first time Amazon has been territorial over the idea that humans now have agents shopping on their website. They run a$68 billion ads business. For your average person, do they trust like an up-and-coming startup? You don't want Muse calling your doctor's office and being like, hello, I would like to make an appointment for Claire. There are some really crazy technical advancements here. Feels like we're moving towards Rant a Human.

0:35Josh Machiz: We're back with another big week. A startup called TypeSafe released Jev, which is a model that never actually writes words. It just answers, picks, and scores in under a second. Amazon blocked Meta's new Mies agent from shopping on Amazon at the same time that Shopify decided to open every one of its stores to it. Skilled AI taught a humanoid robot to play soccer by playing against itself in a simulation. Plus, I have some behind the scenes on the ground footage from MetaConnect. Let's get into it. So Claire, the most interesting launch of this week is a model that can't talk. Talk to us about Jev.

1:14What is it and why is it so interesting?

1:17Josh Machiz: Yeah, this has been really popular. immediately after its launch, Vercel said it was the fastest adopted model in its AI gateway history. And I'm sure you saw a lot of developers playing around with it. There was a lot of excitement over Jeff. For starters, I was like, what is this model? You know, in my head, I just kept like sharing my name, Jeff. Yeah, and if it can't talk, you're like, wait, what was the use? And how is it so different from an LLM? I do think in the launch, they did some really great marketing, like creating a category by calling it a system one model. Yes. And I liked their headline of the build prod, not God.

1:58Josh Machiz: Yeah. So some great marketing, but like, what does it actually do? And what makes it so different from an LLM? Yeah, it came out of a lab called TypeSafe AI. And interestingly, the team consists of someone called Diogo Almeida, who is one of the co-creators of ChatGPT, if you will. There's a lot, you know, can go endlessly on his resume and some of his technical accomplishments. But he really was part of a really important paper in 2022 called Instruct GPT that was pretty consequential to our jump from GPT 3.5 to Chachabut. And so he's been pretty important to the trajectory of AI. But I think one of his frustrations, as you've seen with their manifesto around build, prod, not God, is that we don't necessarily need super intelligence for a lot of the decision making and software processes that fill our day.

2:50Josh Machiz: And if you think about the ways that agents run, there's tons of tiny little decision making moments in every part of the workflow. And yet we run them through these very compute and token intensive LLMs. And so I think the example I bring up often is if you're a company and you're just trying to understand if your client customer email is angry or not. Today, if you feed it through an LLM, it typically feeds you a paragraph. Sure. It's having a full conversation with you. Exactly. When you didn't necessarily need that. Yeah, yeah. And you have it. Oh, this is my sense of this sentiment. Would you like me to draft a reply?

3:27Josh Machiz: Which is great. And there are tons of use cases around that. Sure. But you burned a lot of tokens and it took a little bit. Exactly. Now with their system one model, and we'll get into why it's called a system one model. It's more of a little bit like a traditional classifier where it is built entirely to focus on decision-making. And so you take text in and then it actually has numbers out. So it doesn't produce reasoning. It doesn't produce any text. It basically gives you a choice from a fixed list and scores it on a scale and a yes, no probability. So there's no explanation or chat window.

4:03Josh Machiz: What's great for that is, again, if we go back to that question of, is my customer angry? It can basically just infer the probability of that anger. And then, for example, if it's above 90%, then you escalate to a manager. And I've gotten a couple of questions from folks. How is this different from BERT or some of the classifiers that we've had for many, many years now? And a big difference is it's able to have some of that like generalizability that LLMs have, where if I'm a customer and I don't say I am angry, it's still able to decipher that. So you can say, are you kidding me? How is this possible?

4:35Josh Machiz: And it'll still be able to decipher anger out of that. So it sort of has like classifier behaviors, but with some of the LLM brain and without the extra training that you've historically had to do for classifiers, where it's like you train a specific classifier for a specific action or workflow, and you have to give it very specific training data sets, very specific instructions. This has that same decision-making ability, very fast end-to-end latency, very cheap. I think actually the numbers are$42 per billion tokens. So much cheaper. Yeah. If you think about LLM today, it's counted at a million tokens.

5:11Josh Machiz: And this thing is really fast. So it comes out at 70 to 500 milliseconds, whereas most models today or anywhere from one to 300 seconds, even especially if you're doing anything with long reasoning. Yeah. I also love that they published their limitations page and the things that it's not good at, like math counting dates, multi-step reasoning. So if you're using this, like which of the background decisions that are, you know, cheap enough to get wrong that like you would use this for versus an LLM? Yeah, I think what we're seeing is this sort of new paradigm where a lot of engineers now have optionality, right?

5:47Josh Machiz: And you don't have to run a really big reasoning model through every single task. And you can route things to different places. Yeah, exactly. And I wouldn't say that this is a situation where, you know, we're like, Jeff is replacing LLMs. Like, certainly you're going to use some combination. And so I think what developers are sort of grasping with is like, you have the ability to have both system one and system two reasoning. So that I think was inspired by Daniel Kahneman's book, Thinking Fast and Slow. And the idea that you have a system two reasoning, which is sort of stuff where we sit down, we think deeply.

6:23Josh Machiz: That's the stuff that all alums are very good at. System one is just like the rapid fire instinct, you know, very quickly, yes or no, give me an answer. And so I think that's what we, ideally you should route those sort of decisions to. I think people have played around with a lot of different demos. I don't know if you saw some of my favorite ones, like for example, was a virtual try on one. I don't know if you were able to see it. No, I didn't see that. What happened in the virtual triad? What made it different? You basically get to talk while you get dressed and Jev isn't able to process video, but it can read a live transcript and it is really great decision making.

6:56Josh Machiz: So it can in real time pick items out of your closet. So you can say, Hmm, I want something a little bit warmer. And it's not like you're giving very direct instructions and saying, I want jacket or like, I want a sweater. You're saying, I just want something warmer than what I have. And it can infer and basically make a decision off of the available items in your closet. That's a really great example of how this is like a little bit more advanced than a classifier. So I think things like that, where, you know, in every layer of software, it's, if you think about, again, that customer service email, there are so many points where, do I escalate to a manager is a yes or no question.

7:33Josh Machiz: Is this angry or not is sort of like a probability driven question. So I think there are a lot of different use cases like this. That being said, I think, as I mentioned earlier, the path forward is a combination of the system one and system two, where you use these cheaper JEV-like decision models for that system one quick decision making that does, we know, happens with a lot of these agents. You know, I think even if you will have an extensive discussion on agents later, but even if you think about the idea that we have agents checking our emails or checking if something's high priority or not helping a shop, all of those, yes, consist of system two deep reasoning, but they also consist of these very quick yes or no instinctual things that like sit in our heads.

8:15Well, this definitely feels very special, but of course they're a very new team and this was a$40 million seed. What do you think is the moat here? And like, is it something that the bigger labs could very easily just do themselves?

8:28Josh Machiz: That's a really good question. I'm not sure. I mean, I've been seeing already like open source versions. Yeah. So there isn't a publicly available paper. And so we don't have a strong sense of under the hood what's happening here. I do think this team is, you know, there's a reason why folks have backed this team. Of course, I mean, they're a goaded team. Yeah. And so I do think from a technical standpoint, I imagine at the training and data level, they've done something really interesting here. I don't know what that is because we don't have any papers available. We'll see. and I probably should dig a little bit more into like the OpenGEV versions.

9:01Josh Machiz: But I do think this is really an advantage for people who are playing at the application layer. I wouldn't count out the models building some version of this. I think with the big foundation model providers, there's always going to be a desire to build a suite of tools and a stack of things that, you know, you look at OpenAI with both their frontier models that are extremely closed, but they also have a lot of open source. And so, yeah, I wouldn't count the big labs out. But for now, just from like a Mo and technical standpoint, there's just not enough information out there for me to truly say.

9:34It does seem they've gotten a lot of usage.

9:36Josh Machiz: Another one I saw was actually building a real-time ad blocker. So, again, this idea that, yes, we've had classifiers, but you now have a much smarter classifier with Jeb. And they were able to very quickly on a page decipher whether or not something was an ad or looked like an ad. and it immediately blocked it. So that was pretty cool. Well, maybe from blocking ads to blocking agents. We had another big story this week. Muse has certainly taken over the timeline. I think we've talked about this a lot in the show. Muse, Instinct, all of these agents are taking the internet by storm. But this week, Amazon blocked Muse from crawling its site.

10:19Why did it do this? What are the implications for other agents of its kind? Talk to us about this story.

10:27Josh Machiz: Yeah, there's definitely always a lot of beef around whose agents can use whose platforms and traverse whose interfaces. And so this is actually, I feel like another iteration of that. Actually, back in, I believe, November of last year, I might be quoting the date wrong, but there was a similar case where Amazon sued Perplexity over a similar function where perplexities agents were using Amazon. And their argument was that agents didn't have an identity. This was a massive security issue. But this is not the first time Amazon has been territorial over the idea that humans now have agents shopping on their website.

11:04Yeah, of course. And this is them saying they're just defending their walled garden here. But the other sleeper thing is that they run a$68 billion ads business. What happens to a$68 billion ads business when agents aren't looking at your ads?

11:23Josh Machiz: Yeah, that's pretty much the big reason why they are so adamant about blocking Muse or any of these other agents. Specifically, if you are a user now today, if you try to use Amazon through Muse, it'll say, continued access by an unauthorized AI agent violates Amazon's conditions of use. And I actually think Resi and other platforms have also maybe had some more like anti-bop postures. But the reason why they really don't want agents on there is, to your point, this$68 billion. Yeah, when you go and you shop on Amazon, there's always sponsored links. And there are a number of different advertising products on the Amazon platform that they want you there because it's a big part of their business model.

12:05Josh Machiz: And the difference is if you are an agent, you are not distracted by the random sponsored listing or being upsold on random extra items that are shown to you on the side or in the suggested listings. There's just so much surface area as a human where like, even for me, I just, I go on Amazon for one thing and then suddenly I've eight things in there. That sort of behavior doesn't happen with an agent. If I'm using Muse or Instinct or GrokBot and I'm like, go on Amazon and find me toilet paper, it'll come back with three listings. I'll pick through the three listings and pick one. Done. That's a very efficient workflow, advantageous to me as a agent user.

12:40But if you're Amazon, to your point,$68 billion of ad

12:45Josh Machiz: revenue is really hard to say goodbye to. And so you understand why this is a really big threat. That's a massive threat. Wow. And this is not to say that overnight, I think all advertising is going away. I do think there are, you know, the shape of advertising is changing. I'm so curious what your thoughts are, because you have so much exposure to this and like how eyeballs navigate the internet. But, you know, I've also seen the success of things like ChatGPT ads. I think it sounded like ChatGPT hit a billion in ad revenue recently. So there are models where you can display ads in agentic or chat surface areas.

13:23I also thought it was interesting to see Shopify sign on so quickly to almost take the counter take to Amazon.

13:31Josh Machiz: Why do you think that is? I think they are realizing it's inevitable and that people are going to want to shop this way because there's so much less friction in having a back and forth conversation with an agent, asking it to rank the best products in a category. And also Shopify has lots of products that fall into this category of things you would buy a little bit more on a whim. Yeah. I would also say Shopify is like sort of an intermediary and like a platform provider to actual retailers. Fair. They are. Yeah. And in fact, they democratize those retailers, right? And make it almost able to be as easy to shop on as Amazon and all Muse and Instinct and all these other agents do is they make that even more seamless.

14:18Josh Machiz: Yeah. And so it's an even larger threat to Amazon's business as a whole. Yeah. It'll be interesting though. I would be shocked to see if Amazon can hold out long-term on this. I'm sure they'll develop. They're going to force all of us to use Rufus. There'll be Rufus and like it'll, it'll be shopping. Have you played around with Rufus? I mean, have you used it in the search bar? I accidentally used it. And I keep being like, I need to exit. It takes longer than the search. And if that's the case, like you usually kind of just easily figure out how not to use that feature. It'll be interesting to see if they manage to innovate there.

14:52But I do think that especially with Muse being number one in the app store this week, right? That's a massive thing that I don't think anyone saw coming. I certainly didn't see, like, I thought Meta's Super Intelligence Lab was doing some really cool things, but I did not see them having this massive consumer hit. Again, it felt like they had kind of lost their stride. And Muse and everything Alexander Wang has been posting on the timeline has just completely proven me wrong.

15:22Josh Machiz: Yeah, I definitely want to talk a little bit more about Muse. But my quick thought on the Shopify part is that how you're seeing this play out is that like retailers are very upset about this. Like the actual retailers like an Amazon or Walmart, where they own inventory and that is that threat. Versus if you're Shopify and you're like that intermediary layer. there's no threat you're the platform you're just a platform and actually if anything you're sort of getting a lot more people onto shop pay which is another thing that you benefit from so you know Shopify they in essence sell their monthly subscription they sell the storefront they cut a part of a payment that every that runs through its rails and that's great for them I don't think they're largely impacted but I think to your question I'm just so curious what happens when Like is Amazon and Walmart, are they going to hold strong?

16:11Josh Machiz: But I also recognize the threat where someone had shared stats that with Walmart's usage of chat should be cheap, people actually had lower conversion and higher abandonment rates of their carts because it wasn't like a visual surface area or shopping experience. So, yeah, I don't know. I think. I know. For me, these things make it so easy to shop that I find myself over shopping, right? Where I'm like, oh, this is so fun. And the next thing you know, I've racked up all these things that maybe I needed, but really it's just so fun to shop with Muse and Instinct. And it's so seamless that I actually think it probably makes shopping easier.

16:53Yeah, I'm curious because I feel like there's been sort of a debate on whether or not shopping is like a core agentic use case.

17:01Josh Machiz: I think for sure, like errands, toilet paper, I don't need to look at it. I don't really honestly care. As long as, you know, two ply, like I'm happy. But I have had, I've heard some pushback that with shopping, shopping itself is an experience. Absolutely. Well, it depends. I mean, you and I are always shopping. We're always shopping. But there's like, there are levels to this, right? Like there are things that don't feel like they need to be an experience. And then there's a Cartier Love bracelet or a computer, like things that where you kind of have this like you want to both be informed and have like some sort of an experience when you purchase the thing.

17:44And then there's all the things that maybe you don't need best.

17:47Josh Machiz: I think the discovery part is still going to be very human. Like I love being able to look at a lot of things and figure out which ones I want to go home with. But the actual checkout process, I think, will be agentic. What's one thing that you wouldn't use one of these for that you like absolutely love? And maybe like what's, you know, something where you're like, I'm so happy to delegate this to. So it's not necessarily part of the shopping, but it has been, you know, like as someone who is constantly waiting for product drops. One of my favorite instincts slash use cases is I know something is coming soon or I've heard of it or I've seen it come out first on the EU website.

18:31Josh Machiz: and I know it's going to drop in the US. So I'll ask Instinct and Muse to research that product. And I'll say, can you check every 12 hours if this has finally been listed on the shopping website? And then it immediately does. And then I get to be one of the first few people that buys it. I'm no longer personally refreshing a tab to see if something comes out. I just have Instinct or Muse. I mean, it's game changing for all those kinds of things, like concert tickets. I've found that, especially on flights you know when like you're in a bad seat like just having it be persistent and going after finding you a better seat on the plane is like a total life-changing experience I think it was like Ben Thompson on TBPM but he was saying that like flights he still does not feel like he would give to an agent but I'm like I don't let I don't enjoy having to pick a flight or that's not a shopping experience that people yes look forward to I also feel like there are certain parameters I can give these agents and say like, I prefer window, but then like, if there isn't window, I'll take aisle.

19:34Josh Machiz: And so I actually, as long as it fits within that, I'm okay. Yeah. So I got the new iPhone last week and you did. Yeah. Um, I, I, I didn't do the, I just got the 18 pro max. I like the better camera. I'm sure you do too. Um, and And when I ordered it, I ordered it using Apple's software. I thought about using Muse to like, you know, jump in and get it. But I was like, that's too big a purchase. But as soon as I got it, I was like, wait, I now need the case. And so I use Muse to do all the research on like the best cases for the new phone and then found one that I liked from Muse and then asked it to find the best coupon codes.

20:16and after it found the best one it put it in the cart and then it realized that there was a farewell card that if you had a cart abandoned and got me another five percent off so there's like yeah there are some deals to be had by using these things I have a friend who was even went to Neiman Marcus and tried on a bunch of clothes found some that were too expensive and then just told Muse, hey, if any of these go on sale, let me know.

20:45Josh Machiz: And so it's like, it can even change sort of offline behavior. So there's just so much more we have to explore in this new agentic world. And it's, it's shocking that it came so fast, but I do think for the rest of the world, this is going to take a long time. I don't think Amazon and Walmart are having, you know, an end of day threat today. Yeah, I don't think overnight they're going to lose$68 billion. No. It's the same way Google didn't lose traffic overnight when ChatGPT launched, right? Like there was a hit, but people still use Google. But you did mention the insane growth of Muse. I'm curious, what are your thoughts on the launch and why you think it's been so successful?

21:26It's actually kind of confusing to me. I feel like they launched it sort of like Meta's launched quite a few products where they didn't give it like massive fanfare when they launched it. Like they didn't go as hard as they did with threads, for instance, where there were like threads dinners and like they were really making an effort to go after X in a big way. this felt like they were treating it very much the way a startup launches something where it's like launch it, show that there's fit, and then like pile on the activity. And so I actually enjoyed it. I thought it was a better launch than some of their other launches because it was more authentic to like the startup experience than it was to something feeling so productized.

22:13Josh Machiz: Yeah. And I also think there's been obviously a lot of pushes from meta in the open source or keeping up with the frontier model wise. But this felt like very authentic to the original brand of like meta being a very consumer heavy company. I know kind of back to their roots, which is, I think, exciting. And I mean, look, it was rewarded in their stock. It's up 11%. And that to me was pretty shocking because you would think that their own product would be, you know, at odds with all of the ads that they serve on Instagram and Facebook. And how about Marketplace? You can now go on Muse and shop directly in Facebook Marketplace and negotiate with someone.

22:54And all the ads are not going to be seen by you even during that negotiation process.

23:01Josh Machiz: Yeah. I think what was interesting for me with this launch was that they actually delayed it a couple of months apparently. So Muse as a product apparently was relatively ready even a couple of months ago. But I think there was sort of a a moment where they had seen a lot of the safety headlines come out, Instinct came out. And I believe the team, just from the whispers that I've heard, and I think the information reported on this, but apparently they actually held the product back because they wanted to implement more safety guard reels. And I actually think that's also another moat. As I think about the average consumer, I know that metam people have mixed thoughts on their ability to be trusted and safety given the reason.

23:43We've all given so much data to Meta. I feel like people are concerned about giving more of their data to that company.

23:50Josh Machiz: I think then there's also sort of this question of for your average person, do they trust like an up and coming startup or do they trust Meta, which might not have the best reputation, but they have like an enterprise safety team. And they have the battle scars from these trust and privacy things in the past. I think they specifically hired one of the founders of Signal and also an X-Stripe guy. So I think they were actually quite intentional this time about who they brought on in regards to safety and payments, which we know are probably like if you do make some product mistakes there, you lose a lot of people.

24:23Josh Machiz: And so I actually thought that was a really good move on their part. Yeah, it seems like they've built a lot into the app to show you that you can remove your data if you want to, which you can't really say for some of the other products in the space. And so go meta. This was very impressive as a launch. And it's exciting to see the timeline so excited about something since ChatGPT. One thing I am excited to track, and I've also been hearing whispers about this is talked about on TPPM, but apparently they are going to start thinking about having like a human concierge. So I believe Grokbot and Instinct have both launched phone calling.

25:04Josh Machiz: Yes. But for Muse, they're actually thinking through, is there a world where there are certain actions that yes, Muse can handle. You can handle a little shopping cart, but like, can they actually handle returning your box? Like the full stack of this is - Like a human concierge, like the Amex concierge, but for Muse, gets on the phone and does this for you. I don't think right now today, Muse is going to be able to call and have a full customer service, like full duplex conversation with someone on the other side, or even if I think about like - You don't want Muse calling your doctor's office and being like, hello, So I would like to make an appointment for Claire.

25:42Josh Machiz: Exactly. And it can do it for virtual forums, but it's not there yet for like some of these human conversations that still are very prevalent in our lives. So I don't know what that looks like. Is it a startup that, you know, I think actually there was a YC startup a couple of batches ago that was like rent a human, which is this idea. It feels like we're moving towards rent a human. Yeah. And I don't know what that looks like, but yeah, that was just an interesting tidbit that I was like, okay, this will still evolve. And I'm curious how they rolled that out. Well, we're going to learn a lot more about some of the products that Meta is launching.

26:14From Muse to Meta Connect this week, I saw you were there.

26:19Josh Machiz: I was. I know you have some on the ground footage to share with our audience. Yeah, I have some very special behind the scenes footage for you all.

26:40It's a simple idea that when you empower people, our alliance can't happen. Do you want to introduce yourselves, everyone? Hello, everyone. I'm Agrippa, and I'm here to brainstorm and get things done with you. One, two, three. Hi! It's great game. Hey, Cooper, you're on stage. Why don't you give people a wave? we have finally solved the truly very hard problem of getting new legs in VR.

27:11Josh Machiz: Isn't the best part? Honestly, it's just doing the action part. Okay, I think these fit well on the keyboard, but if you don't want to carry that around.

27:23All right. And then this week, Skilled launched a very cool video on the timeline of a robot playing soccer with one of its employees. This isn't the first time a robot's played soccer.

27:35Josh Machiz: We've talked a lot about humanoid Olympics. We love the Olympics. We've watched humanoids do dishes. We have them cleaning. MMA fights. MMA fights against humans. Yeah. Crazy week. But what makes this different, Claire? Yes. So I think output wise, you might just be like, to your point, another video of robots doing cool things. But I think under the hood, there are some really crazy technical advancements here. Before I get into why it's such a breakthrough, it's important to understand how robot brains get built today. And Skilled in particular is targeting this idea that you have one brain built for many different robot bodies.

28:12Josh Machiz: So as opposed to me specifically building a pancake flipping model just for a pancake flipping machine, I have one brain that I can plug in and it can run in context learning and generalize across many different contexts to new rooms that it's been in. That's what Skilled is trying to build and why it's such an interesting vision. However, historically, there's been a data problem in how they get trained. What's the data problem? Yeah, there are a couple existing methods. So you either engineer the behavior directly. This is like hand coded behavior. This is one task at a time. Or there's teleoperation and imitation where you have a human pilot that sort of uses the same robot hands running the same action thousands of times.

28:58Josh Machiz: And then the model imitates the recordings and learns from that. And that costs a lot and you need a lot of human time. That's like not very scalable. And then the more recent advancement that came out a couple weeks ago that we chatted about is this idea of one-shot video imitation. So you show the model one video and then it's able to copy you. So that's what they launched in August. With the pancakes. Yes. And that is actually already a major, major change. Incredible. But the difference here is that they took a model and they dropped it in a simulator. And what I mean by that is it had a copy of itself and the researchers just walked away.

29:34Josh Machiz: They didn't teach anything. The model had like existing knowledge on running, walking, kicking, but they basically had the model run in the simulation and equivalent about 140 years of play in the simulation and the model improved by playing itself. So every time it got better, it got reward when it got, it scored a certain way. that was a feedback loop back to the model, that that's a method you can use to win. And so over this 140 years of hypothetical simulated play, it learned to dribble, tackle, shoot, even recover from falls. Nobody programmed that into the model. And then they actually loaded that policy into a real robot and it played a human.

30:14Josh Machiz: And what was crazy is it worked. And you see that in the video. And this matters because it basically shows that you can sort of remove humans from the training and teaching loop, which is currently the most time consuming and unscalable part of robotics. So if you have training data that can come from the model in a simulation and its own play, you almost might even have a world where the skills exceed how humans teach it. You saw that actually with AlphaGo, where AlphaGo actually stopped learning from human games. They had a similar simulation where it derived its own methods. This is sort of the same principle, but in soccer.

30:55This is like the third breakthrough we've seen from Skilled in a couple of months here. It's starting to feel like robotics is accelerating at the same pace that ChatGPT and Claude and all of the models on the AI front that were moving just as quickly on the robotics side at this point.

31:16Josh Machiz: Yeah, I think this is going to be huge for robotics getting better without human data, which we currently don't have very much of. And still to be unlocked, but this idea that robots might get better than their teachers, whatever that looks like. I know. Scary. It feels like we are certainly getting to a place where robots are going to be better than us. And we've seen it from the Olympics. It's getting closer and closer to surpassing humanity and so many different vectors. Very exciting to see the pace of robotics catch up this quickly.