After LLMs: Spatial Intelligence and World Models — Fei-Fei Li & Justin Johnson, World Labs

27:27So would you need two different models to do the visual one and the architectural generation, you think? Eventually, like there's not anything fundamental about the approach that you've taken on the model building. it's more about scaling the model and the capabilities of it or like is there something about being very visual that prohibits you from actually learning the physics behind this so to speak so that you could trust it to generate a CAD design that then is actually going to work in the real world I think this is a matter of scaling data and and bettering model I don't think there's anything fundamental that separates these two?

28:05Yeah, I would like it to be one model. But I think like the big problem in deep learning in some sense is how do you get emergent capabilities beyond your training data? Are you going to get something that understands the forces while it wasn't trained to predict the forces, but it's going to learn them implicitly internally? And I think a lot of what we've seen in other large models is that a lot of this emergent behavior does happen at scale. And will that transfer to other modalities and other use cases and other tasks? I hope so. But that'll be a process that we need to play out over time and see.

28:32Is there a temptation to rely on physics engines that already exist out there that are, you know, basically the gaming industry has saved you a lot of this work? Or do we have to reinvent things for some fundamental mismatch? I think that's sort of like climbing the ladder of technology, right? Like in some sense, the reason that you want to build these things at all is because maybe traditional physics engines don't work in some situations. If a physics engine was perfect, we would have sort of no need to build models because the problem would have already been solved. So in some sense, the reason why we want to do this is because classical physics engines don't solve problems in the generality that we want.

29:07But that doesn't mean we need to throw them away and start everything from scratch. We can use traditional physics engines to generate data that we then train our models on. And then you're sort of distilling the physics engine into the weights of the neural network that you're training. I think that's a lot of what, if you compare the work of other labs, people are speculating that Sora had a little bit of that. Genie 3 had a bit of that. um and gd3 is like explicitly like a video game like you have controls to yeah to walk around in and i i always think like it's really funny how the things that we invent for fun actually does eventually make it into serious work yeah the whole ai revolution started by graphics chips partially misusing the gpu for uh for generating a lot of triangles into generating a lot of everything else, basically.

29:56We touched on Marble a little bit. I think you guys chose Marble as I kind of feel like you're sort of a little bit coming out of stealth moment, if you can call it that. Maybe we can get a concise explanation from you on what people should take away, because everyone here can try Marble, but I don't think they might be able to link it to the differences between what your vision is versus other, I guess, generative worlds that you may have seen from other labs. So Marble is a glimpse into our model, right? We are a model of spatial intelligence model company, we believe spatial intelligence is the next frontier.

30:29In order to make spatially intelligent models, the model has to be very powerful in terms of its ability to, you know, understand, reason, generate in very multimodal fashion of worlds, as well as allow the level of interactivity that we eventually hope to be as, you know, complex as how humans can interact with the world. So that's the grand vision of spatial intelligence, as well as the kind of world models we see. Marble is the first glimpse into that. It's the first part of that journey. It's the first in-class model in the world that generates 3D worlds in this level of fidelity that is in the hands of the public.

31:15It's the starting point, right? We actually wrote this tech blog. Justin spent a lot of time writing that tech blog. I don't know if you had time to browse it. I mean, Justin really broke it down into what are the inputs we can, multimodal inputs of Marble, what are the kind of editability, which is, you know, allows user to be interactive with the model. And what are the kind of outputs we can have? Yeah. So Marble, like basically one way of looking at it, it's the system, it's a generative model of 3D worlds, right? So you can input things like text or image or multiple images, and it will generate for you a 3D world that kind of matches those inputs.

31:54And it's also interactive in the sense that you can interactively edit scenes. Like I could generate this scene and then say, I don't like the water bottle, make it blue instead. Like take out the table, like change these microphones around. And then you can generate new worlds based on these interactive edits and export in a variety of formats. And with Marble, we were actually trying to do sort of two things simultaneously. And I think we managed to pull off the balance pretty well. One is actually build a model that goes towards the grand vision of spatial intelligence. And models need to be able to understand lots of different kinds of inputs, need to be able to model worlds in a lot of situations, need to be able to model counterfactuals of how they could change over time.

32:30So we wanted to start to build models that have these capabilities. And Marble today does already have hints of all of these. But at the same time, we're a company, we're a business. We were really trying not to have this be a science project, but also build a product that would be useful to people in the real world today. So while Marble is simultaneously a world model that is building towards this vision of spatial intelligence, it was also very intentionally designed to be a thing that people could find useful today. And we're starting to see emerging use cases in gaming, in VFX, in film, where I think there's a lot of really interesting stuff that Marble can do today as a product.

33:06And then also set a foundation for the grand world models that we want to build going into the future. Yeah, I noticed one tool that was very interesting because you can record your scene inside. Yes. It's very important. The ability to record means a very precise control of camera placement. In order to have precise camera placement, it means you have to have a sense of 3D space. Otherwise, you don't know how to orient your camera and how to move your camera. So that is a natural consequence of this kind of model. And this is why this is just one of the examples. Yeah, I find when I play with video generative models, I'm having to learn the language of being a director because I have to move them out, like pan, you know, like dolly out.

33:54But even there, you cannot say pan 63 degrees to the north, right? You just don't have that control. Whereas in marble, you have precise control in terms of placing a camera. Yeah, I think that's one of the first things people need to understand. It's like, you're not generating frame by frame, which is like what a lot of the other models are. What are, you know, people understand that NLLM generates one token. What are like the atomic units? There's kind of like, you know, the meshes, there's like the splats, the voxels. There's a lot of pieces in a 3D world. What should be the mental model that people have of like your generations?

34:29Yeah, I think there's like what exists today and what could exist in the future. So what exists today is the model natively output splats. So Gaussian splats are these like, you know, each one is a tiny, tiny particle that's semi-transparent, has a position orientation in 3D space. And the scene is built up from a large number of these Gaussian splats. And Gaussian splats are really cool because you can render them in real time really efficiently. So you can render on your iPhone, render everything. And that's how we get that sort of precise camera control because the splats can be rendered real time on pretty much any client-side device that we want.

35:00So for a lot of the scenes that we're generating today, that kind of atomic unit is that individual splat. But I don't think that's fundamental. I could imagine other approaches in the future that would be interesting. So there are other approaches that even we've worked on at World Labs, like our recent RTFM model, that does generate frames one at a time. And there the atomic unit is generating frames one at a time as the user interacts with the system. Or you could imagine other architectures in the future where the atomic unit is a token, where that token now represents some chunk of the 3D world.

35:30And I think there's a lot of different architectures that we can experiment with here over time. I do want to press on, double click on this a little bit. My version of what Alessio was going to say was like, what is the fundamental data structure? of a world model because exactly like you said like it's either a Gaussian splat or it's like the frame or what have you uh you also in in the previous statements focus a lot on the physics and the forces which is something over time which is loosely i don't see that in marble i presume it's not there yet maybe if there was like a marble 2 you would have movement or is there a modification to Gaussian splats that makes sense or would it be something completely different Yeah, I think there's a couple of modifications that make sense.

36:09And there's actually a lot of interesting ways to integrate things here, which is another nice place of working in this space. Then there's actually been a lot of research work on this. Like when you talk about wacky ideas, like there's actually been a lot of really interesting academic work on different ways to imbue physics. We can also do wacky ideas in industry. All right. But then it's like Gaussian splats are themselves little particles. There's been a lot of approaches where you basically attach physical properties to those splats and say that each one has a mass or like maybe you treat each one as being coupled with some kind of virtual spring to nearby neighbors.

36:38And now you can start to do sort of physics simulation on top of splats. So one kind of avenue for adding physics or dynamics or interaction to these things would be to predict physical properties associated with each of your splat particles and then simulate those downstream, either using classical physics or something learned. or, you know, the kind of the beauty of working in 3D is things compose and you can inject logic in different places. So one way is sort of like we're generating a 3D scene. We're going to predict 3D properties of everything in the scene. Then we use a classical physics engine to simulate the interaction.

37:10Or you could do something where like as a result of a user action, the model is now going to regenerate the entire scene in splats or some other representation. And that could potentially be a lot more general because then you're not bound to whatever sort of, you know, physical properties you know how to model already but that's also a lot more computationally demanding because then you need to regenerate the whole scene in response to user actions but i think this is a this was a really interesting area for for future work and for uh adding on to to into potential marble too as you say yeah there's opportunity for dynamics right what's the state of like splats density i guess like do we can we render enough to have very high resolution when we zoom in are we limited by like the amount that you can generate the amount that we can render?

37:55Like how are these going to get super high fidelity, so to speak? You have some limitations, but depending on your target use case. So like one of the big constraints that we have on our scenes is we wanted things to render cleanly on mobile and we wanted things to render cleanly in VR headsets. So those devices have a lot less compute than you have in a lot of other situations. And like if you want to get a splat file to render at high resolution, high like 30 to 60 FPS on like an iPhone from four years ago, then you are a bit limited in like the number of splats that you can handle. But if you're allowed to like work on a recent, like even this year's iPhone or like a recent MacBook, or even if you have a local GPU, or if you don't need, if you don't need that 60 FPS, 1080p, like then you can relax the constraints and get away with more splats and that lets you get higher resolution in your scenes.

38:43One use case I was expecting, but didn't hear from you was embodied use cases. You're just focusing on virtual for now? If you go to WorldLab's homepage, there is a particular page called MarbleLabs. There we showcase different use cases. And we actually organize them in more visual effect use cases or gaming use cases, as well as simulation use cases. And in that, we actually show this is a technology that can help a lot in robotic training, right? This goes back to what I was talking about earlier in speaking of data starvation, robotic training really lack data. You know, high fidelity real world data is absolutely very critical, but you're just not going to get a ton of that.

39:32Of course, the other extreme is just purely Internet video data, but then you lack a lot of the controllability that you want to train your embodied agents with. So simulation and synthetic data is actually a very important middle ground for that. I've been working in this space for many years. One of the biggest pain point is where do you get the synthetic simulated data? You have to curate assets and build these, compose these complex situations. And in robotics, you want a lot of different states. You want the embodied agent to interact in the synthetic environment. And Marble actually is a really potential for helping to generate these synthetic simulated worlds for embodied agent training.

40:22Obviously, that's on the homepage. It'll be there. I was trying to make the link to, as you said, you also have to build a business model. The market for robotics obviously is very huge. Maybe you don't need that or maybe we need to build up and solve the virtual worlds first before we go to embodied. And obviously, that is to be decided. I do think that... Because everyone else is going straight there, right? Not everyone else, but there is an excitement, I would say. But, you know, I think the world is big enough to have different approaches. Yeah, I mean, and we always view this as a pretty horizontal technology that should be able to touch a lot of different industries over time.

41:06And, you know, Marble is a little bit more focused on creative industries for now. But I think the technology that powers it should be applicable to a lot of different things over time. And robotics is one that, you know, is maybe going to happen sooner than later. Also design, right? It's very adjacent to creative. Oh, yeah, definitely. Like, I think it's like the architecture stuff. Yes. Yeah, I mean, I was joking online. I posted this video on Slack of like, oh, who wants to use Marble to plan your next kitchen remodel? It actually works great for this already. Just like take two images of your kitchen, like reconstructed in marble, and then use the editing features to see what would that space look like if you change the countertops or change the floors or change the cabinets.

41:41And this is something that's, you know, we didn't necessarily build anything specific for this use case. But because it's a powerful horizontal technology, you kind of get these emergent use cases that just fall out of the model. We have early beta users using a API key that is already building for interior design use case. I just did my garage. I should have known about this. Next time you remodel, we can be of help. Kitchen is next, I'm sure. I'm curious about the whole spatial intelligence space. I think we should dig more into that. One, how do you define it? And what are the gaps between traditional intelligence that people might think about LLMS?

42:24When, you know, Dario says we have a data center full of Einstein's, that's like traditional intelligence, it's not spatial intelligence. What is required to be spatial intelligent? First of all, I don't understand that sentence, a data center full of Einstein's. Ooh. I just don't understand that. It's not a deep one. It's an analogy. It's an analogy. Well, so a lot of AI as a field, as a discipline is inspired by human intelligence. right? Because we are the most intelligent animal we know in the universe for now. And if you look at human intelligence, it's very multi-intelligent, right? There is a psychologist, I think his name is Howard Gardner, in the 1960s, actually literally called multiple intelligence to describe human intelligence.

43:15And there is linguistic intelligence, there's spatial intelligence, there is a logical intelligence and an emotional intelligence. So for me, when I think about spatial intelligence, I see it as complementary to language intelligence. So I personally would not say it's spatial versus traditional because I don't know tradition means what does that mean? I do think spatial is complementary to linguistic. And how do we define spatial intelligence is the capability that allows you to reason, understand, move, and interact in space. And I use this example of the deduction of DNA structure, right? And of course, I'm simplifying this story, but a lot of that had to do with the spatial reasoning of the molecules and the chemical bonds in a 3D space to eventually conjecture a double helix.

44:20And that ability that humans, or Francis Crick and Watson had done, it is very, very hard to reduce that process into pure language. And that's a pinnacle of a civilizational moment. But every day, right, I'm here trying to grasp a mug. This whole process of seeing the mug, seeing the context where it is, seeing my own hand, opening of my hand that geometrically would match the mug and touching the right affordance points. All this is deeply, deeply spatial. It's very hard. I'm trying to use language to narrate it. But on the other hand, that narrated language itself cannot get you to pick up a mug.

45:09Yeah, bandwidth constraint. Yes. I did some math recently on like, if you just spoke all day, every day for 24 hours a day, how many tokens do you generate? At the average speaking rate of like 150 words per minute, it roughly rolls up to about 215 ,000 tokens per day. And like your world that you live in is so much higher bandwidth than that. Well, I think that is true. But if I think about Sir Isaac Newton, right? It's like you have things like gravity at the time that have not been formalized in language that people inherently spatially understand that things fall. Right. But then it's helpful to formalize that in some way or like, you know, all these different rules that we use language to like really capture something that empirically and spatially you can also understand.

45:56But it's easier to like describe in a way. So I'm curious, like the interplay of like spatial and like linguistic intelligence, which is like, okay, you need to understand some rules are easier to write in language for then the spatial intelligence to understand. But you cannot, you know, you cannot write, put your hand like this and put it down this amount. So I'm always curious about how you leverage each other together. I mean, if anything, like the example of Newton, like Newton only thinks to write down those laws because he's had a lot of embodied experience in the world. Right. Yeah, exactly.

46:28And actually, it's useful to distinguish between the theory building that you're mentioning versus like the embodied, like the daily experience of being embedded in the three dimensional world. Right. So to me, spatial intelligence is sort of encapsulating that embodied experience of being there in 3D space, moving through it, seeing it, actioning it. And as Fei-Fei said, you can narrate those things, but it's a very lossy channel. It's just like the notion of, you know, being in the world and doing things in it is a very different modality from trying to describe it. But because we as humans are animals who have evolved interacting in space all the time, like we don't even think that that's a hard thing, right?

47:04And then we sort of naturally leap to language and then theory building as mechanisms to abstract above that sort of native spatial understanding. And in some sense, LLMs have just like jumped all the way to those highest forms of abstracted reasoning, which is very interesting and very useful. But spatial intelligence is almost like opening up that black box again and saying maybe we've lost something by going straight to that fully abstracted form of language and reasoning and communication. You know, it's funny as a vision scientist, right? I always find that vision is underappreciated because it's effortless for humans.

47:35You open your eyes as a baby, you start to see your world. We're somehow born with it. But you have to put effort in learning language, including learning how to write, how to do grammar, how to express. And that makes it feel hard. Whereas something that nature spends way more time actually optimizing, which is perception and spatial intelligence, is underappreciated by humans. Is there proof that we are born with it? You said almost born. So it sounds like we actually do learn after we're born. When we are born, our visual acuity is less and our perceptual ability does increase. But we are, most humans are born with the ability to see.

48:22And most humans are born with the ability to link perception with motor movements, right? I mean, the motor movement itself takes a while to refine. But, and then animals are incredible, right? Like I was just in Africa earlier this summer, these little animals, they're born and within minutes they have to get going. And otherwise, you know, the lions will get them. And in nature, you know, it took 540 million years to optimize perception and spatial intelligence and language. The most generous estimation of language development is probably half a million years. Wow. Yeah. That's longer than I would have got to say.

49:05I'm being very generous. Yeah, no, I was, you know, sort of going through your book and I was really realizing that one of the interesting links to something that we covered on the podcast is language model benchmarks and how Winogrand actually put in all these sort of physical impossibilities that require spatial intelligence, right? Like A is on top of B, therefore A cannot fall through B is obvious to us, but to a language model, it could happen. I don't know. Maybe it's like a part of the, you know, the next token prediction. And that's sort of what I mean about like unwrapping this abstraction, right?

49:38Like if your whole model of the world is just like saying sequences of words after each other, it's really kind of hard to like, why not? It's actually unfair. Right. But then the reason it's obvious to us is because we are internally mapping it back to some three dimensional representation of the world that we're familiar with. This, the question is, I guess, like, how hard is it? You know, how long is it going to take us to distill from like, I use the word distill. I don't know if you agree with that. To distill from your world models into a language model. because we do want our models to have social intelligence, right?

50:07And do we have to throw the language model out completely in order to do that? No. No, right? Yeah, I don't think so. I think they're multimodal. I mean, even our model, Marvel today takes language as an input. Right. Right. So it's deeply multimodal. And I think in many use cases, these models will work together. Maybe one day we'll have a universal model. I mean, even if you do, like there's sort of a pragmatic thing where people use language and people want to interact with systems using language. Even pragmatically, it's useful to build systems and build products and build models that let people talk to them.

50:40So I don't see that going away. I think there's a sort of intellectual curiosity of saying how, like, intellectually, how much could you build a model that only uses vision or only uses spatial intelligence? I don't know that that would be practically useful, but I think it'd be an interesting intellectual or academic exercise to see how far you could push that. I think, I mean, not to bring it back to physics, but I'm curious, like if you had a highly precise world model and you didn't give it any notion of like our current understanding of the standard model of physics, how much of it it will be able to come up with and like recreate from scratch and what level of like language understanding it would need.

51:17Because we have so many notations that like we kind of use that like we created, but like maybe we'll come up with a very different model of it and still be accurate. And I wonder how much we're kind of limited, but I, you know how people say you won't always need to be like humans because the world is built for humans. And in a way it's like the way we build language constrains some of the outputs that we can get from these other modalities as well. So I'm super excited to follow your work. Yeah, I mean, like there's another, I mean, you actually don't even need to be doing AI to answer that question.

51:46You could discover aliens and see what kind of physics they have, right? And they might have a totally different - We are so far the smartest animal in the universe. But that is a really interesting question, right? Like, is our knowledge of the universe and our understanding of physics, is it constrained in some way by our own cognition or by the path dependence of our own technological evolution? And one way to sort of do an experiment, like you almost want to do an experiment and say, like, if we were to rerun human civilization again, would we come up with the same physics in the same order?

52:14And I don't think that's a very practical experiment to run. You know, one experiment I wonder if people could run is that we have plenty of astrophysical data now on the planet or celestial body movements. Just feed the data into a model and see if Newtonian law emerges. My guess is it probably won't. That's my guess. It's not. The abstraction level of Newtonian law is at a different level from what these language LLMs represent. So I wouldn't be surprised that giving enough celestial movement data, an LLM would actually predict pretty accurate movement trajectories. Let's say I invent a planet surrounding a star and giving enough data, my model would tell you on day one where it is, day two where it is.

53:09I wouldn't be surprised. But F equals MA or, you know, action equals reaction, that's just a whole different abstraction level. That's beyond just today's LL. Okay. What model would you need to not have it be a geocentric model? Because if I'm training just on visual data, it makes sense that you think the sun rotates around the earth, right? But obviously that's not the case. So how would it learn that? Like, I'm curious about all these, like, you know, forces that we talk about. It's like sometimes maybe you don't need them because as long as it looks right, it's right. But like, as you make the jump to like trying to use these models to do more high level tasks, how much can we rely on them?

53:54I think you can need kind of a different learning paradigm, right? So like, you know, there's a bit of conflation here happening where saying, is it LLMs and language and symbols versus, you know, human theory building and human physics? and they're very different because an LL, like the human objective function is to understand the world and thrive in your life. And the way that you do that is by, you know, sometimes you observe data and then you think about it and then you try to do something in the world and it doesn't match your expectations. And then you want to go and update sort of your, your, your, your, your understanding of the world online.

54:26And people do this all the time, constantly, like whether it's, you know, I think my keys are downstairs. So I go downstairs and I look for them and I don't see them and oh no they're actually up in my bedroom so we're like because we're constantly interacting with the world we're constantly having to build theories about what's happening in the world around us and then falsify or add evidence to those theories and I think that that kind of process writ large and scaled up is what gives us f equals ma in Newtonian physics and I think that's a little orthogonal to you know the modality of model that we're training whether it's language or or spatially the way I put it is almost like this is almost more efficient learning because you have a hypothesis of here are the different possible worlds that are granted by my available data and then you do experiments to eliminate the worlds that are not possible and you resolve to the one that's right.

55:12To me, that's also how I also have theory of mind, which is like I have a few thesis of what you're thinking, what you're thinking, and I try to create actions to resolve that or check my intuition as to what you're thinking. You know, and obviously our LLMs don't do any of these. A theory of mind possibly also will break into even emotional intelligence, which today's AI is really not touching at all, right? And we really, really need it. You know, people are starting to depend on these things probably too much. And that's a whole topic of other debate. I do have to ask because a lot of people have sent this to us.

55:52How much do we have to get rid of? you know, is sequence to sequence modeling out the window? Is attention out the window? Like how much are we re-questioning everything? I think you stick with stuff that works, right? So attention is still there. I think attention is still there. I think there's a lot, like you don't need to fix things that aren't broken. And like, there's a lot of hard problems in the world to solve, but let's focus on one at a time. I think it is pretty interesting to think about new architectures or new paradigms or drastically different learning ways to learn. But you don't need to throw away everything just because you're working on new modalities.

56:27I think sequence to sequence is actually, in world models, I think we are going to see algorithm or architecture beyond sequence to sequence. Oh, but here actually, I think there's a little bit of technological confusion. And transformers already solved that for us, right? Like transformers are actually not a model of sequences. A transformer is natively a model of sets. And that's very powerful. But because a lot of the transformers grew out of earlier architectures based around recurrent neural networks, and RNNs definitely do have a built-in architectural, they do model one-dimensional sequences.

57:01But transformers are just models of sets, and they can model a lot of, those sets could be 1D sequences, they could be other things as well. Do you literally mean set theory? Yeah, yeah. Galois, whatever. Yeah, yeah. So a transformer is actually not a model of a sequence of tokens. A transformer is actually a model of a set of tokens. Right? The only thing that gives that injects the order into it in the standard transformer architecture, the only thing that differentiates the order of the things is the positional embedding that you give the tokens. So if you choose to give a sort of 1D positional embedding, that's the only mechanism that the model has to know that it's a 1D sequence.

57:35But all the operators that happen inside a transformer block are either token wise. So that either you have an FFN, you have QKV projections, like you have per token normalization. All of those happen independently per token. And then you have interactions between tokens through the attention mechanism. But that's also sort of, it's permutation equivariant. So if I permute my tokens, then the tension operator gets a permuted output in exactly the same way. So it's actually natively an architecture of sets of tokens. Literally a transform. Yeah. In a math term. I know we're out of time, but we just want to give you the floor for some call to action either on people that would enjoy working at WorldLabs.

58:13What kind of people should apply? What research people should be doing outside of World Labs that would be helpful to you or anything else on your mind? I do think it's a very exciting time to be looking beyond just language models and think about the boundless possibilities of spatial intelligence. So we are actually hungry for talent, ranging from very deep researchers, right? Thinking about the problems like Justin just described, you know, training large models of world models. We are hungry for engineers, good engineers building systems, you know, from training optimization to inference to product.

58:53And we're also hungry for good business, you know, product thinkers and go-to-market and, you know, business talents. So we are hungry for talent, especially now that we have exposed the model to the world through marble. I think we have a great opportunity to work with even a bigger pool of talent to solve both the model problem as well as deliver the best product to the world. Yeah, I think I'm also excited for people to try Marble and do a lot of cool stuff with it. I think it has a lot of really cool capabilities, a lot of really cool features that fit together really nicely. In the car coming here, Justin and I were saying people have not totally discovered that, okay, it's only 24 hours, I have not only discovered some of the advanced mode of editing, right?

59:44Like turn on the advanced mode. You can, like Justin said, change the color of the bottle, change your floor and change the trees. Well, I actually tried to get there, but when it says create, it just makes me create a completely different world. You need to click on the advanced mode. It's a good UI issue. We can improve on our UI UX, but remember to click on it. Yeah, we need to hire people who work on the product. but one thing we got that was clear from you guys are looking for is also intellectual fearlessness which is something that I think you guys hold as principle yeah I mean we are literally the first people who are trying this both on the model side as well as on the product side thank you so much for joining us this was fun thank you guys thanks for having us yeah

1:00:37you