Making taxes fun with Pikachu and AI | Cadi Zhang

0:00We really haven't seen robots operate in the real world yet. Getting their own data and then training on that, I think a very small subset of robots that have been able to like successfully go out in spaces. Because no one really wants, you know, 100 pound robot in their house and like you said, fall on their cat. That would just be catastrophic, right? I was building out PokéTax was just like my spin on filing a tax form in like Pokemon games. And it requires, I feel like, some level of thought and guidance to the AI. Like, what if you never filed any of your taxes in Pokemon? Because you're always earning money as you battle trainers.

0:33And so it's like, you really have to provide some level of guidance.

0:36Parth Patil:The Token Grantee Program gives$1 ,000 a week in tokens to high potential creators already deep in AI. Across film, gaming, comics, print, and digital art. There are no tool restrictions. The freedom to choose is the point. Grantees also get access to my own custom fleet of agents. and an ongoing collaboration with me. The goal, to close the gap between an idea and the world. Welcome to another episode of Read Riffs. We've got our real Parth, not just Parth AI. It's also the real Read, not just Read AI. And we have an excellent guest, Katie. Thank you for joining us. Yeah, I'm super excited to be on.

1:19Thank you guys so much for having me. It's our pleasure and honor. And why don't we start with, okay, so Parth,

1:25Parth Patil:How did you select Katie? Yeah, I'm super excited to have Katie on. Katie, I think we've known each other now for maybe like three or four years. When I met Katie, she was working on a hackathon trying to build a kind of co-pilot for game design and game development. And me being obsessed with video games, I was like, oh, my God, I want this. Like, this would be awesome. Like, I saw what Cursor did for me for programming. But imagine if we had something similar for making games. And so that's when I met Katie. but over the last couple of years, I've gotten to know her as she's exploring AI, but also getting to know her like wide ranging career arc.

1:59Parth Patil:So she has experience in AI for accounting, some crypto transaction analysis, as well as like now she works in robotics as a product operations specialist. So she has this incredible like exploratory career arc, which has been really exciting to follow and seeing how she connects the dots across all of these different problem spaces and using these like the AIs to fill in our gaps has been super inspiring. So I figured, I mean, it would be awesome to like accelerate your own creative projects and see what you can do and bring you onto the show and see what you've, you know, you've been learning over the last few years.

2:34Parth Patil:So welcome Katie. Tell me more about how you got into AI and what your initial goals were and where you kind of ended up and talk about your journey to this point over the last few years. So I think overall, I had a pretty unconventional path into AI in general. I was working on basically creating games in Unity. I was just super obsessed, really just wanted to create something and was working on 2D platformers and such and kind of like entered in and was really lucky at the time with robotics just being there and jumped on because we were basically programming VR interfaces with robotic dogs.

3:14And it was kind of my foray into like actually starting to program and learn. And I think at the time it was, we just started getting introduced to like GPT and just having that as like the magical thing where you're copying and pasting code snippets literally in at the time. And so it's like hilarious to see kind of where we are at now. But that kind of really spiraled into me doing like haptic suits, coding more VR stuff and unity to now, you know, working in AI for a bit for AI accounting, and then now moving over to more of like the data collection robotics portion of, you know, how can we really gather the data?

3:53And what is the data that is needed for the next generation of like embodied intelligence or robotics in general? Very cool. So go a little bit into kind of how building game worlds and intelligence for the physical world. What do they have in common? What do the fields have to learn from each other? What's the bridge, as it were, from bits to atoms as you're moving from game worlds to intelligence for the physical world? They have a lot of crossover than I initially expected. I was quite surprised. And I think the easiest way to even think about it is teleoperations, for example. You are basically in these VR headsets in a game world operating these robots.

4:37And I think like on a software level too, you kind of get to see the advent of these world models coming into play, just really being able to simulate and just recreate these environments and have robotics or robots reasoning through them. I think the main bridge that is being crossed or I guess the similarity here is like, how can we basically simulate 3D worlds, physical realities on top of having physics within them. And I think that's like the bridge that's being crossed. But world models or robotics is kind of taking this to like the next level of ambition, really just going end to end in many ways.

5:19By the way, quick follow-up. What are the things that you've seen so far that are the biggest gaps between them? Because it is classic kind of Silicon Valley AI point of view. Oh, you build it all in simulation, then it moves much faster and it learns the physical world, but there's unevenness in some parts of the ground. What are the biggest gaps you've found so far? Yeah, and I think this is really able to be represented. I think when Project Genie came out and it was just like bloodbath of like stocks of like Unity, Roblox, like all these like gaming companies. And I think they provide like, I think a really great take on like what is the gaps currently in these like world models where essentially like physics and world models is not yet deterministic.

6:01Also, there's no object permanence in some cases. It's only short horizons where you're having the ability to see and interact with the world in a way. So with these in mind, it's like, how do you create these longer form interactive experiences and games other than just a quick first person shooter, a quick tycoon game, on top of the fact that if you don't have determinism, how can you create multiplayer games? It's just something that is not yet you're able to achieve. So I find that as like the gap. In some sense, like I find that there's also an opportunity to even bridge this where, for example, I think a lot of people are talking about why don't you use world models as an input into these like more deterministic systems to create these games.

6:48So there's like kind of that path versus the more implicit path of just pure, like straight ai um and creating these hyper realistic games yeah i think i i showed an early

7:01Parth Patil:what clued me into this paradigm was i saw i saw a demonstration of minecraft i think it was running on oasis maybe on the etched chip so it was like a more of a demonstration of the chip but it was like a minecraft generator trained on minecraft and but it had no uh like it had memory like the memory of a fish so anything it would see that was outside anything that went outside of the frame would be forgotten and then so if you walked up to something and then you turn around everything would change because it wasn't able to like remember and hold on to the details of the world beyond what it could see in front of it what is projectini could you clarify that like how did you how would you describe that to someone that maybe didn't get to play with it or see it yet i think definitely it's like basically you're taking a prompt and then basically able to see and create all these different worlds and interactive environments i would say you know there was Genie 1, which was definitely kind of the very beginning where you're able just to see the introduction into just, you know, interactive worlds.

7:59Genie 2 and Genie 3 now, which is kind of the next level of interactivity, you know, the possibilities of even having the world at your fingertips with just a prompt. Right.

8:11Parth Patil:Like the world unfolds as you explore it on the fly and instead of being predetermined, right? Yes. What's a recent project where AI let you move from an idea to a working prototype faster than you could have before? So I've been playing a lot with how AI and games kind of overlap. I think prior to this, I was struggling a lot figuring out how to use AI and bridge that over to games. I remember the first time I was like, just let it vibe code. Just let it go. and it wrote me 10 ,000 lines of like this horrible HTML, CSS file. It broke, could not get it to work. And I feel like now I feel extremely supercharged in a way of having AI and being able to just create these games.

8:59I think the one of the projects I was working on was essentially it's kind of a spin on the imposter game where everyone's a duck and there's like one killer duck within the mix. And your goal is to act like an NPC and complete a set of tasks. And so that duck is trying to monitor to see which character is actually the human. And it's really been amazing to have AI help me iterate on so many core gameplay loops. It's just amazing to just go to bed, you know, be in the shower and then have AI just spin up five gameplay loops, send them to friends, take that feedback and just rip at it again. And I think the core thing is like, how can we make the game fun?

9:43And that requires a lot of time, a lot of effort and research. And it's being able to minimize that and really just kind of get it into the hands of people is just incredibly powerful. I'm curious what you've learned so far in this is how much does AI understand, you know, kind of well the concept of fun? I'm expecting you to say uneven. but but like given my just made my token prediction of uneven what's the actual answer i would definitely agree i think it's i think ai can approximate i think you can approximate based on you know the games that it probably has seen it has played and it has like in knowledge what has been trained on and so like for example if you give it uh just like examples of the games that you enjoy and want.

10:32Definitely approximate exactly like what are the core gameplay mechanics and give you and provide you suggestions. I've noticed like if it's something like fairly new it struggles. It's not I think it requires a lot of context building still so I wouldn't say AI has a good grasp on fun yet. It still requires a lot and a lot of time and effort being put into it of just like really honing what are the gameplay mechanics it's just faster in the way that it provides you

11:02Parth Patil:like the code to like see it in real time yeah i i see i see the same thing i'll be working on a game and actually most of what i have to do is play the game and then just point out things i'll just i i it's like my job is put the game in the hands of other people and then see and just play the game and be like okay what is anti-fun and then i go to the ai i'm like this experience was very frustrating we should fix it and then the ai comes up with like six alternatives to patch that experience and i'm like oh these are pretty good ideas we can like try them out but definitely it doesn't understand fun like we understand fun and i think that's like the very human kind of level of like being in the game and playing and trying to have your own like experience uh which is exciting because i think i hope that like we're not automating the part of experiencing the game although it is cool to have ai playing the game before you do and highlighting and finding errors is like you kind of have it like bug fixing and like mapping things out yeah i agree i think there's like this level of taste that is in there especially on top of like story creation if you're having these like role-playing games i think i was building out like poké tax was just like my spin on filing a tax form and like pokemon games and um just like even the dialogue the first pass of the iteration i was like well it's just like you know the stereotypical like let me battle you don't go you know to the next level um but i'm like wait what if you know and it requires i feel like some level of thought and um guidance to ai like what if you know you never filed any of your taxes in pokemon because you're always earning like money as you battle like trainers and stuff like there's that whole system um and so it's like you really have to provide some level of guidance in some cases speaking more about guidance like can you point uh you know and talk about a prototype and point to the moment where your input made most of the difference, like where you feel that that guidance comes into play?

12:52I think for me, I spend a lot of time and I love just kind of the visuals of a game. I think it just makes a huge difference of what is like production grade quality. I spend a lot of time figuring out how to like make the assets what I want. I threw AI at blender just like just try it you know just go it did not get what i want um it was quite like an approximate version of like the trees the setting of just just like this kind of like serene space that i wanted and i find that for me it was like i enjoyed just like drawing out like particularly the assets that i would want and so i just used paper and pencil i would just draw it out give it to gbt have it run an image and be like yep that's it and iterate on the prompt with you know play write I would then basically take that create a character sheet and so having that available just made it so much easier to go from like 2d to 3d so either if that was like throwing it into blender or if it was throwing it I think to it's like image to 3js was one of the things I used a lot it was just so much faster to iterate and get to like where I want it to be.

14:04Two follow-ups a tiny one, but I'm very curious, which is why hand draw before putting in? Why not like voice prompting an image generator and then going cycles on that and then putting it in? I'm curious like what the learning is for the kind of the creativity workflow. So that's the small one first and then a second one in a moment. I think for me, it's like I really, I have a particular vision of what I want the assets to be and like how they look. I think AI in general, this is very hard for me. And maybe this is a skill issue, but it's like, I struggle to like get to where I want. There's always these small little things and little details that AI misses, even if I speak to it.

14:51And so I find that, you know, in general with just providing like reference images or any kind of like approximate of like what you need just gets you to just one shot and like closer to the step instead of having to be like this like bow tie is off it's not this color or you know in this manner I would want it in this style so it's just like there's so many things you can probably minimize if you had like coming in with a vision but if you probably didn't I would say I typically would do the same thing that you would mention, which is just continuously iterate with AI. Yeah. Well, given that some of us like me don't have any good drawing skills, that's my only path forward.

15:34But I was curious what the alternative is. Now, the other thing is a little bit like there's kind of in games, there's kind of the theory of fun, which you talked about before and kind of gestured at. There's also the kind of the richness of the world, which then parallels the physical intelligence. So what have you found in kind of, as it were, the world building part of it, where both amazing capabilities of AI have happened, and also still yet TBD challenge elements? Because that'll be part of the lens in terms of getting into the physical world. Yeah. And I think that's a really interesting question.

16:14I think for me, what I see is like the challenge is like the level of granularity, like how much can you drill down to like each individual asset? It's just like I think it's like incredibly easy just to throw AI at like Blender or, you know, at these worlds and be like, just generate me something. But the asset barely moves, you know, like how granular can you get? And so it's like if you have like a packet of tea, there's the packet itself, there's like the tea bags and then inside as well as just like the tea. So it's like how much can you drill down? And I think it's very fascinating to see now AI being thrown at these problems, be like just go at it continuously on a loop, iterating again and again at the level of granularity to reach where everything can be represented fully within these worlds.

17:04I think like that's like the gap that I see.

17:06Parth Patil:More detail, making more hyper-realistic, more detail. Three hours later, I'm like, oh, this is not bad. And then I'm trying to put it in the game. It's like, remove the detail so it renders properly. You speak a little bit about your work and the tools you're using and Blender, especially the latest models. It's been so much fun seeing AI like use your computer, use the tools on our computer, use Blender building 3D assets. what are some tools that people may not know about that you think could deeply impact their work maybe some of your like maybe like top top three top five maybe i think for me what i've been playing around with a lot these days is i mean i love pixel art games i think these are kind of like childhood like octopath traveler and stuff like that um and so i've been playing around with like pixel lab and just like plugging that into asprite and seeing what kind of you know characters I can generate and just like the varying quality.

18:02I think this has been a struggle for me and previously using things like, you know, GPT just to generate pixel art assets. I think now it's gotten like overall across the board a lot better on all fronts, like not just GPT, but Pixel Labs is just fantastic to use and definitely worth a shot. I think it's also really fun to just kind of like see different spaces and try them And so I typically always love to just use the world models in general, like world labs, and just seeing what I'm possibly able to create and using those as like even reference images for the 3D worlds and getting to those like particular assets that I would like.

18:44Let's also dive into one of the things I think is particularly important part of what we're doing and all these kind of creativities, which is there's an amazing amount of capability that AI adds. there's actually some amazing creativity that AI adds, you know, all forms of amplification intelligence. But like also one of the things that we're learning on this journey, and I think it's important to kind of see all the AI cast forward, but also where are the elements of kind of human contribution for amazing, you know, kind of like results? So, and this kind of gets to kind of questions of taste, questions of judgment.

19:25What have you been finding that has been the most important ads from you and from people in the kind of taste and judgment? We got a little bit of like Parth, like I got to play the game to kind of get the theory of fun. But how would you elaborate more of that? I think for me, it's definitely related to like style. And I think a lot of the game actually is figuring out just kind of the visual effects. Like what is like the art that you want to bring into the game, into people's hands? I think art style is just a huge part of like what makes things fun. Like, of course there's that on top of the fact that I still think like AI for me really struggles to hit exactly where you need to be.

20:10Like I've tried having it rig a system and kind of like have that as like an asset within my game. And it still hasn't been able to do that properly. And maybe there's possibly some other products out there that might be doing it way better than what just having like Fable or like Opus run at it. But I definitely think it's a lot about it. It's like the fine tuning. It's like you don't really need to do the things of like, you know, re-skinning an asset, just like kind of things that takes up so much time, but rather focus on like what is like the highest leverage points to get things to be fun.

20:45and to me that is just like the small individual details and of those assets um just like you know the style consistency is this like accurately represented in the game and does this make sense for a character to be kind of like enjoyable and fun for people very cool yeah i kind of like kind

21:02Parth Patil:of i kind of take it back to your theory on robotics when i was in high school we saw the first prototypes in silicon valley of like self-driving cars kind of driving around the parking lot and my dad was like oh this is going to be the future this is going to be the future and then it took like it took like 15 years like it took a long time for them to and now i now i take waymos every i take waymo everywhere um so it's finally here i kind of put off buying a car until the car could drive itself do you feel that with i mean especially in robotics it seems like a lot of people work in robotics you work in robotics i kind of have this like skepticism of like some people like oh it'll be here next year i'm like i don't know i don't know if i trust a robot to like i have robots in my house very minor like very like specialized robots for like handling you know taking care of the cats but there's like minor levels of automation i'm not sure i would like how comfortable i feel with like a robot walking around my house you know like just just don't hurt the cats you know like that's a simple like expectation i have how do you think about how long this will take to this like the robotics wave will take to unfold and how do you think um we get the best version of this robotics future uh what like how do you think about like where this goes and how do we make the best possible future come to fruition?

22:12Yeah, this is a really good question. I think definitely it seems like there is still quite a bit of, you know, iteration that needs to be done. I think the U.S. in general, for me, seems to be much more focused on kind of a research and development phase rather than a deployment phase. We really haven't seen robots kind of like operate in the real world yet. and we haven't really gotten the flywheel of like robots getting their own data and then training on that it's been I think a very small subset of robots that have been able to like successfully go out in spaces because no one really wants like this like you know 100 pound robot in their house and like you said fall on their cat that would just be catastrophic right or like trip and like go out of the window and just, you know, harm someone.

23:01So I think definitely we haven't really seen robotics kind of like in general, the data. It's like LLM's had their moment with kind of just the text available on the internet. And so like, where do we really see that being with robotics? And there's this like question of like diversity and what it means to, you know, really be in these 3D spaces and like basically train AI to understand physics. 3D worlds that has yet to kind of been solved. And so I think a lot of it is trying to discover like what that means. You know, there's YouTube data that is possible. You know, there's also world models and also like egocentric data sets.

23:42But yet, like, is that enough? It's like still remains to be the question. On top of -

23:47Parth Patil:Ego-centric, is that like the first person - Like human, like, yeah, iPhone data. So it's like first person POV, like doing a task or like exploring like that kind of data set? Exactly. It's just like, you know, looking at your hands, just basically those people who used to, you know, like film themselves, like making Subway sandwiches, like similar, very similar to that. It's very interesting to see. And I think there's this level of granularity that has not yet been reached, which is, you know, what is sensor and tactile like related? Do we need these in kind of like these world models and data sets?

24:22Like we're not able to take a YouTube video and be like, yeah, this guy applied X amount of force to grab, you know, this edge of a paper and fold it. So I think like that's the limitation. And it's just very interesting to see kind of like where we'll be. I know it's a very roundabout answer to say that I think robotics still need some time to like really truly take off. I would say definitely we'll see a lot more robotics in commercial environments in like the next like, you know, three to five years. I would say that residential is much more of a scarier place to deploy a robot. But we'll see if, I know everyone loves humanoids.

25:01I think humanoids are not the best form factor. I was going to ask this question, actually.

25:06Parth Patil:Yes, like what is it about the humanoid form? And then also like there are plenty of things we need robots to do that would require shape or abilities that I'm not sure humanoid sized robots can't have. What are your thoughts on that? Yeah, because I think about it in the way it's like, do you really need someone with hands to, you know, grab boxes? Or could you just really, you know, have like this Roomba grow under a pallet, pick it up and then just move like hundreds of boxes, you know, at once? It's just really like, why would you need human hands? I think humans in general, it's extremely cool to see, you know, humanoid robots exist and possibly even solve like the issue of like cross embodiment of like data where a lot of the sets or data sets that are coming out are like egocentric.