How decision making will change when AI answers are cheap and (too) easy

26:53That's the thing. It's going to fit the average Joe. What is or who is the average Joe? And how does that apply to you? For example, it's going to go to Instagram and it's going to look at who's got the most likes and followers, decide that those people have the best lives, and then take the attributes of those people, how they look, how they talk, the level of education that they say they have, whatever, and say, well, here's what you need to do to be like these people who the data tells us people think have the best lives. Is that kind of a version of what you mean? Something like that. More convoluted, because something that is worth realizing is that what machines have over us as an advantage is memory and attention, right?

27:38And what I mean by this is if I flash 50 digits on screen right now, and then I ask you to recall it, you're going to have no idea, then I can kind of go back to those 50 and be like, yeah, the machine remembered it for us this whole time. It is clearly better at memory than John is. Then we flash these things. I say, quick, what's the sum of these digits? Again, difficult for you, easy for a machine. And so anything that kind of fits in our heads as we discuss it is going to be a shortcut of what's actually possible when you have memory and attention at scale. In other words, we've described this Instagram process, that fits in our heads right now, but you should expect that whatever is actually going on with these systems is just too big for us to hold in there.

28:23So sure Instagram, and sure some other sources, and probably some websites about how to live a good life, and even applied philosophy, all kinds of things, all jumbled together. Something too complicated for us to understand what it is, but the important thing is not tailored to us specifically, not without us putting in quite a lot of effort to feed in the information required for that tailoring, which I encourage that we do. I certainly, understanding that advice is cheaper than ever, will frame up whatever is interesting to me, give it to the system, of course removing the most confidential things, but I've asked all kinds of things about how I might, let's say, improve looking at real estate for me with my particular situation and my particular tastes.

29:14Very different answer from if I just say, well, how do I invest? And then I've even improved silly things. Like I discovered that I tie my shoelaces too tight. I had no idea. Thank you, AI. I now have better technique for feet that are less sore. Did you discover through AI that you tie your shoelaces too tight? Yeah, I went, I went debugging. I wanted to try to figure out why my feet are sore. And so, uh, help me diagnose this. I gave a lot of information about me, about when my feet are sore, what I'm doing at the time, what shoes I'm wearing, right? And we got through some little debugging process.

29:52Okay. First thing we'll try is use a different shoelace tying technique from the one that I have, change which loops, and then loosen it a little bit. I'm like, wow, now my feet don't hurt. How awesome. So whatever it is that's bugging you, you could go and try to debug it a little bit with the AI system and just see what you get. Maybe it's useful, maybe it isn't. But if you simply give it nothing and say something like, how do I be as healthy as possible? You'll probably not get any information about what to do with your shoelaces, right? You're just going to get something from very averaged out, smoothed out soup.

30:30So in order to get something useful, you have to bring something to the table. You have to know what's important to you. You have to know what you're trying to achieve. Sometimes because your feet hurt right now, it's important to you right now. And you're kind of reacting the way that I was. I probably wouldn't ask any proactive questions about my shoelaces. But sometimes what really helps is stepping back and saying, well, what is there in my life right now that could be better? And then why not ask for advice? AI makes advice cheaper than ever before. That's a big revolution. It also helps with all kinds of nuanced advice, like pulling out some of your decision framing.

31:08Help me frame my ideas. Help me ask myself the questions that would be important for getting through some or other decision. Where are most people making the biggest mistakes or where do they have the biggest blind spots when it comes to decision making? Is it on asking the right questions? Is it deciding what they want? What would you say it is? One is not getting in touch with their priorities. Again, when you're not in touch with your priorities, anyone's advice, even the best person can give you advice, which will be bad for you. And this is something that also applies to the AI sphere. If we aren't in touch with what we need and want, and we just ask the soup to give us back some, you know, average first draft, and then we follow it to a T, what are the chances it actually fits us?

32:05Let me put a specific on this because I'm the parent of a soon-to-be 17-year-old, second semester junior in high school, who's getting ready to apply to colleges. And this is one of the first major decisions that young people make. And it's two-sided, which is really fraught because you're deciding where to apply and the schools are deciding who to let in. So it seems like that applies here, too, because some people are going to apply to a school because their parents went there or because it's Ivy League or because it's not Ivy League. I don't know. So through that framing, can you talk about the types of mistakes that people make from the perspective of a high schooler applying to college?

32:49I'm going to always keep trying to play this game of tying this back a little bit to what we can learn about our own interactions within LLM, because I think that's helpful for people as well in this brave new world of how do we use these tools. So we have three stages approximately of you have to figure out what's worth asking, right? And what's worth doing. Then you need some advice or technical help or something, some execution bit, which might be you, might be the LLM, might be your dad giving you great advice. And then when you receive the advice, you have some moment where you evaluate, is this actually good for me?

33:32Do I follow this? Is it good advice? Is it bad advice? Do I implement it? Do I execute it? Right? These kind of three stages. And so the first one, the least comfortable one, is asking yourself, well, how do I actually frame what I'm asking? So to apply it specifically to your kid, it would be, what is the purpose of college for me? Why am I even asking this question? What am I imagining are some things I might get out of this college versus that college? What would make them different for me? What are my priorities? Why are these priorities my priorities? These are things where if you are not in tune with your answers to them, what you will do is you will receive advice from wherever, from culture, from the internet, from your dad, and you are likely to end up doing what is good for them rather than what's good for you, right?

34:31From not asking yourself enough. Like the magician scenario, they feed you an answer subconsciously and you end up spitting that back without even realizing it's not what you really wanted. Or consciously. You could have, you know, your dad might say, as my dad did, economics is a really interesting and cool thing to study. And this kind of went into my head when I was maybe 13 and kept knocking around in there. And so that's how I found myself in economics classes and ended up majoring in economics at the University of Chicago. So actually, it's not always true that what your parents put in there makes its way out, of course, because both of my parents were physicists.

35:07And I very quickly discovered that I wanted nothing to do with physics because of the constant parental, you know, you should do better physics and you should take more physics classes. And then, of course, after I rebelled in college, I ended up in grad school taking physics in my neuroscience program. So there you go. Comes around full circle. But the point is that you have to know what you want, what's important to you, really be in touch with this so that you're not pushed around by other people's advice. And even the best advice, this is important, even the best advice could be bad for you.

35:42So when you think someone is competent and capable, and so I should absolutely take their advice, that's a mistake. Because if what's important to them is not what's important to you and you haven't communicated clearly to them or they don't have your best interests at heart, this intelligent advice is going to lead you off a cliff. And so that brings me to the AI setting, right? With AI, it could be a performance system, but if you haven't given it the context to help you, it's not going to help you. AI presents itself as very competent and very certain that it's correct with very little variation that I've seen based on the actual output.

36:21It's not saying, I'm not totally sure, but I think this when it's about to hallucinate versus, oh, here's the answer when it's absolutely right. It's sure almost 100 % of the time. So that's a design choice. Whenever you have actual probabilistic stages in your AI output. You can instead surface something to do with confidence. And this is achievable in many different ways. For some models, even some of the basic models, what happens there is you get a probability first, and then that converts into action or output that the user sees. For other situations, you could, for example, in the back end, you could run that system multiple times.

37:10And you could see, let's say you would ask, what is 2 plus 2? And then in the back end, you could run this, let's say 100 times, and you discover that 99 out of 100 times, the answer comes back with a 4 in it, right? You could show then some kind of confidence around this being at least what the cultural soup thinks the answer is, right? Let's say, what is the capital of Australia? If the cultural soup says over and over that it's Melbourne, which it isn't, or that it's Sydney, which it also isn't, for those for whom that's a surprise, it's Canberra is the right answer. But if enough of the cultural soup says Sydney, and we're only sourcing from the cultural soup, we're not kicking in some extra logic to go specifically to Wikipedia and only draw from that, you would get the wrong answer with high confidence, but it would be possible to score that confidence.

38:00And in situations where the cultural soup isn't so sure of it, then you would have a variety of different responses coming back, being averaged. And then you could say, well, the thing I'm showing you right now is only showing up in 20 % of cases, 10 % of cases. Or you could even give a breakdown. This is the modal answer, the most common answer. And then these are some answers that also show up. Not to do this is very much a user experience design decision plus a compute and hardware decision. It's also a cultural issue, isn't it? It's a cultural issue. It seems to me like in the US, and maybe this is true of a lot of Western cultures, we value confidence and we value certainty even more sometimes than we value correctness.

38:44Like there's this culture in business where we sort of expect right down to the moment when a company fails for the CEO to say, I'm really confident that we're going to make this work. And, you know, because people want to follow somebody who's confident. And then like the next day they say, ah, well, I failed. It didn't work out. And we kind of accept, oh, well, they gave it their best and they were really confident. Same in sports, right? The team's down three games to one in a best of seven series. And the team that's only got one win, they're like, oh, we're really confident we can win. Well, really, the statistics say you're probably not going to win, but we know that they have to be confident if they're going to have any chance.

39:17So we sort of accept that? And in a way, haven't we created that AI in our own image? We've certainly created the AI in our own image. There's a lot of user experience design that goes into that. But I don't think it's an inevitable thing. I know that on the one hand, there is this concept of the fluency heuristic. So a system or person that appears more fluent, less hesitation, less uncertainty, is perceived to be more trustworthy. This is research done. It's old, old research in psychology. Now, you see that the fluency heuristic is absolutely hackable. Because if you forget that you're dealing with a computer system that has some advantages, like memory, attention, and, well, fluency, so you could just rattle off a bunch of nonsense you don't understand very quickly, that lands on the user or the listener as competence and so should be more trustworthy.

40:18So our fluency heuristic is absolutely hackable by machine systems. It's much harder for me to hack it as a human, though we do have bullshit artists who manage it very well, but it's very difficult to speak fluently on a topic that you have no idea about and you don't know how any of the words go together. And that only works if that's the blind leading the blind, where no one else in the room also knows how any of it works. But on the other hand, I'll say, at least for me, I think it has helped me in my career to form a reputation that, well, I say it like it is. And so I'm not going to pretend I don't know a thing when I don't know it.

40:55I even, you asked me about neuroscience, I told you, ah, it's been a long time since my graduate degree. You know, maybe we should adjust what I'm saying, right? I do that. That is not for all markets. Let's just say many would think she has no idea what she's talking about. Maybe we shouldn't do business with her. But for sure, there is value. And I've definitely found it's helped me to become a sort of battle tested trustworthy. Now that said, when it comes to designing AI systems, that stuttering lack of confidence would not be a great user experience. But similarly, some of the things that I talked about here would be expensive compute wise.

41:35So what I see a lot in the AI industry is that we have business people thinking that something is not technologically possible, because it is not being given to users and particularly given to users at scale or even offered to businesses. Quite often, it is very much technologically possible. It is just not profitable to give that feature. There is no good business case. There's no sign that users will respond to it that will make it worth it. So when I'm talking about running something 100 times and then outputting something like a confidence score, also you would have some decision making around, is it 100?

42:17Is it 10? Is it 1 ,000? and this depends on a slew of factors, which of course we could get into if that's the kind of problem that you as a business are solving. But when you just look at it on the surface, I'm saying essentially 100 times more compute, right? Run this thing 100 times instead of once. And for what? Will the users respond to it? Will the business care about it? So yeah, frequently, yeah, you'd be amazed at what's already possible. Agents like operator, the cloud computer use, Project Mariner, all these things, they are underperforming relative to where they could be performing on purpose because it is expensive to run them well.

42:57So yeah, it will be very exciting when businesses and users are ready to pay more for these capabilities. We need to take a quick break. We'll be right back.

43:17Support for the show comes from Rippling. If you're a business owner, here's the truth. SaaS promised to make work easier, but now the average company is buried by hundreds of apps that silo your teams, slow you down, and simply don't work together. That's not SaaS. That's SAD. Software as a disservice. That's why you need Rippling. Rippling is the unified platform for global HR, payroll, IT, and finance. They've helped millions replace their mess of cobbled together tools with one system designed to give leaders clarity, speed, and control. By uniting your employees, teams, and departments in one system, Rippling removes the bottlenecks, busywork, and silos your software created.

44:01Automated, perfectly in sync, and seriously simple to use, Rippling gives your company one source of truth for your people, their data, and everything they touch. With Rippling, you can run your entire HR, IT, and finance operations as one. Or pick and choose the products that best fill the gaps in your software stack. And right now you can get six months free when you go to rippling.com slash decoder. Learn more at r-i-p-p-l-i-n-g dot com slash decoder. That's rippling.com slash decoder for six months free. Terms and conditions apply.

44:41Refresh your good vibes mit TicTac. With the right vibes, it's...

44:48Oh man!

44:55...to...

45:01Now try! The different TikToks from mint-y-frish to fruct-y-sue. TikTok Refresh your good vibes.

45:14We're back with decision scientist Cassie Kozarkoff switching gears to her time at Google and some of her key career decisions. Back me up now because you left Google about two years ago, a little less than that. You were there for about 10. It's long before the open AI chat GPT wave of AI enthusiasm had swept across the globe. but you were working on some of this stuff. So I want to understand both the work at Google and what led you there. I think you mentioned that your dad first mentioned economics to you when you were 13. And that sounds really young, but I think you started college a couple of years later, right?

45:58So it actually wasn't, you were actually on your way to the studies at the time. What made you decide to go to college that early and what was motivating you? One of the things we don't talk about enough is that knowing what motivates someone is that tells you more about them than pretty much anything else could. Because if you're just observing the outcomes and you're having to make your own inferences about how they got there, what they did, why they did it. Particularly with survivorship bias, it might look like they're such total heroes. Then you look at their actual decision process, and that may tell you something very different.

46:41Or you may think someone's not very successful without realizing that they are optimizing for a very different thing from you. This is all a very long way of saying that I'm glad we're friends, John, because I'll go for it. But it's always just such a private question. But yeah, why did I go to college so young? Honestly, it was because I had skipped grades in elementary school. The reason I skipped grades in elementary school was I came home, I was nine or something, and informed my mother that I wanted to do this. I cannot remember why. For the life of me, I don't know. I just, I was doing something on a nine-year-old whim.

47:20And skipping grades wasn't a done thing in South Africa where I was growing up, so my parents had to really battle with the school and even the Department of Education to have it be allowed. So there I was getting to high school at 12, as one does. I actually really enjoyed being younger. Okay, you get bullied a little bit, but I enjoyed it. And I enjoyed seeing that you could learn a lot. And I wasn't intellectualizing it the way I am right now, but you can learn a lot from people who are older than you because they kind of push you. And I'm a huge believer in just the act of being surrounded by people who will push you, which is maybe my biggest argument for why college still makes sense in the AI era.

48:00Just go be in a place where everyone's on a journey of self-improvement. So I had, in learning this, ended up making friends with 12th graders when I was 13. And then at 14, they are all out already and in college. And I had spent most of my time with these older kids. And now I'm stuck and I basically want my friends back. And so that is why I went so young. It was 100 % just a teenager being driven by being a social animal and wanting to be around my peer group. But be fair to yourself too. It sounds like you just wanted to see how fast the car could go, right? That's part of what it was at nine is that you realized that you were capable of bigger challenges than the ones you had been given.

48:47So you were kind of like, well, let's see. Right. And so and then you went and you saw and you were actually able to to handle that, you know, the intellectual part. And people probably said, oh, but the social part would be hard. Hey, I got friends who are seniors. That part's working, too. Well, let's see if I can actually drive this at college speed. That was part of it. Right. I am so easy to manipulate with the words you can't do X. So easy to manipulate. I'm like, no, let me show you. I love a challenge. Let's let's get this thing done. So, yeah, I think you're right. I think you're right in your assessment.

49:17And so then you went on into graduate work, University of Chicago and then beyond. Neuroscience, some economics in there. So I actually went to Duke for neuroeconomics. That was the field, neuroeconomics. And that is, you know, you get macroeconomics and microeconomics. Well, this was like nano or pico economics, right? This is how the brain implements decision making. And so, of course, courses involve around experimental microeconomics. We're part of it. But this was from the psychology and neuroscience departments. And so it's technically a graduate degree in psychology and neuroscience with a focus on the neuroscience of decision making, which is called neuroeconomics.

50:01And then I also went to grad school twice, which is definitive proof that I am a bad decision maker. If anyone was going to think that I personally am a good one, I've just got the technique, folks. I'll advise you. But I went to grad school twice. No, I'm just kidding. It was good for me to go to grad school twice. And my second one was for mathematical statistics. My undergraduate work was econ and statistics. So then I went to MathStat, where I did a lot of what we called back then machine learning, what we would call today AI. How many PhDs were involved there? No PhDs were harmed in the making of this person.

50:36Okay. So, but study on both of those disciplines. And then what were you going to do with it? So coming back to college, where I was taking courses around decision-making, despite having the majors of economics and statistics, I got a taste for this. So I'll tell you why I was in the stats major. The stats major, because at about age eight or nine, just before this jumping of grades, I discovered the most beautiful thing in the world, which everybody knows is spreadsheets. That was, for me, the most gorgeous thing. Maybe it's the librarian urge to put, you know, order onto chaos. So I had this gemstone collection.

51:13Its entire purpose was to give me another row for my spreadsheet. That was the whole thing. I'd get an amethyst, I'd be like, oh, it is purple, and how hard is it? And it's translucent. And I still find, though I have no business doing it, that the act of data entry of the nice glass of wine is just such a soothing thing to do. So I had been playing with data. Once you start collecting it, you also find that you start manipulating it. You start to have these urges like, oh, I wonder if I could get the data of all my files on my computer all into a spreadsheet. Well, let me figure out how to do that.

51:47And then you learn a little bit of coding. And yeah, so I just got all these data skills for free. And I thought data was really pretty. So I thought stats would be my easy A. Little did I know that it's actually philosophy. And the philosophy bits are always the bits that should kick your butt or you're missing the point. But of course, the manipulating data bits, super duper easy. Statistics, as I realized as I began to soak in the philosophy, is the discipline of changing your mind under uncertainty. Economics is the discipline of scarcity and the allocation of scarce resources. And even if money is not scarce, something is scarce.

52:24You know, people are mortal, time is scarce. So how are you going to make allocations of what you might call decisions, got in there through economics, changing your mind, what is your mindset to what actions are on the table? What would it take to talk you out of it? Got in there through, what is it, statistics? Yes, I know what I'm saying. And then how does this actually work in the human animal? And how could it work better came in through the psychology and neuroscience side of things. So I was studying decision making from every perspective, and I was hoarding. So here as well, did I know what career I was going to have?

53:05I was actively discouraged from doing this. When I was at the University of Chicago, even that liberal arts place, my undergraduate college advisor said, I have no idea what job you think you're going to get with all this stuff. And I said, that's okay. I'm learning. I think this is kind of important. And I hadn't articulated back then what I'll say now, which is data's pretty, but there's no why in data. The why comes from the decision maker, right? that the purpose has to come from people. It's either your own purpose or the purpose of the people that you represent. And that why, that why gives direction to all the rest of it.

53:49So just studying data where it feels like there's a right answer because the professor set the problem up so that there's a right answer. If they had set it up differently, there could have different answers and realizing that the setup is infinite choices, that is what gives data its why and its meaning. That is the decision piece. And that's the most important thing I think any of us could spend our time on, though we all do spend our time on it and do approach it from different lenses. So then why Google? And why did you promise yourself you wouldn't work for a company for any more than 10 years.

54:28Now we're really getting into all the things. So Google is a funny one. And now I'll definitely say some things that I don't think I've said on any podcasts. But the true story of that is that I was in the MathStat PhD program. And what I didn't know was that my advisor, this was at North Carolina State, my advisor had just taken an offer at Berkeley where he could not bring any of his students along with him. Uh-oh, right? That's pretty bad. Pretty bad mid-PhD thing. Now, separate from this going on that I had no idea about, I take Halloween pretty seriously. It's my thing. At COSR, it's a holiday, so we're a holiday, so that people can Halloween properly if they want to.

55:16And I had come on Halloween morning, dressed as a punch card, as one does, with proper Fortran to print Happy Halloween, as one does. And a Googler was giving a talk. And I was sitting in that audience, the only person in costume, because everyone else is lame. Let that go on the record, my former classmates. You too should have been in costume, but we can still be friends. And so at 9am, I'm dressed like this. The Googler later is talking to the head of department. It's like, who's that grad student who was dressed as a punch card. The head of department, not having seen me, still said, oh, that's probably Cassie.

55:50Last year she was dressed as a Sigma field, just from measure theory. So I was being a huge nerd. The Googler thought, culture fit 100%. Let's get her application in. And so the application was just for a summer internship, which seemed like a harmless thing to do. Sure, let's try it. It's an adventure. It's Google. As I was signing up for it, my advisor was like, this is a very good thing for you. You shouldn't even hesitate. You shouldn't, you should, don't be asking me if I want you here doing summer research. Definitely go to Google. You can finish your PhD there. Go to Google. And the rest is history.

56:34So a much, much better option than having to restart and re-figure out with a new advisor. How did you end up becoming this translator between the data people and the decision makers? The role that I ended up getting at Google, it's formal during the internship. The formal internship name was decision support intern. And I thought to myself, we'll figure out the support and we'll figure out the intern, but decision, this is what I've been training for my whole life. The team that I was in was sort of a SWOT team for data-driven decision-making. They were very, very close to the revenue, to Google's primary revenue.

57:23So this is a no-messing-around team of statisticians, though it calls itself decision support. It was hardcore statistics flavored with data science. and it also had a very hardcore engineering group, very big group. I learned a lot in there and I also applied potentially to stay in the same group for a full-time role with the strong prompting of my PhD advisor. I thought I was going to join that group and then a completely tangential thing happened, which is that I took a weekend in New York City before going to Mountain View, which is where I picked out my apartment. I thought I was going to join this group.

58:07I was really, really excited to be surrounded by deep experts in what I cared about. These experts were actually working more on the data side of things because it's so regimented and what the decisions are and how we approach them. So regimented in that part of Google. But I took this trip to New York City and I realized, and this was one of the biggest, like gut punch moments, decision-making moments for me. I realized I'm making a terrible mistake that if I go there, I will just not enjoy my life as much as if I go to New York City, right? So that was, there was so much instinct. There was so much, oh no, I should actually really re-evaluate what I'm doing.

58:52Am I going to enjoy living in Mountain View? I was just so set on getting the offer that I hadn't done what I really should have done, which is to evaluate my priorities properly. And so New York City was a massive change for me. So the first thing I did was I called the recruiter and I said, whoa, whoa, whoa, whoa, whoa. Can I get a role in New York City instead. It doesn't matter which team, is there something we can find for me to do here? So I joined the New York office instead. Very, very different projects, very, very different group. And there I realized that not all of Google has this regimented approach to decision making.

59:32There is so much translation, even at a place like Google, that's necessary for products that are less close to the revenue stream. So then there has to be a lot more conversation about why and how do we do our resource allocation? And who's even in charge here? Things that, you know, when you're moving billions around at the click of a mouse, you tend to have those questions answered. But in these other parts of Google, so much more color in how you could approach it. And such a big chasm between the people tasked with that and any of the data or engineering or data science efforts we might have.

1:00:12And so to really try to fill that gap, try to put a bridge on it so that things could be useful, I would work way more than my formal job said that I should. Trying to build infrastructure, I built early statistical consulting because that wasn't there. Like you couldn't just go ask a statistician who'd sit down with you and talk through what your project was going to be. So I convinced people to offer their 20 % time, stats people by specialization to offer their support on projects that were not their own project, put some structure to this, made resources and courses for decision makers for how to think about dealing with data folk, really like tried to bring these two areas together.

1:00:59Eventually, it became my job. But for the longest time, it wasn't. Sometimes there were questions about what are you? Who are you? Why are you actually doing what you're doing? But just seeing that things could be more effective and kinder to the experts who are going to work on poorly specified problems if you specify the problems well first was motivating. So that's why I did it. Trying to tie this all together. It sounds like that values and goals piece in the philosophy element that you talked about in school being important, coming back into play versus just focusing on the external expectation.

1:01:37Like, of course, you're going to go to Mountain View. That's where the power is. That's where the data people go. And you're smart enough to be with the data people. So if you're going to run the car as fast as possible, you're going to go over there. But you made a different kind of decision, perhaps, than the nine-year-old Cassie made. And go, wait a minute. Actually, what's going to be best holistically for me? And how can I work within that pulling in some of this other information. Yeah, for sure. And I think that something that we can say to your 17-year-old is that it is okay. It's okay if it's difficult when you're young to take stock of what you actually are.

1:02:19You're not formed yet. And maybe it's okay to let the wind take you a little bit, particularly when you have a great dad who's gonna give you great advice. but it would be good if you can mature eventually into more of a habit of saying well I'm not the average joe so what do I actually want and working for what is thought of as I don't want to offend any internal googlers but you know they don't have a reputation for kind of being the top team so if you wanted to be number one and then number one again and number one some more times that would have been the way to do it. But again, maybe, maybe it's worth having something else that you optimize for in life.

1:03:01And I, as it turns out, I'm a theater kid, lifelong theater kid. I'm an absolute nerd of theater. I'm going to London for just a few days in two weeks, and I'm seeing like every evening, matinees. I'm just, I'm just gonna hoard as much theater as I can for the soul. and so living in New York City was going to be just a better fit not not only for theater but for so much more that that city has to offer as I'm sure you know John so having lived in both Silicon Valley and the New York area I promise you that yes the theater is far better than New York I mean I did I'd go I went to all the plays in um Silicon Valley as well and you know I did my homework.

1:03:47I knew what I was getting into or out of. But yeah, it takes practice and skill to know that some of those questions are even questions worth asking. And I've developed that practice and skill from originally knowing how to do it to help others, knowing the, having studied it formally, being book smart about it. These are the questions you ask. This is the order you ask them It's something else to turn that on yourself and ask yourself the hard questions. The book smartness isn't enough for that. That's good for all of us, whether we're running businesses or just trying to figure out life. We've all got decisions to make.

1:04:24Cassie Kosar-Koff, founder and CEO of Kosar, former chief decision scientist at Google. Thanks for joining me on this episode of Decoder. Thanks for having me, John.

1:04:37I'd like to thank Cassie for taking the time to speak with me and thank you for tuning in. I hope you enjoyed it. If you'd like to let us know what you thought about this show or what else you'd like us to cover, drop us a line. You can email the team at decoder at the verge. They really do read every email, or you can hit me up directly on X or threads. I'm at John Fort on all platforms. Decoder also has a TikTok and an Instagram. Check those out at DecoderPod. They're a lot of fun. If you like Decoder, please share it with your friends and subscribe wherever you get your podcasts. Decoder is a production of The Verge and is part of the Vox Media Podcast Network.

1:05:15Decoder is produced by Kate Cox and Nick Statt. The show is edited by Ursa Wright. The Decoder music is by Breakmaster Cylinder. See you next time.

1:05:28Thank you.

1:05:57e-commerce and more. And the best part? Odoo replaces multiple expensive platforms for a fraction of the cost. That's why over thousands of businesses have made the switch. So why not you? Try Odoo for free at odoo.com. That's O-D-O-O dot com.