GoogleX Chief Scientist On Imposter Syndrome, Career Growth, Project Taste

25:13And so I think what really helps far more than intelligence is focusing on what outcomes need to be solved for a project or for the company, what metrics matter for those outcomes, because often there are things that don't really matter that much. Like, you know, there are like requirements that are unimportant and requirements that are super important. So focusing on the most important requirements and doing that work and focusing on the most important outcomes for your division or company. And then focusing on only the most important requirements is going to get you far, much farther than the intelligence.

25:50I think you have an interesting perspective because you're in academia because you're lecturing at UCLA. but you've also had a lot of success in the industry as well. When I was in school, everything was very obviously intelligence-based. Maybe unless there's a, aside from hard work, but there's a test, you either get it right or not, aside from group projects and things like that. Obviously, coming from that place, intelligence feels like everything. and then you get to industry and I agree with you 100 % intelligence is not everything in industry. But because in college, it's very obviously meritocratic, whereas in industry, there's other things too, like do people like you or other things like that.

26:41Would you say that career growth is meritocratic in the industry? In my experience, it was. There were cases where people were being promoted because a vice president basically pushed really hard or SVP pushed really hard and said, you know, they need to be promoted, period. Otherwise, we're not going to be able to keep them. And that's never a good reason to promote somebody, right? Because you don't uphold standards that forever be able to look at. And then you end up with a bunch of people that are not great. And everybody's like, well, why shouldn't I be promoted? Because that clown is promoted, right?

27:12But by and large, I would say it was meritocratic. You know, I remember when I was on these committees, we would look at the accomplishments and the complexity of the accomplishments, the impact that they made for the company. We'd look at the communication skills. We looked at, for instance, patent portfolios. Were they helping the company with intellectual property, which was important back then? I don't know how important it is now. So it was generally pretty fair. And I saw Google too. It was very fair. There were very reasoned discussions about each person. So I think it is. I think it is.

27:45It's not just, You ever see those charts where they have like a circle and then they have like a sort of a polygon inside? Right, right. And it shows like how your intelligence is versus how your technical work is. And it had to be pretty, you know, pretty well-rounded for the senior levels or at least be really good in some areas. So then at Google, you went to Google X working in a new cybersecurity division. And then a company was spun out of that, right? Chronicle, if I recall correctly, but it's still under the alphabet umbrella of companies. That's right. Which was then reacquired by Google Cloud.

28:21Could you share a little bit more about that story? Yeah, sure. So we started as a stealth project. Nobody knew we were doing cybersecurity initially. The project name was Project Lantern, and this is inside of X. And we literally started from zero. We didn't know what we wanted to build. There were lots of debates, and we knew generally what we wanted to do. But we spent, I think, a good six months trying to just figure out what we were going to build. We then basically converged on an idea. We started hiring a bigger team, started working on building prototypes of the product out, finally got to an MVP, started working with partners, which was great.

28:56Actually seeing real customers use it was super useful. And seeing that we would actually go into customer sites before we had the product and just watch how they did their work and saw where they struggled, which was super useful. Really interesting, actually, like watching cybersecurity teams work. Some of the people who were like stoned, you know, they're clearly out of it. You know, like these are the people that are using your product, you know, for better or worse. So the product was a product that cybersecurity engineers would use. Yeah. So best way to think about, in a nutshell, about Chronicle's product, which was called Backstory was basically Basically, cybersecurity today is a big data game.

29:33Okay. And what you want to do, especially if you're trying to discover attacks in your environment or investigate attacks, which is a big part of cybersecurity, some of it's proactive, some of it's blocking the attacks before they come in. But a lot of it is they're going to get in and we have to know where they are, when they got in, what assets they accessed and so on. And so as it turns out, virtually all software and hardware that's used in corporations today generates huge amounts of logs. A firewall will have every connection, the source IP, the target IP, what protocol, web proxies will tell you what websites were visited, again, what machine visited them.

30:08You have telemetry like DHCP that tells you what machine's IP is associated with a MAC address, associated with a machine name. You have email logs. You have client logs, what software was installed, right? All that data is super valuable for identifying attacks. But there's such high volume of that data that people couldn't really process it. And so the customers that we were starting to work with would use a competing product, which I won't name, but they would literally go, they would ingest a certain fraction of that data, very little of it because it was so expensive to maintain it. And they would go for coffee for 30 minutes while waiting for a query to finish to look up just one piece of information in that data.

30:49And so we said, look, we have Google planet-sized compute and storage. how could we totally turn this around? And what we did was we built a product that would ingest all that data, petabytes of data, like literally some for some companies, a petabyte a week or a petabyte a month, a huge amount of data of every device, every connection, every file installation, every settings change, like all that kind of stuff. And then we indexed it and that's what I worked on. That was my sort of addition, like so that it would be more like at the speed of a Google search than a 30 minute, Let's go get some coffee.

31:25What would be a use case? Let's imagine that you discovered a piece of malware on a computer. You might have a hash for that malware. You might know the IP address where it came down from or was downloaded. You might have the file name. You might know the directory was installed. Our product would allow you to take any of those artifacts and plug it in. And then we would instantly tell you which devices also have that artifact on them, what related artifacts there were to that. But, you know, so you could say, oh, well, this file had a different name, even though it had the same hash, let's say.

31:58And so we'd better check for this name because this might be on some other computers so you can pivot. We could tell you how many devices were impacted in your environment, whose those devices were, when was the first infiltration, when's the last infiltration, is it still active, and do that in like two seconds. You know, those kind of use cases. Wow. As opposed to 30 minutes where, you know, hopefully it gives you an idea. You know, when it comes to Chronicle, even just doing the research, I got a little confused. Sounds like it spun out and then it came back in. What is the benefit of doing that stuff?

32:29Why not just do it, you know, within Google and that's kind of that? That's a great question. So I think X's initial goal was to create viable businesses, you know, that could spin out and, you know, be world changing. Okay. And so we were following that playbook, which was, hey, let's incubate it. Let's get really good people to work on it. We have really smart people. Let's not encumber the team as much as we might a normal Google team. Let them work fast. Let them do what they need to do. And then let's spit it out. You know, I can't really talk about why they required it, partly because I only have hypotheses and I don't know.

33:12but let's just say that it was a tight fit with Google Cloud. In other words, Google Cloud offers services to customers. This was a cloud hosted service. It used a lot of storage and a lot of compute, which by the way, Google Cloud had and Google Cloud could bill for, right? And so I think that for various reasons, which I can't talk about, after we spotted out, we were still an alphabet company like Waymo. Okay, so we were still like under the alphabet umbrella. We were the C in Alphabet for Chronicle. But I think they thought, you know, they had competing products. They wanted to integrate them.

33:47There were a bunch of reasons that they brought it back in and sort of integrated it. So we were, for a time, not part of Google. You know, for instance, Google, I don't know if you've ever heard, but Google performances are notoriously lengthy and time-consuming. And you have to write pages of stuff about yourself and what you did and PRs that you did and all the stuff, right? At Chronicle, we're like, we don't want to waste time on that. We want to build a project. So as soon as we spun out, we had one page performance reviews. And I think they were in a Google slide. It wasn't even like a big page of written stuff.

34:19So we could move more quickly, you know? And so that was great while it lasted. Why is it that you eventually left Google? That's another one which has to do in part with imposter syndrome. This is a regular thing in my career. And in part, it has to do with wanting to work at a startup as opposed to in a bigger, stodgier sort of Google environment, whereas things are more slower moving and there's more regulations and things you have to deal with. Not that we didn't have to deal with those things in Chronicle, but more so. Part of me didn't feel like, like there were interesting problems that I could have done.

34:52For instance, the storage architecture that we came up with, part of which I built, and I think my code's still in there. I feel very proud of that, you know, six, seven years later, whatever it is. Part of that was an architecture which was very expensive, and there were ways to basically move that into different file formats and get off of things like Spanner, which was, you know, very heavyweight, where I could have dived in and tried to solve those problems and they'd be nice, big, juicy problems. I didn't have the confidence in myself to do that. And I was afraid, especially when you get to senior levels, there are very high expectations for you, right?

35:26And so like, if you go off and try to do something and then people are like, well, what have you been doing the last couple of months? And it didn't land. And like, you're like, oh, I tried, you know, I didn't feel the confidence in our leadership that I could go off and take those risks and have them have my back. At Semantic, I did. Like, I knew my bosses for years. And so they just knew that if I were going to go do something, I'd either be successful or if I failed, it was for a good reason. And they'd give me the rope. Okay. But at Google, I didn't quite feel like I had that. And they offered me other roles too.

35:56They said, oh, you want to work on secure databases. There were some really interesting things there of like, how do you compute and do database queries entirely in encrypted space rather than decrypting and doing it, you know, basically taking private data and exposing it where malware or other attacks would get to it. They were an interesting thing. I really wanted to try something, you know, sort of over cybersecurity. And while I was at X, I was talking to all the Waymo guys, especially before Waymo even became Waymo. And I was learning all about self-driving cars. And so that was what I really wanted to do.

36:27And that's why I decided to leave. Yeah, before we get into the self-driving cars, I'm curious because I talked to someone who felt the high expectations of the highest levels was a little bit limiting or kind of put too much pressure on them. And they requested a demotion. Did you ever consider something like that? Not semantic. Because at semantic, I could do whatever I wanted to and it didn't really matter. Right, right. Yeah. But at Google, I had thought about those types of things. actually absolutely came to mind. The problem is that even at lower levels, there are certain expectations that I probably wouldn't have met if I want to go off for a couple months and just think about something, which is the way I've been most successful in my career.

37:14And so, you know, I got to be honest with you, I'm a loosey-goosey kind of guy. Like I can produce prototypes and optimize algorithms and do relatively interesting stuff. But when it comes to dotting every eye, crossing every T, making sure I test every edge condition in my unit test. That's not my thing. I don't enjoy that. But at Google, that's just like, that's what you do. And so for me, that wouldn't have necessarily made a difference because I would have had to do the things I didn't like to do in order to do the things that I wanted to do. Okay. Going into autonomous vehicles. So you went to Lyft.

37:47It sounds like that was mostly because of personal interest in the space. Can you talk about how you were hired and the story behind that? Sure. So at Lyft, I actually had a former student from UCLA that was working at Lyft. And he said, oh, you should come work here. It's really interesting. And I thought, well, never hire me because I actually tried to apply for Waymo when I was in Google X. And, you know, again, this is another problem with being very senior. When you're very senior and you don't have domain expertise in a new space, they're less likely to say, oh, we'll try you because you're very expensive, right?

38:20And, you know, you might not have the skills that they need and, you know, they're paying somebody that they can't use. So Waymo didn't want me instead of Google or Alphabet. So I said, well, they're probably not going to want me because I don't have any experience here. But why not? And so my former student arranged a meeting with their president. We had had coffee. And then I went through a set of interviews again, seven, eight interviews. Those did have some coding and design problems. That went fine. Fortunately, there was no dynamic programming because I can't do that. I mean, you can't like, you know, maybe certain cases I can.

38:54But, you know, I'll fail every dynamic programming interview question. And I was hired. What were the projects like or what was the thing you were most interested in working on that you did? So for me, the big project that I sort of was proud of was transforming the architecture from one which was a classic robotics architecture like you would have seen in the, you know, even at Waymo until probably the mid-little 2010s, which was largely handwritten algorithms and, you know, sort of not expert systems, but handwritten algorithms that would make decisions like, oh, it's time to do a left turn.

39:31Let's run the left-turn decision-making system and figure out, is it safe? Do we initiate? Do we not initiate? Those systems in the mid-2010s were using neural networks, but they were only using it for vision. So in other words, recognizing vehicles, pedestrians, and so on, maybe their angle and so on and their velocity and acceleration. But at Lyft, they were using that sort of earlier approach, which I'm sure came up through places like Carnegie Mellon, where a lot of it was hand coded. And to me, I looked at that, especially during my time at X, seeing neural networks and semantic even, seeing neural networks and said, there's got to be a better way.

40:13Because if you start hard coding an algorithm to figure out how to do a lane change, and then somebody swerves in front of you, now you're doing an avoidance maneuver, right? Now, avoidance maneuver, do you switch out of your lane change algorithm in order to do an avoidance algorithm? Or do you stay in lane change and handle avoidance? It just made no sense. And so, you know, the team was also, the SAP was also hand parameterized. So literally, you know, they're tweaking something. How close do we want to get to the curve? You know, okay, how close are we willing to get to pedestrians? What about bicyclists?

40:45What about, you know, okay, so if we get too close to a bicyclist today, let's tweak it and make that bigger. but wait a second now it's going to hurt our curb distance and so like there were you know 100 parameters that all had to be tweaked and it was a really difficult problem and by the way companies like tesla were doing this until recently too that that approach and waymo was doing that um for a long time is a dead end uh it's a dead end and so the approach that you know i think that probably waymo is using now i don't know but my guess is uh and i know that tesla is now using they've announced it, is an end-to-end neural network-based perception and behavior planner system and prediction.

41:26Basically, there might be multiple heads on these neural networks, and there are multiple functions that they're performing, but effectively, it's a single or small number of networks working together to figure out the movement. And so figuring out and doing a lane change is not necessarily a lane change algorithm, although there may be a little bit of that, but it's mostly about, you know, we know we need to go there. We know there's a left turn lane. Let's, you know, let's start maneuvering into the left turn lane and turning on the signal. And so my project that I was most proud of there was basically designing with the head roboticists of the team, who are old school, by the way, an architecture which would accommodate basically an end-to-end neural network-based driver, but put guardrails on it.

42:12In other words, have a safety layer that would ensure that if the network went off the rails and told it to go through a red light, we would slam the brakes. The safety layer would take precedence over the system. But the safety layer was there only for basically ensuring that collisions never happened, basically bringing it to a safe shop or basically following legal rules that the neural network may not do perfectly. Does that make sense? Right. That makes sense. And so that was, I was very proud of that project. Unfortunately, this is like a bit painful for me. I didn't get as much credit for that as I would have liked given the work I did, which is like one of the things I learned is like, you really have to toot your own horn.

42:51You have to really talk about what you're doing. Somebody else took credit for a lot of that. But the hard and really great part of that was this was no coding, really. This was all about working with roboticists who did not want to change the approach and getting them to consider a new approach, working together to define the approach together rather than telling them how it should be. Because I didn't quite know, but by the way, I thought I had a better idea, but they didn't want to hear that because I had no degree in robotics. And then eventually coming up with a product that was a collaboration where they were able to buy in and push it themselves, if that makes sense.

43:27Right, right. And that was all like, you know, influence and not, because I can influence, but not based on my skill because I don't, they didn't have any autonomous vehicle skill. I think that's a common topic that people wonder when they're doing shared projects is how do you make sure you get credit for what you worked on? And so maybe you could talk about that. I wish I had a good recipe for that. I, you know, in general, when you're, when you're more junior, I think it really makes a lot of sense in the moment when you finish a project or you're, you're almost done with it to take notes on what you did and what the big accomplishments were and what the metrics for so that when it comes time to write up a promo packet or even your sort of yearly review packet, you have all those details which you're going to forget later on.

44:12Even like two years later when you're going up for a more senior promo, right? So having that, I think, is useful. To be honest with you, I never did that. But like in retrospect, that would have been very useful for me because I would come out of it from my head and then ask people like, what was that? How much faster was that? What an hour was able to detect that we could have detected before? but I think that does help a lot. And you have to toot your own horn. Like, you know, when you're writing your performance review without embellishing, I think embellishment is really bad actually, but, you know, really state what you did and the value added and the sections that you worked on.

44:45Don't claim credit for everything. Claim credit for the parts that you worked on because I guarantee you when a committee is reviewing your packet, they're going to be like, wait, why are they taking credit for all this when we know that so-and-so did all of this great work? And then you lose credibility. So because I've been on those committees. Right. I've seen that. I've seen that. But so, you know, be very specific and granular about what you did, what the benefits were, how you collaborated. I think that helps a lot. When you're more senior, it's more difficult because it's it's a lot of soft power.

45:16It's a lot of influence. I was actually reluctant. Like I didn't even try to take credit because I didn't want to alienate my co-collaborators who also were part of this. And so I didn't go around and start talking to the president saying, we've come up with a new approach. I let them talk about it. I let their boss talk about it. And that actually hurt me because I never got, you know, when it came to review time, they said, oh, their manager initiated this project. I'm like, what? Really? This is news to me. Oh, no. So yeah. Yeah. So that was very I had PTSD from that experience, I have to say.

45:52Before we leave Lyft, I'm kind of curious because that space is super competitive. There's like, I don't know, a billion different self-driving companies, especially back then. What did it look like for Lyft to kind of win in that space? You know, it wasn't clear that we had a strategy to win in that space. We were definitely a nascent organization. I think they'd been around a couple of years when I first started versus like 10 years for Google and X working on that technology and Waymo. So I didn't go in, for instance, thinking that I would be building the next generation of self-driving car that would actually overtake a Waymo.

46:32I went in thinking this is an opportunity to learn something new and work with really smart people. And that was my outcome that I was trying to achieve. So I don't know if there was a plan necessarily. And the division was eventually sold to a subsidiary. So they got the IP. That was great for them. And, you know, at that point I said, OK, I'm retiring. You know, I'm kind of curious because how did you get into actually becoming a professor at UCLA and lecturing there? I know you get a lot of joy out of it. What's the story behind going back and lecturing? So back when I was in my late teens, early 20s, I was teaching programming in a place called Learning Tree, which you probably have never heard of.

47:17but Learning Tree is a for-profit school where they will teach you gardening, guitar, knitting. And back then they started teaching programming. And it wasn't very easy to find people who could teach programming because it was early. That was probably in 1990, 1991. So I applied because one of my friends was doing it and I really enjoyed it. And I was teaching people from DeVry. Have you ever heard of DeVry? Oh, I've heard of that one. These are those commercials, right? Well, they were trying to learn from me, a learning tree, so they could teach at DeVry back then because they didn't even have a programming class there.

47:51Right. So I was teaching people and really enjoyed it. And it felt really good to get up in front of people and explain things and try to be really clear. And so I had some experience doing that. And at UCLA, when I was an undergrad, I would teach little classes. We would find a room in the evening, and I'd just invite people who had problems with some of the material, and we'd just go over it on the whiteboard together or blackboard. So I enjoyed that kind of stuff. And when I was at Samantac, one of my colleagues was a guy who was a part-time lecturer at UCLA. We were having lunch, and I said, oh, you know, I really enjoy teaching.

48:24And he's like, oh, you should apply. And I'm like, UCLA would never hire me. I don't have a PhD. It'll never happen. And he said, you know, let me bring you an application just in case. And I said, okay, whatever you want. And a couple of days later, he brings his application. He's like, here. Filled it out and gave it back to him. Didn't hear anything for months. I don't remember if it was six months. I don't remember how long it was. And two weeks before fall quarter of 2001, so like December of 2000, I get this call, a frantic call from UCLA. Can you still teach? Because our lecturer bailed on it.

48:58And I said, of course. So I had two weeks to plan a curriculum and basically teach an undergrad course at UCLA. And, you know, it's 25 years later now. So I'm very glad that they did end up picking you because you are one of the best lecturers. I think so. I appreciate that. A lot of my peers think so as well. And that's why I kind of want to ask you, how do you make these CS lectures so engaging and interesting? Well, first of all, it's very kind of you to say that. So there are a couple of things that go through my mind. So first thing is, remember I told you I didn't think I was that smart.

49:32So I think whatever intelligence I have and whatever level that is, helps me write better lectures because I feel like unless I could understand something myself, being, I think, sort of slow, other people can't understand it too. And so I try to design lectures for what I think is one of the lower common denominators, which is myself. And I don't try to teach to the top 5 % of the class. Maybe that's a problem for some people that I try to teach for the, maybe the 30th percentile or 50th percentile. And I think like, what would, what, what would I want to know if I were being taught this for the first time?

50:07I try to have empathy for the student. I think like, where are they coming from? What have they learned about? Have they do, do they even know this concept? Should I introduce this first before I do that? So I think a lot about like, I try to put myself in their shoes and ask, what would they know? what concepts were they going to be fuzzy at versus concepts I'll pretty have pretty solid where I can just use the concept and explain it. And so that's a lot of what goes into my lectures. And so just for now, like I'm trying to improve some slides. I'm literally spending days back and forth with ChatGPT03 discussing concepts and trying to simplify it so much, but still get the essence right.

50:45And then I'll go to Gemini and verify and see, figure out where they have differences and then all, because there's a lot of materials on the internet that are actually really good, to be honest with you. And the textbooks all suck too, I have to say. We talked about outcomes earlier. And for me, an important outcome is that students not only learn something, but enjoy the process. I want them to have a good time. And so I'm always thinking when I'm making slides, can I make them funny? Can I make some joke or something silly? Can I make them like sort of colorful or, you know, add like an emoji or something to make it a little bit more fun so that they, there's just a little bit of surprise when they come to class.

51:21They never know what they're going to see. Might be a little inappropriate, you know, might be a little silly because I don't just want them to learn. I want them to learn and enjoy, want to come to class. One thing I'm also curious, because I think a lot of people are scared of public speaking, but I, you know, you're, you're very good out of it. Do you have any tips on, on speaking well and practice? Just practice a lot. The more you do, the more fluent you're going to get. I even find like when I'm not teaching, because I'm part-time teaching right now, I'm retired, I'm taking my dog for walks and working on side projects, playing with LLMs and stuff.

51:55And I find that my speaking deteriorates over time when I'm not actively using it, even over the course of a year. And that might just because I'm getting older or whatever. But in general, I found it too, like earlier in my career. So if you want to get better at presenting, if you want to get better at communicating, you have to practice a lot. And so that might mean getting a lunchroom and giving a talk about a project you're working on so that you can explain to other people what you're doing, even if you don't need to. You're not doing it because it's required to transfer over your project or to integrate some technology, just to do it.

52:26And people love that, by the way. And you'll probably screw it up the first couple of times. You'll get better and better at it. And eventually you'll find that people will listen to you and take you more seriously if you're a great presenter, a great communicator. And in fact, story really quickly back to my time at Google X. I remember I actually had a talk that I used to give at Symantec and I got permission. It was on Stuxnet, I think, and on maybe malware detection. I had another one. And I got permission from Symantec to give that talk in privately inside of X. And I remember people coming up to me afterwards and saying, wow, you know, you're one of the smartest people I've ever met.

53:02I'm like, literally, you really don't know. But people will think you're intelligent and they will give you more credit and they'll introduce you to other opportunities based on your ability to communicate because people associate that with intelligence, if that makes sense. And like I can go into endlessly about times in my career where communicating effectively helped my career. So it's super important. But yeah, practice. You know, with all the LLMs and AI coming in, are you seeing people cheat more often with LLMs? Are people learning less or more? Last year in fall, I allowed students to use LLMs, not to write the whole project, but for autocomplete or to write simple parts of the project which weren't really relevant to the material that we were covering.

53:44And I think in retrospect, that was a bad idea because I think people were autocompleting a lot more than just like a function of the uppercase of string or something, like that kind of thing. They were using it in ways that hindered learning. I've recently been reading a bunch of papers about how it helps and hinders learning. And actually, I don't know if you saw this recently, a study at MIT that like synaptic connections are down from like 70 % to 50 % or something. I forget the numbers, but like I just saw a blurb. When people use LLMs to solve a problem rather than working through it themselves.

54:18So I'm actually, this coming fall, I'm going to allow LLMs for learning, like clarifying concepts, asking for what does this mean? But I will not allow them ideally for projects because I believe that it did hurt understanding. And we saw that in exams. Like if you look at the exam understanding versus project scores, there's a big delta. Perfect project scores, but getting everything wrong. Although I have to say the projects were not solvable entirely with LMs, but you, you know, with the latest models now, you could probably get a 95 % on them with really bad code. But we were evaluating correctness, not code.

54:54How do you even tell if they're using LMs, though? um one way you can tell is that when they autocomplete often they'll autocomplete the elements will generate error checking right the error checking will typically have an exception with an error a message and the messages will be very consistent across different implementations and so in fact we have cheat checking software that checks n squared different you know everybody's project against everybody else's project and we will have flags where it's like 30 percent of this code is similar. And a lot of it is like word for word, similar error messages and similar variable names and, you know, like similar idioms.

55:33And so like pretty obvious people are using these things extensively. You know, a lot of people are worried about LLMs kind of automating software engineering and, you know, should they even get a software engineering degree anymore? You know, what do you think about that? It's a great question. And I, and I think that it depends on what these models evolve into. Imagine you could literally give a project to an LLM just like a junior engineer, and it would produce a component which is largely correct with tests, with security factored in, with proper modularity, DROI, all the other good stuff you want.

56:11If we get to that point where bigger and bigger tasks of more complexity are solved correctly with good style and no code smell and all the other good stuff. That is, let's call that like AGI programming for the time being. Contrast with what we have today, which is models that can actually solve tightly specified subproblems pretty well, build tests for them, but you still need some supervision. Did it use a good algorithm? Did it like do a deep copy when it should have done a shallow copy of a data structure, like all this stuff like this, right? I think if we stay in a world where we get really good, but not AGI good, And based on that definition I gave you earlier, I think software engineers, software engineering will still be a great field to get into because someone is going to have to go and look at that code and understand the mission of the company and understand the standards and so on, and then make sure that it's doing the right thing.

57:08And that requires real thinking and introspection and corrections. And even if you have the model fix things, you still have to know what to have to fix. And so I do think that having that degree and having the skill of being able to write code and recode is super valuable. And we can talk about if you want, like, but what about all the jobs going down right now? You know, which may be caused by the by LLMs probably in part. So that's one situation. The other situation, if you truly have an AGI where you can go and delegate something to it and it will do like a L5 job, it will do a really good job.

57:42It might need a little bit of a couple of tweaks, but they're minor tweaks. takes on projects that would take weeks, just gets them done. I think the world is a lot different. And in that world, where literally all the software of whatever complexity can be written correctly, securely, with right tasks, I think all bets are off. And I think it's a different set of skills that are necessary, personally. And I can tell you what I think those are, but I think it's different. What are those skills? Project management, soft skills, those things? So in a world where software can be written, And like arbitrarily complex software can be written, tested, you know, good style.

58:21Everything is, you know, good stuff like you'd expect of a senior engineer. To me, the world changes into a place where you're where engineers are going to be focused. And maybe they're not even engineers anymore on what we build, not how we build it. OK, so what problem are we solving? Why are we solving it? I think of in that world, the greatest engineers will be people who really understand a problem that they're trying to solve. for a customer. That might be an internal customer, might be a customer that you're like a consumer, might be a business. And you know exactly what pains they're facing and how they measure success and what gets them really pissed and what is hard for them to do now and you want to make easy.

58:59And then figuring out how to really clearly communicate to an LLM those requirements to get to do all that hard programming work that you would have had to do over weeks and months. And that is a really hard problem in its own right. So if people who have really great clarification skills, really great sort of outcome analysis skills, like what outcomes is the customer trying to achieve? What are the metrics by which the customer measures success? What's important to them? What's not important to them in those outcomes? How can we communicate to a model in a way that tells a model what we need it to do and to meet those requirements?

59:35Because models will get some of that wrong. Those are the people that are going to be successful. And in that world, I think you have big companies like Google and Meta and so on, Amazon, but you're going to have 10 ,000 smaller companies that are going to now be able to tackle problems like building software for pet sitters that never was, you know, tackled before, you know, or building software for like doggy daycare. I think about that because our dog goes to a doggy daycare and the software just sucks that they use, right? It's really bad. If you have a bunch of domain experts who can use these tools, now you have a million small businesses each solving a problem in a way that's really perfect for those customers and not having to worry about the engineering.

1:00:13Does that makes sense so it's a different world still a lot of software engineers but different skills what you said it sounds like and i don't know the product management function uh description too well but it sounds like it sounds like someone who's understanding the customer the business communicating well it's almost like the llm is like a software engineering team but it's you know a little query engine it would be like a competent product manager and i gotta say in my life in my lifetime i've met very few confident product managers but yes you could call product manager would be a good name for it yeah if program product managers were actually competent most of them are not i gotta say what what makes a competent product manager by the way you know i think a competent product manager a lot of them are good at communicating although many are not but a competent product manager in my mind is somebody who really understands again customer outcomes and customer metrics so like for instance if you've heard of jobs to be done or outcome-driven innovation.

1:01:07These are methodologies which are more deterministic and then just touchy-feely. So they actually have methodologies where they say this is how you discover a customer's outcomes. Like what jobs are they trying to get done during their day? Where are they struggling? How do they measure success? Like what metrics are important to them? Like, you know, maybe, you know, like for instance, when I'm brushing my teeth, I drool. I don't know about you. Do you drool when you brush your teeth? Sometimes, yes. Yeah, I drool a lot. Like I guess like I got a lot of saliva. Okay. Yeah. And for me, like one of the really annoying things about brushing my teeth is it like, like the drool to go all down my arm and I have to rinse my arm after I brush and it's like worn everywhere.

1:01:46And like, that's a, that's a metric by which I judge whether my toothbrush is great. Of course, I haven't found a good one. Maybe I shouldn't design a toothbrush. But basically you really need to understand the pains that customers go through and what they care about and what's really not that important. Because a lot of things that you might think are important internally, customer doesn't care about. So I think most product managers are touchy-feely. They're like, well, I think I know what the customer wants. And I saw this really cool feature in this product here. So I'm going to go and do what they did, but do it a little bit better without really understanding what the customer is struggling with.

1:02:20How often they struggle with that thing. Is it important to fix? Or maybe it's not. Like, you know, maybe it seems cool to you. And maybe they add that feature because they have some extra team members that can do it. But that's not necessarily the right thing. And so, yes, it would be a very competent product manager who really understands the customer. maybe worked in that environment, knows the problems, has suffered through the problems, and can then basically tell LLM how to solve that problem. I see. So competence, if I'm understanding correctly, is user empathy. It's knowing what actually matters for them and building for that and communicating for that and measuring that and all those things.

1:02:56But a lot of people think that's a touchy-feely skill. Like it's something you just sort of develop and you sort of have intuition about the customer. I think it is a repeatable process done through interviews, done through observing the customer. It's not something that you just sort of get better at by feeling it. It can be repeated, is what I'm saying. And very few product managers will do that. I see. Most of them just say, oh, I know the product. I've been working on it for five years. I know the customer. I talked to Joe last week at, you know, customer name. And, you know, this is what they really need.

1:03:23You really know that? Like, you think you know that, but what are they trying to get accomplished? Right, right. Park managers typically think in terms of features, not in terms of customer pain points and what they're trying to get done. I see. In my experience. Right, right. I'm sure there are instructions. Yeah, yeah, yeah. I'm sure there are. I just never meant it. Yeah. Okay. Coming to the end of the interview, I always love reflecting back on the careers. I think you've had a full career at this point, more retired at this point. So I'm curious, like looking back on your career, is there anything that you you regret or something that you wish you would have changed that maybe others can learn from?

1:04:01Yeah, I mean, a bunch of things I would say, like, first of all, I should have left my job at Symantec earlier. I feel like you should stay in a job as long as you are learning new things and building new skills. And as long as you feel empowered to grow and try things that might be uncomfortable for you and not have to worry about your, you know, what if I fail occasionally? Like have the space to try things and get them wrong, but to be, to be able to create great things. And I had that during a lot of my time at Symantec, but there was a point where I just wasn't growing anymore and I was stagnant and I didn't want to leave, not because, you know, it was interesting, but because I just didn't have the confidence to go somewhere else.

1:04:39And so I would say there's a lot of value in staying in a job as long as you're growing. And that means maybe switching teams and staying in the same company. There's a lot of value of having that institutional knowledge of a platform that you're working on or set of systems of your reputation. Reputation goes a long way. If you were really good in one in one area, you switch to another team. You can use that reputation to help you in the other area often. And so I think staying in a job for five years, even 10 years can be as long as you're learning can be really great. I don't advise people to switch every couple of years necessarily.

1:05:11On the other hand, you know, staying to staying too long, you know, you can get stale. It gets easy to just say, you know, I'm comfortable. I'm not really having to be challenged. I can do my job. I can wake up and have some nice coffee in the morning and I do whatever I do. And it doesn't really, you know, when you get more senior, often you can just do that. And so when you get to that point, it's time to leave and, and challenge yourself some more. And the problem with that is when you do leave, you do start over. Like I found that I had a pretty great career at Semantic. If you asked anybody at Semantic from the 21 years I was there, who I was, people would know me.

1:05:45They'd say hi to me in the hallway. I even had people come up to me on the street because I would be on TV talking about Stuxnet and stuff. Like, you know, I was on Fox and MSNBC and Wall Street Journal, New York Times and all that stuff. And it was really exciting. But then you go to Google and they're like, what have you done for us? We don't care that you did those things. It only matters what you've done for us. And so that requires a lot of rebuilding up trust and building up, you know, sort of a reputation and doing a lot of good work. And, you know, that's stressful and it takes a lot more work than staying where you are.

1:06:18So if you do that, you got to choose wisely. I hear I'll probably misattribute who said it, but there's some quote about you should either be learning or you should be earning in your career. What are your thoughts on your golden handcuffs somewhere? You're no longer learning, but you're earning a ton. Do you would you still advise that that person leaves? I would never say leave if there's a good package and we have to optimize for multiple things in our life. Learning is definitely important, but also being financially stable and different people need different amounts of money to feel comfortable about being safe in their life and having enough to take care of their family and themselves.

1:06:57I think there's a place for both and it can be okay to be comfortable and making a lot of money for a while. But I think that if you're solving for enjoyment and fun and hard problem solving, which a lot of people are, that can be toxic over too much time. Definitely. It sounds like early in your career, you were lucky to find what you enjoyed. A lot of those early problems were super interesting. Do you have any advice for people who want to find what they enjoy in software engineering? You know, for people who are in college now, graduating soon, I would say, like, try lots of internships if you can.

1:07:38And that might be difficult now, given the way things are going with jobs right now. Everything's cyclical, but right now jobs are tough. I found cybersecurity entirely randomly. You know, it wasn't something I was like, I want to be in cybersecurity. It's like I had an internship. My first year was doing file management. My second year was doing blah, blah, blah. Third year, oh, viruses. is the more you can explore, the more you're going to potentially discover a passion. And like, I think there are many jobs where you would think it's going to be totally boring. But if you go and start working on the problem, you're going to realize there's really interesting problems to solve.

1:08:10And you might find a passion for a field that you would never expect that you would enjoy. And then you're like, wow, this is really interesting. There's really fun problems like the people I'm working with. Customers are interesting or maybe they're a pain in the ass, That's interesting to deal with. Like, I would say just try things. Don't try to wait and find the perfect job. Find a job where you have a good manager. There's some hard problems to solve. And you don't know anything about it and you'll learn. Like that's, you might find your dream job and that must be 25 years. So before you know what you enjoy, you're saying, you know, search with breath.

1:08:47And then once you find it, then you can kind of go deeper. That's right. I mean, look, you might get lucky if your first year internship is really interesting and you're doing well. I would stick with that. You could try breath there, but I would say like a greedy breath. The last question I have for you is if you were going to go to yourself right when you were graduating from UCLA and give yourself some advice, knowing what you know now, what would you say? It wouldn't be one thing. I mean, the biggest thing would be don't let fear of failure hold you back. You probably can do more than you think you can do.

1:09:21I might've had a richer career had I listened to that. I would say focus on the outcome. So whenever you're working on a project, think about who's going to use it. What do they care about? How do they measure success? And then just optimize for that. And don't try to make the perfect thing or don't try to make a very round thing if all you need is a little square piece here. Like often problems can be solved without having to be perfect and still be really good. So like focusing on people's outcomes. Big one is when you're when you're presenting, for example, like this had to learn. I used to go into presentations and talk about the technology to senior leaders.

1:09:58And like in retrospect, those senior leaders don't care how the algorithm works. They don't care. They just want to know, is it going to be faster? How much faster is it going to generate more revenue? How much revenue is it going to? you know, fix a problem that currently takes 20 people to do it, make it 10 people to do it. And so you really need to get in people's heads and think about what they're solving for. Again, this whole idea of outcomes and then speak to their needs, not your own. So that's a big one. I said, find a good manager. I'll just say that again. Manager can make or break your career and your life and your like happiness.

1:10:32So, you know, a good manager that trusts you and gives you some rope is super valuable. Learn how to collaborate with people. Just like, don't assume that you're right and just tell people they're wrong. You have to really learn to work with people and understand their point of view. I still struggle with that, but it's a work in progress. Those would be big things. Yeah. Yeah. Well, thanks so much, Kerry, for your time. I really appreciate it. I was really looking forward to it. I think there's a lot of stuff in here that people are going to benefit from. So thanks so much for your time. That was my pleasure.