1011: The Math Still Matters: Deep Skills in the Age of AI, with Dr. Catherine Williams

23:38And maybe it starts to approximate what we have, but I do think, you know, there's, there's many more layers to go before we're done.

23:45Jon Krohn:For sure. For sure. And that's, uh, these places, math and programming have been the easiest places for machines to have superhuman capabilities because it is, those are the easiest areas to create training data because you can, if the solution works out, it's a pretty good indicator that the algorithm solved it correctly. And so you can create lots of training data there that is, you know, in the real world problems don't have that same kind of, you can't simulate data as reliably because you can't be confident that an answer makes sense just because you generated it. So, yes, yes, yes. So the world models thing, as you say, yeah, Yann LeCun, Fei-Fei Li, and others trying to, you know, they're raising billions of dollars in seed rounds to be able to create these huge data sets.

24:40Jon Krohn:And it'll definitely make a difference. But I still think, you know, the way that information flows through machines has differences to the way that information flows through our own brands. and yeah, hopefully we'll continue to provide some value to machines for a while. Well, it is interesting, right? So raising all this money for world models and so forth seems to point to the future being just like the machines getting smarter and smarter and smarter where actually I think the really interesting frontier is machines into humans, right? Like that interface between their thinking and our thinking, like the way I like to use LLMs is use them to help me improve my own mental models.

25:26And then I prompt back, you know, I don't know. It's not a very well articulated thought, but I do think that there's a lot there that I haven't heard well explored there other than people sharing their prompts and, you know, comparing notes.

25:38Jon Krohn:Oh no, I know what you mean. And I think that this kind of thing, I don't think we'll be able to get billion dollar seed round funding for the kind of innovation that you're describing because it's the kind of, because it's, I don't know, there isn't the same kind of obvious enterprise use case. You know, it's... Right. Like there theoretically is, you know, you could have a learning and development department be investing in these kinds of educational tools that allow us to better understand calculus and algebra and engineering and tie all those things together to be better ML engineers, for example.

26:13Jon Krohn:But it doesn't seem to me like the TAM, the total addressable market there, is nearly as big as there is for like developing world models and kind of this idea of everything being fully automated. But what you're describing to me as somebody who likes learning stuff does sound way more interesting. And yeah, I mean, there's lots that we can do today, but having tools that make it, we're obviously at a time where it has never been easier to learn. and some people are taking advantage of that, but it seems like it's probably a minority of people who have access to these tools and probably some multiple.

26:58Jon Krohn:So of the people who are using large language models to understand things more deeply and say, be able to do better on a college paper or on some work project, there's probably a multiple of those people who are using them just to spit out an answer and circumnavigate learning. Exactly. You don't have to build your own mental model because you can just, you know, copy paste something instead and get by. Yeah. Yeah. Which is an approach. I mean, but yeah. Yeah, it's an approach. There was, I was, so there's an economist article that came out this week at the time of you and me recording that is about how at UC Berkeley, which is a tough school to get into.

27:48Jon Krohn:It's a prestigious university. One in eight people, first year undergrads, have math, and this is, it was in math, I think it was in a program where math matters. Like, you know, math was a prerequisite for getting into this program. One in eight people that get admitted to Berkeley for that technical degree program have math abilities below high school level. Wait, what is happening? I don't get it. Yeah, exactly. And so it seems like it's hard to know what the causal factors are, but it seems like it's probably a blend of LLMs. And also, I think the pandemic had a big impact on learning. That tracks, but wow.

28:33Jon Krohn:Yeah. Anyway, all right. Well, fascinating conversation there about your background, Catherine. Let's now dig into the journey since. So from pure mathematics research, you went to AdTech. So companies like AppNexus and Xander, which were doing revolutionary things in AdTech at the time. And then you went to Enterprise SaaS at Qualtrics, another really well-known name. And now you're in the nonprofit sector at Candid. Each of those transitions represents a significant shift in industry, culture, and purpose. And something else that happened over that time is data science changed a lot. So you started as a professional kind of data scientist in 2012 at AppNexus.

29:28Jon Krohn:And so you started as a quantitative analyst at AppNexus, it looks like from your LinkedIn profile, and then manager of quantitative analytics, head of data science after that, and chief data scientist. and by the time you left AppNexus after almost seven years, you were chief data and marketplace officer. And so you have about as much experience as a professional data scientist as anybody could because I believe it was around 2012 when you had your first kind of data science role that that term even came into being. And I'd love to hear kind of simultaneously, if this isn't too crazy a question, how your career journey personally evolved while our industry, data science, was kind of born and evolved to where it is today.

30:19Yeah, it actually, they are very, very tightly intertwined. And I think I wouldn't have been able to have the career trajectory that I had if either one had been offset by even a couple of years. So remember, I said I was a pen and paper mathematician. So I taught myself how to code. I knew a little bit of data. I learned some SQL online. I wasn't a computer scientist. I was hired at AppNexus because they wanted people who could do probability math. Because at that time, all the data that was backing up the real-time bidding systems was stored in aggregate form. And so they were using complicated sort of Bayesian predictions in order to build out the algorithms to be used for the bidding.

31:03And so that mathematical fluency, the fact that I had a PhD in math, even though I hadn't done anything around probability, I could pick it up because like I said, I'm mathematically fluent. So they hired me despite a complete lack of data skills, but because I was mathematically fluent. But within a couple of years of being there, the technology was changing really rapidly. So Hadoop came out and all of a sudden our systems, we were able to start getting access to log level data. So now we're talking big data, not aggregated data, raw data. And that requires a different set of tools and it opens up a totally different set of possibilities.

31:37And so I was at that time managing then a team that had been quantitative analytics, but increasingly it became clear that there were more techniques. And so the team and I learned together. We sort of bootstrapped ourselves up. We read papers and went to conferences and learned, you know, what one can do with data and started building out different kinds of systems that weren't Bayesian. They were more sort of what we consider as traditional ML, logistic regression models, et cetera, decision trees, not decision trees, random forests, that kind of thing based on the different flavor of data. And that then continued.

32:18My own career then kind of diverged a little bit from data science. And I took on additional responsibilities within AppNexus that were still sort of mathematically flavored. So I was looking at marketplace dynamics and got deep into auction theory, which is actually super interesting. so not as much a traditional data science path but that's what then took me to be chief data marketplace officer i was marketplace czar for a while at app nexus i mean with yeah

32:44Jon Krohn:so it was around the same time i mean so it was i also had my first data science job in 2014 so a couple years after you and relatively early on and and you kind of you talked about there being a traditional path and i don't know what that would have meant at the time you know we were all kind of figuring it out together. I remember, you know, in that first data science job, this Dr. Amit Bhattacharya, he was, he was like the existing data scientist at Omnicom. And he was like showing me Jupyter notebooks. And I was getting into Python with him because I had previously been programming in R and MATLAB.

33:20Jon Krohn:And yeah, it's funny. Pandas took the world by storm. Wes McKinney, like worked out of the AppNexus office. We were like early, you know, beta testers of pandas like oh really yeah oh that's super cool uh and if if people don't already know who west mckinney is so west mckinney created pandas which i presume most listeners know is the standard data frame library for python and uh if you want to hear him on the show he was in episode number 523 but him being on this podcast for one hour is not nearly as cool as having him work in your office that's early days amazing yeah yeah it's funny you're right it was the wild west at the time.

33:58So it is kind of funny that I referred to it as the traditional path. I think it's only in hindsight, you know, Kaggle came along around that time. There started being sort of in hindsight, the things we were doing became kind of canonical or more canonical or more mainstream. I don't know. So maybe that's why I'm thinking of it that way. But at the time it was anything but well-trodden or well-understood. So, yeah. Yeah.

34:22Jon Krohn:There was a relatively short period, But in the beginning, you know, kind of that 2012 to 2015, maybe 16 area where there was an expectation for a lot of data science roles that industry was offering at the time that you have a PhD. I heard a lot of people with PhDs. Yeah. Yeah. And it's kind of unimaginable at this time. Like the industry has become so big that you couldn't possibly be only like, where would you get the people if you were only willing to take people with PhD backgrounds? So you did have that. And it was, you know, it was kind of, they were looking, it seemed like one of the primary things people were looking for in early data science roles was a PhD in a quantitative discipline, which, you know, both of those boxes tick for you.

35:06Jon Krohn:And, you know, it was kind of like, it's that idea of, I can't remember who said it. I've actually had them on the show, whoever coined this term, but I can't remember who it is right now. You might even remember, because I think they were also kind of in that New York, West McKinney group of people. they coined the term that a data scientist is like a math is a mathematician who's bad at programming and a or a programmer who's bad at math it's like kind of both together it's like this venn diagram it's like you yeah a data scientist is someone who's neither good at math or programming was the idea or it's kind of the joke well it certainly is neither discipline Yeah, it's definitely a third thing.

35:49Yeah.

35:50Jon Krohn:And the idea that we're, you know, kind of borrowing a bit from both and trying to make it work in practice. I think the hiring PhDs, I mean, I hired people with PhDs not because their research was relevant, but because I wanted to make sure that I was hiring or it was often a good credential to signal that somebody could go really deep on hard problems and wasn't scared of not knowing a well-trodden path because it was the Wild West at the time. Because I needed you to go figure out what needed figuring out, you know, even if it hadn't been done before. I think now things are better understood and that kind of frontier mindedness is not maybe as necessary.

36:34I don't know.

36:34Jon Krohn:Yeah, it wouldn't have been uncommon to have a daily standup or even kind of the expectation that as a data science manager, you probably frequently gave tasks to data scientists or related occupations on your team where your expectation is you're going to hear from them in a week or something on. Yeah, like, hey, we have this big open problem. Go see what we could possibly do. And something that you and I over Zoom last week, when we were chatting about what we could cover in this episode, you and I both lamented how we miss the kind of the in-person whiteboarding on problems, which, which for me, I don't, I mean, there's actually no reason why we couldn't still have that today, but it seems like you and I both as individuals, just in kind of the professional choices we've made, like in the, Up until the pandemic, I spent more time at a whiteboard working through problems with data scientists on my team than writing code.

37:31Jon Krohn:And I loved doing that because it's this amazing experience to get this kind of mind meld with a bunch of different people. And when we would do this at that time, the startup that I was at, I was a chief data scientist there at a startup called Untappd. And we were at a WeWork in Midtown Manhattan. And whenever somebody on the data science team got into a problem where they hadn't been making progress, and you could kind of tell that you could kind of intuit that you should be able to make some progress here. And then so I would reserve a meeting room that had no TV and no one could bring their laptops.

38:10Jon Krohn:You would just bring a notepad, a pen, and the person who's running into the problem, I'd be like, all right, get up at the whiteboard and just start drawing pictures and bulleting what you're trying to do, get it into all of our RAM in our brains. And 100 % of the time that I did that, we came out of that meeting, you know, it might be an hour or two. And we come out of it with a clear direction of okay, like these are the experiments that you need to run, or this is what you need to look into. And it would always unblock the problem. And I miss I love, love, loved being like having smart, funny people and having all of our brains intermingled like that.

38:50Yeah. That kind of intense in-person collaborative problem solving. It's just, you know, it's one of a kind. I also have fond memories, maybe not quite as structured as you just described, but the AppNexus offices were on 23rd street in Manhattan above the Home Depot and I had these huge windows and you could peer down and watch people shopping at the Home Depot below. But yeah, I have various memories of being different conference rooms and everybody kind of like looking at, you know, looking at a whiteboard and thinking hard or, you know, I don't know, really fun. I still have pictures on my camera roll from some of those whiteboards, you know, cause you'd take a picture to immortalize it.

39:27I'm like, Oh yeah. What were we thinking about?

39:29Jon Krohn:Exactly. Yeah. But, um, anyway, yeah, it's okay. I think I've completely, uh, taken your, your conversation off the tracks where you were kind of going through early stages, AppNexus, growth there, and then how data science has evolved in the past decade to where it is today, but how that lines up with your career. Yeah. So my career journey, starting from math, moving into data science, then went kind of, as my teenager says, wobbly wiggly. And the latter chapter of my time at AppNexus and then Xander, data science provided the foundation, but wasn't as central. It was more centered around marketplaces.

40:09But because I had been leading the data science organization that sat in the middle of AppNexus and saw both all of the technology and algorithms supporting the buy side, buyers of slots on advertising, and the sell side, publishers selling slots, I was able to see how both sides were shooting each other in the foot. And so that positioned me then to be, I was made marketplace czar for a while at AppNexus to try to think about how we then optimized the whole marketplace, both sides, rather than each side thinking of it as zero sum. And so in that marketplace role, again, not super data science, but we did do a number of things like rethink how we forecasted revenue, for instance, rather than separately forecasting revenue from our buy side clients and our sell side clients, recognizing that they're deeply coupled.

40:57And in fact, we built out a revenue forecasting model that included some machine learning models to forecast like what the transactions like were likely to be at different price points and so forth based on the inventory and so forth. So there was some data science that kind of worked its way into that world, trying to make sure that we were running a robust and healthy marketplace. So that took me through the end of my time at AppNexus. We were purchased by AT &T. It became Xander. I did some more marketplace thinking there, but led their data science organization again. And when I left, I left all the marketplace stuff behind and went to Qualtrics to, I walked into Qualtrics to own IQ, which was their sort of branding term for all things intelligent.

41:44And so I owned sort of a hodgepodge of different things, including text IQ, which involves some natural language processing and stats IQ, which of course is what it sounds like. And I think there were some other IQs that were kind of part of that portfolio and took all of that and wound up building out a machine learning function and platform and team right in time for the BERT paper and for chat GPT to arrive and actually make that a lot more relevant and immediate. So that was the progression sort of in and out of the data science and machine learning flavored things.

42:21Jon Krohn:Back to the magic of embeddings, despite me saying earlier in this episode that that kind of embedding based knowledge that llms have it means that machines are quote-unquote thinking differently than humans are but it is still nevertheless pretty magical pretty wild i mean it isn't literally magic but being able to experiment with burt you know early encoding large language models and being able to see yeah like it was able to do things, something that up until the Burt moment, I used to always try to like, uh, like you described earlier in this episode, you know, you'd like to be able to go all the way down and kind of understand the math and be able to really understand everything since Burt on, I don't really understand.

43:12Jon Krohn:I don't really... I can't... Basically, up until BERT, with any of the kinds of techniques we used in data science, statistical or machine learning approaches, programming approaches, data structures and algorithms, ideas, all of those things, I could create a Jupyter Notebook and write some Python code and then mess around with some variables and see how that changes things. And basically, since BERT onwards, I have not had that. I don't have the same kind of level of intuition of exactly how these wild capabilities emerge from scaling up transformers, you know, to a very large size like I did.

43:59Jon Krohn:It is wild to me that we get the emergent properties that we have from scaling. I 100 % agree. And I share that sentiment. Now, some of it is that it's not made available to us, right? It's proprietary. And so we don't have access to the information that probably the researchers inside of those companies do that maybe they're able to, surely some of them are better able to wrap their heads around what's happening. But yeah, I agree with you. I can understand the BERT paper, but then when I see modern LLMs and I think, is that just a bigger context window? How does the architecture do that? I don't know.

44:32So it's it's wild. And so the top down thinking and figuring out enough of a mental model to be able to reason on is like the one, the one thing one can do, you know, but the embedding stuff, which you wrote a book about, and I really appreciated your book as a matter of fact, is, is really wild. When I started at Qualtrics, a lot of the text IQ machinery was still syntactically based, you know, keyword based and rules based. And some of it was really good, but that just all gets blown out of the water once the embeddings are good enough.

45:04Jon Krohn:Exactly. I was there to watch that revolution. And so for a period of a few years, data scientists had, and all the related occupations, so I'm including in data science. And by the way, in one of your recent responses, you used not super data science in a sentence. And I think that that is the first time in over a thousand episodes of this show that someone has said super data science on the super data science podcast and not mean it anything to do with our name, but just be saying, you know, that technique is not super data science. Yeah, first organic use. Yeah, exactly. Yeah, you talking about, Bert reminds me how there was this rich period of a few years where data scientists could be overhauling processes that did not have embeddings in them and find a way to have a BERT model and even very small, you know, can easily run on a single GPU in real time, you know, small encoding models, but being able to unleash magical, not literally magical, but magical feeling capabilities within some kind of workflow or some kind of platform where previously it was just unimaginable and you had these kinds of hard-coded rules, like you were saying, like, oh yeah, you know, you had these, all these if-else statements or, you know, clever code written to be able to handle some situation pretty well, but then you're like, wow, if we have an embedding model in here instead, we can do so much more and it's so much more reliable.

46:46Jon Krohn:And that was a fun time. And that kind of went into, that was associated with maybe a slightly longer period where downloading open source model weights, fine tuning those to specific tasks, and being able to outcompete GPT-3 or GPT-4 on some capability was a fun thing that we could do as data scientists for a while. But now how often, I don't know, listeners, if you have the opportunity still to be training LLMs and having those be outperforming the capabilities of what you're getting from an off-the-shelf open source model or a proprietary API, wow, cherish that. Because I don't think many of us are doing that anymore.

47:26Jon Krohn:Your thoughts, Dr. Williams? No, that's exactly right. That is what we did a lot of at Qualtrics is fine tuning because the big problems were categorizing open text comments, like customer experience type comments, sentiment, topic analysis, et cetera, and fine tuning to get the exact, you know, ontology, right. If it was a topic-based model or something like that, that was a big deal. And now I'm sure zero shot, I mean, I'm not there anymore, obviously, but I'm sure zero shot capabilities just wipe most of that away. Now, there still is a cost element. I do think that the bill hasn't come due for modern LLMs fully.

48:06And so I think that there will be another wave of creativity, I think, when true costs truly come to light and companies are really bearing them. So who knows? Stay tuned. Maybe there is some fine tuning to be done down the road.

48:18Jon Krohn:So I would love for you to be able to explain to me why my thinking is wrong on this. But I don't think that that is going to become a problem in the future. So people make that argument. So yes, VCs are subsidizing our proprietary LLM API calls today for sure. And so people will often, the most common parallel I hear is how cheap was an Uber when Uber first came to New York and they were hemorrhaging money. And now what are you paying for Uber and how profitable is that business? But the economics of LLMs is obviously very different from when you're having to, at least for now, have Uber drivers that need to eat and expensive machinery.

48:59Jon Krohn:I mean, obviously there is still expensive machinery with GPUs, but if you take, so like say today, the state of the art, at the time of us recording, the state of the art AI capabilities probably Mythos 5, Fable 5 model. Who knows what it'll be when this episode comes out in a few weeks. or hopefully we have access to those models again by the time this episode comes out. But if you say, so the trend has been now for several years that if you fix some level of capability, so today we say, okay, what does Fable 5 cost me today? The trend has been that two years later, that same Fable 5 capability will cost 1 % of what it costs today.

49:44Jon Krohn:and so even if they're so and this is you know due to like distillation techniques and other tricks engineering tricks that people come up with um you know in the deep seek popularized a lot of last year like there's always tricks coming up that allow us to be way more efficient with compute and it seems like that drives this underlying and like that's a huge multiple two orders of magnitude cheaper two years later and there's a lot that we could do with you know fable five capability allows me to do tons, to automate tons of workflows. And if I can have that same capability at 1 % of the cost two years from now, I don't think it matters that much if VC subsidies are no longer happening.

50:24Jon Krohn:I don't know, but I'd love to hear. No, I actually agree with you entirely. And I guess, so fine tuning models, yeah, probably has gone the way of the dinosaur for most use cases, but it's exactly that pressure to figure out the distillation techniques, to find the cost optimized ways of running some of the current frontier things that that's the creativity that I actually mean is like there continues to be this pressure to figure out the the cost optimized the the cheap the sustainable way of of using this technology I think so I guess that's still some of that's happening inside of the big AI shops and some of it's probably happening outside like you've got the deep seeks of the world and others outside too yeah it's the open source always nipping at their heels which I think also will provide a lot of price pressure because you're like, okay, well, you know, this midsize Quen model does everything that I need.

51:17So I'm just going to use that.

51:20Jon Krohn:So we should get to what you're doing now at Candid. Sure. So tell us about Candid and what they're up to and how you're able to, yeah, how you're able to improve things, automate things as the chief data officer there. Sure. So first, let me tell you a little bit about Candid. It is a nonprofit that provides data to other nonprofits, essentially. So you can think of it as a meta or an infrastructure kind of a nonprofit. And so in particular, it provides comprehensive data about the social sector. So like information about nonprofits, foundations, and grants. And so it makes it easier for nonprofits and funders, largely foundations, to connect.

52:06The organization was formed about six years ago when two pre-existing longtime organizations merged. Those were called Foundation Center and GuideStar, one of which collected comprehensive information about nonprofits. The other one collected comprehensive information about foundations. The two married. And now we have sort of the world's biggest, richest collection of information about each other. So my job as CDO is I look after lots of different aspects of data within the organization. This is the first role I've had, data role I've had, where the data that we work with wasn't instrumented by us, right?

52:43Like it's an external thing that we get largely from the IRS and some other governmental organizations. Some of it we collect directly from nonprofits who fill out profiles and tell us information about what they're up to. but since we don't instrument it, the quality and cleaning is a much bigger issue because when you instrument your own data, you go correct the mechanisms to make sure that the data that you get downstream meets your needs and that you're able to analyze it and so forth. But when what's upstream is an IRS form and you don't know what people were thinking when they filled these things out or they fill it out wrong or whatever or make errors, then that just creates a whole bunch of additional challenges.

53:22So what I focus on at Candid is making sure that the foundation works, that when we pull in the data, then we're cleaning it and augmenting it, enhancing and sort of extracting meaning from it that then can be interpretable to our clients. That all happens smoothly and efficiently and so forth.

53:44Jon Krohn:Sounds fascinating. And I guess it must be nice to be working somewhere that's doing like so explicitly doing social good. I think a lot of people, you know, we've had organizations going back a decade in New York, we had, you know, these like data for good organizations. And I think there's a drive from a lot of people in our space to be working at an organization that is so explicitly doing something good as opposed to, you know, there is value in having a more efficient ad marketplace, for example. Is it an arbitrary example of a thing that someone might do. For example, yeah. But there must be something really nice about doing something so socially good.

54:27Jon Krohn:So obviously socially good. Yeah. Now, on the one hand, one can argue that at least at AppNexus, right, part of the mission was to support independent journalism. And right, like independent journalism, ad supported would be good for democracy, et cetera. So I was able to sleep at night, right? And similarly at Qualtrics, helping organizations have better customer experience and employee experience helps humans all, you know, so it wasn't like I felt like I was dirty necessarily, but there is then still a distinct difference between working for an organization that is explicitly mission aligned, not trying to enrich any shareholders in any fashion.

55:04And really it's, it's goal is to just try to help end nonprofits, help their communities, right? Like get funding to those nonprofits so that they can do the good work that we need them to do. Because actually, nonprofits, I think many people are unaware how much nonprofits do in society. They're largely invisible. They don't have big lobbying arms who talk about all their accomplishments. They're embedded all over the place, doing amazing work from arts and human services and research and all kinds of things. So So helping that sector run more smoothly does feel like it's mission aligned. Now, that said, Candid's role in it is a little bit indirect.

55:46Like, so in some ways, we're supporting a marketplace with information and helping, trying to help that marketplace be more efficient and equitable. And that's great. But it's the upside and the downside of being in a systems level role of, on the one hand, you're helping a whole sector. And on the other hand, it feels sometimes like, oh, are we helping anybody? I want to, I want to move the needle more. So, you know, it's that tension. I see. Yeah.

56:10Jon Krohn:It is a little bit abstract. So yeah. So even though it's explicitly this mission and you know, yeah, you can't, you can't, can't pin down like, oh, who is the orphan who's being saved from a fire today because of my tool? Exactly. Like this nonprofit was able to use our tools to figure out how to get a grant and they got that grant and then they were able to do their work. But that's a lot of hops of causality where I'm like, well, but it adds up. I will say the community of people, both inside the organization and in the broader sector, that sort of broader sense of mission alignment is just really lovely.

56:41I hadn't worked in a nonprofit space before I'd volunteered, you know, but like, it's, it's nice. It's different than Silicon Valley. So.

56:48Jon Krohn:But you have some really cool things going on there as well. In addition to, you know, some feel good factor, if even abstract, you do also have really cool things happening from a technological perspective. So for example, Candid recently announced an Anthropic partnership and the Candid MCP connector model context protocol offers real-time verified data directly inside AI tools like Claude, which is a pretty cool thing to be doing. You know, all this cutting edge stuff. Yeah. So it's really exciting to be able to be pushing out some of these modern capabilities. The history of Candid back in the old days, Foundation Center, one of our precursor organizations published books, like literally books with the information.

57:32And then they came to the digital age and they published CDs and then everything went to the cloud. And we have SaaS subscriptions for our customers where they can log into our system and poke around and search directly in our software. But now as people are moving to getting their information from Claude or from ChatGPT, we're trying to move with the times too and have our information be where they are and where they're finding, where they're looking for information. So the stars aligned there. We were experimenting with some of the modern MCP technology and got a call from Anthropic and it seemed like a perfect opportunity to try it out and be part of that leading edge.

58:08Jon Krohn:Love that. As we start to wrap up your episode here, I'd like to take advantage of the wealth of wisdom that you've developed in 12 years of data and data science leadership. Again, about as much as anybody could possibly have in the world, in our field. And so I'd love your thoughts on, with the amount of success you've had, things like an acquisition by AT &T, when you're a C-level executive, and lots of C-level VP roles, how can people, how can our listeners, maybe who aren't yet leaders or like to be bigger, better leaders than ever before, do you happen to have any guidance for them? The one piece of guidance I would give, which also mirrors sort of my own experience, is if you want to lead, you need to think about the environment the next level up from you, right?

59:04What are the constraints and the problems being faced? You know, if you're a director, what are the things happening at the VP level? What are the problems and the concerns and the constraints and people not sleeping at night because of it? and think about the solutions and what you can provide in that context. If you're an IC, think about the problems that your manager is facing, et cetera. I think that is what got me invited into rooms. That context helped me put forth ideas that were helpful in those rooms. And that's what ultimately sort of propelled my career there. It's not always the most relaxing path.

59:40Like sometimes it's nicer to just think about the piece of code in front of you and doing a good job on that. maybe you don't want to think about the problems one level up. But if you do want to lead, like that's the way to start thinking about it, I think.

59:51Jon Krohn:That is a great answer. Thank you. And then we talked about the transformation of data science over the course of your career as a data science leader. What do you think is going to happen next? And how can all of us be setting ourselves up for success in the future, you know, this LLM enabled future, especially where so many of the technical skills can be handled autonomously by machines. Yeah. I really do not have a crystal ball on, like we're in a complex dynamical system, right? With a bunch of externalities and I do not know, I can't magically do it, but, but I do have a, but I think the thing that I think will hold true for at least some amount of time is the idea of investing in your own, your own models, right?

1:00:43So when you're using LLMs and you're outsourcing and you're building agents, that's all great, but make sure that you are also retaining the information about what's happening, right? Don't outsource, grow along with, I think that's gotta be critical, right? Because then that's going to give the fluency to build the next thing and understand the next thing and the next level of complexity. So I guess that would be advice I would give for navigating the tidal wave.

1:01:07Jon Krohn:And so when you say maintain your own models, you mean mental models? I mean, mental models. Yes. Not your own LLM. You need to have your own open source LLM learning everything that you're learning. Go get a data center. And then, yeah. Exactly. That's a good idea too. That's a pretty safe bet here is have a data center. That's a good way. Yeah. Have a cutting edge AI data center and you should be set for at least a few years. Um, really cool. Thank you, Dr. Catherine Williams for coming on the show. Before I let my guests go, I always ask for a book recommendation. Do you have anything for us?

1:01:43I do. I read a book a couple of years ago that lives rent-free in my head, and I think about it all the time, and it's called A Brief History of Intelligence. I love that book. I think it's really profound in helping conceptualize what intelligence is and how humans embody it and how AI might embody it and what the future might look like. It's highly recommended.

1:02:08Jon Krohn:The author is Max Bennett, and I will have this in the show notes for people. I also do highly recommend it. It's a very popular book for being such a technical subject. It has over 5 ,000 reviews on Goodreads, and it's a brief history of intelligence, evolution, AI, and the five breakthroughs that made our brains. I was talking to a friend of mine who's an Oxford neuroscience professor, but he works at the intersection of neuroscience and AI. And I told him that I was thinking of writing this book. And then he said, well, that book already exists. And it's pretty good. And what's really, I think, I think something that's really remarkable about this book is that the author does not come from, he's not an academic.

1:02:52Jon Krohn:He's a startup co-founder from New York. He comes out of our world, but he got interested and started digging and he did all the digging and then shared his findings with us. And it's all the questions that I had that he just like, anyway, it's fantastic. It's really good. It's almost as though a PhD is just, you know, as anybody could do the work of a PhD, whether you formally get that piece of paper or not. lots of clever people out there who could be doing amazing things. Yeah, taking. It's just a credential that describes persistence. You know, that's it. All right. And for people who would love more of your brilliant thoughts after this episode, is there anywhere we can find you online?

1:03:37I am not really a social media person other than I do have a LinkedIn presence. And so if you really want to talk to me, come find me there.

1:03:44Jon Krohn:We will have your LinkedIn in the show notes for sure. And that is the most common answer by far on this show these days, but really appreciate you taking the time. I don't think you do a lot of these kinds of interviews these days, do you? So we're very lucky to be able to get your thoughts. Thank you so much for agreeing to be on the show. And yeah, maybe in a few years we can check in and see how things are coming along. Sounds great. Thank you so much, John. This has been a pleasure. What a wonderful episode today. In it, Dr. Kathryn Williams detailed how she was hired at AppNexus purely for her mathematical fluency despite having almost no data skills and how she and her team taught themselves modern machine learning as Hadoop and ever-larger datasets arrived.

1:04:32Jon Krohn:She talked about what Candid does as a non-profit supplying data to other non-profits, why deep technical understanding still matters even when AI can do the math for you because that fluency is what lets you build the mental models to reason at the next level up, her view that the most valuable data professionals going forward will be the ones who can move up and down the whole stack of complexity rather than specializing in a single slice, and how fine tuning has largely gone the way of the dinosaur now that ZeroShot capabilities have wiped it out. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Catherine's social media profiles as well as my own at superdatascience.com slash 1011.

1:05:15Jon Krohn:There's a fun binary episode number for you today. Thanks, of course, to everyone on the Super Data Science podcast team, our podcast manager, Sonja Breivich, media editor, Mario Pombo, partnerships manager, Natalie Zajski, our researcher, Serge Massis, and the great Kirill Aromenko, our founder, for producing another super episode for us today. One that's really out of this world. for enabling that super team to create this free podcast for you we are deeply grateful to our sponsors you can support the show by checking out our sponsors links which are in the show notes and if you'd ever like to sponsor an episode yourself you can get the details on how by heading to johnkrone.com slash podcast otherwise please help us out by sharing this episode with folks that would love to hear from dr williams and such a great episode like this one we had today review the podcast on your favorite podcasting app or on youtube for some reason it seems like text reviews on apple podcasts are especially helpful for people discovering this show and seeing that it might be one that they'd like to listen to subscribe if you're not a subscriber obviously but most importantly just keep on tuning in i'm so grateful to have you listening and i hope i can continue to make episodes you love for years and years to come till next time Keep on rocking it out there.

1:06:31Jon Krohn:And I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.