987: AI Infrastructure, Ray, and Why Nonlinear Careers Win, with Linda Haviv

20:27You had to be with a label. Today, it's very democratized, whether it be content, whether it be building something you don't need to hire, maybe as many people, but also the software developer take what's in their mind, what they wanted to solve and do it for themselves. And the cost is lower in some respects, right?

20:43Jon Krohn:In fact, it's almost just as in music. So in music and performance probably in general, it seems to me now like there's almost an expectation that somebody already has developed a social media following before they could get a record deal, right? And so that kind of reversed things, whereas previously you'd get the record deal, you'd be on the radio, and then you have a following. You were in the mercy of, same with news, right? I used to work in TV news, and one thing I realized was, oh, I love media, but you know who the happiest people, in my opinion, were? The people who came as specialists.

21:20The lawyers. The doctors. Why? When they needed to not be on air that day because they weren't feeling well, they could not be on air. But when you live a news cycle, you don't have—I mean, there's no balance in many things, but I think in TV news—and it's the 1%. And in a way, you have so much of your own authority of what you want to talk about, what you don't want to talk about in your own media. Now at the time, it wasn't as common in 2016 to have your own media channels. But for me, I was like, I prefer to come in as an expert versus just, I think what journalists do are incredible. I also realized what I enjoy versus not enjoy.

21:54And I think you have to ask yourself that because I think you could curate your life today. You could be the artist of your life and curate it based on the things you want because you have the power today to take it. And I think it's like everyone could build something, but what you build is really important. and going deep and figuring out where you bring that skill set, where you find joy. I think we're in that era. I think there'll be a creative renaissance in a way too. So live your wildest dreams.

22:16Jon Krohn:Yeah. So all of these things that we were discussing earlier in the episode is being scary from one perspective where as a software developer, as a data scientist, as an AI expert, we used to have this inbuilt moat that only, you know, this small percentage of us in the population could write the Python code or whatever, the Rust code. Or design, like you could be an engineer, but I'm not good at designing. I'm not good at project management, but like AI helps me a lot. Right, right, right, right, right. So yeah, so there's all of these different aspects where you still bring your particular background, your particular expertise, gives you probably more confidence to pursue any given direction.

23:00Jon Krohn:You know, if you're thinking, I wanna build a media business, with no experience in media, you can go to Claude and have the conversation, but are you going to have the same level of confidence in these recommendations you're getting from the Claude chat than if you've lived that path in media, you've seen what works, you've seen what doesn't work, and you've developed connections in that space as well. So you can say, hey, I, you know, so-and-so from media company X, you know, I've built this web app that I think solves these problems that you have. Would you like to see it? them already knowing you, they're much more likely to take the call to look at your app than if you send them a cold email.

23:39Right. That's what it's like. I think for people, especially who are like many years in the industry and they're like toying with that, there is a strength to it. And I think, of course, obviously, you have the more junior developers who are struggling right now coming out of college. I think it's harder to get junior jobs. So they're also seeing them also going very entrepreneurial as well. There is a democratization. There is an abstraction We've seen this happen though, you know, in the age of bootcamps, you know, 2016 time. I think I went to a coding bootcamp at the height of coding bootcamps.

24:06I went to the Flatiron School.

24:07Jon Krohn:I think you did go at the height. That was like the time. There were, cause at that time there was, there were also really great data science ones like Metis. That was, I mean, that was a cool, I went - It was in person too. Yeah, in person. And I think that was a key to it be, was it Flatiron in person as well? Yeah, Flatiron in person. And I think, you know, to do an online bootcamp, I mean, and we actually, you know, we, this, the Super Data Science podcast, our name comes from superdatascience.com, which runs online bootcamps. And there are absolutely advantages to doing it that way. They have, you know, we have instructors all over the world, in Australia, in Asia, in New York.

24:47Jon Krohn:You can get the best people and you can obviously be anywhere in the world in any kind of personal circumstances and do the curriculum. So there's definitely advantages to that. I didn't mean to say that there isn't. However, for me, and it sounds like probably for you, I think we might skew towards extroverted. Just a little. And that kind of experience, like for me of knowing that I'm gonna be getting dressed, showering, going to Flatiron, to go to Flatiron school, to be around these 20 people that are in different stages of life, different backgrounds, but all of us looking to become software developers, that is so exciting for me, having coffee with them.

25:27Jon Krohn:and then they would have like, the reason why I know Meta so well is because I would come and I would lecture once a term. Yeah. So you were the lecturer. Yeah. And that even, you know, as the person providing the lecture, like that's a lot more fun for me than going on a Zoom. Totally. And I think there's a part of it that's also really mindset. When you're trying to do a new, a different, a career change, so much of it is believing you could do it, especially when things get very overwhelming and when you're learning something and actually sticking to it in three months. I think for me, I joined the bootcamp in person because I wanted community.

25:56I'm a people person. I learned better. And let's say if you're building something in a silo, okay, so you built something, but you know where you learn when you read somebody else's code and then you have to have it work with yours. That's where like, you know, where you're reviewing each other's code. So when we had to do, and yes, we were able to do that virtually, but it was a different vibe for sure. And I think we're going to find this layer, in my opinion, again, I think in-person events are going to be very popular right now because as we are able to do a lot digitally, I personally, again, this is my personal opinion.

26:23I think we're going to need the mental part of it, like to build together it's not that we can't do everything virtually it's that we're going to need it in some respects uh to reconnect with our human element and even if we're struggling to figure out what to build i think by being in person you just like you find your answers through not just sitting you know taking a walk going taking a shower you know like they usually have to be doing something i think community is so crucial right now um especially as we navigate game. We learn from each other. We each come with different expertise. There's no way to be an expert in everything.

26:57But we all bring different strengths and we all maybe also want to get better at certain things. And by leaning into that mindset and by being with people who are building and are doing that and finding your community, I think it helps you up level as well. And that's just as important as learning everything, building.

27:16Jon Krohn:For sure. Yeah. And for For me personally, there's just a lot of fun and joy. I mean, just like us being able to record in person here today. Yeah, it's a lot more energizing for me, even though I have to, you know, it takes a bit more time to come in in person to get dressed. Yeah, but it's worth it. I don't know. I really enjoy this in-person stuff. Anyway, we ended up talking about this. This has been like a really interesting, like 20 minute long conversation. We did. But this all started with me saying that you'd recently left AnyScale. Yeah. And you have touched on some parts of the answer here in the last 20 minutes of conversation.

27:58But why is now such a great time for people to be doing things independently or to be

28:06Jon Krohn:building things on the side, even if they're not ready to go completely independent yet? Yeah. Yeah. I think one of the reasons I went into software development in general is because it's something you could do on your own. It's a skill that no one could take from you and it's not reliant on a company. Right. Because companies, the people could be great, but you're a number. The people you work with might not be and the relationships I have from companies are my lifelong friends and people who changed my life. But the company as a whole, you're a number. You're a human resource. Yes, you're a human resource.

28:37And it's like the system is built. It doesn't matter. No one really controls that really. So I think for me, I learned that early on, in order to show up as my best self, even in my full-time job, I always liked not having all my eggs in one basket. And for me, it was also because I just loved community. It started as something that I wasn't actually doing. Sometimes if you pursue your passions.

28:58Jon Krohn:So yes, you're talking about this is why it's valuable to have a side hustle. It's like insurance. Right. It's insurance. And it also ended up adding to my full-time job because by creating a personal brand, my job at Amazon happened because they found me talking about cloud computing on TikTok. So sometimes life is not, you know, you can make a plan, but life does not work that way always. Especially right now, I think also being an artist of that direction is really important. So back to your question on why I made that shift, I really believe that, one, we're living in obviously an unprecedented time.

29:35And I always think to myself, what would I have done when the internet came out? And we're living right now in that equivalent. And I'm like, well, I've already, I've worked 10 plus years. I actually want to continue working with the companies I've worked for in other ways. But I think I could say, I could be a bar raiser in combining my skill set and communicating.

29:56Jon Krohn:And I do have here a list of some of the brands that you've worked with independently, which includes Anthropic, Dell, Amazon, GitHub, Microsoft, NVIDIA, and quote unquote, many more. Yeah. And, you know, that's the thing. You know, I was like, oh, I love corporate. I love working for a company. And part of me was like, but I could still do that in other ways and consult. and actually I could be much more intentional because sometimes when you work inside a company, you are pulled into a lot of things that just take time and understandably they're like red tape that happen and that's normal. But when you're not in that scenario and you're not full-time employee and you have a project you need to commit to and a campaign you need to do, it actually ends up being more efficient for me.

30:40And at this time, I was like, this is the right time for me. But I also think that, I don't know the percentage, but maybe even 50 % of jobs will be like small companies because right now everything's getting leaner, right? I think things are getting, and again, this is not some scientifically backed metric I'm giving. It's very much where I personally see, like I think there's going to be a leaner, obviously, way to work. I think we're going to see two to three billion, two to three person billion dollar companies, maybe even one person billion dollar companies. And I also think, obviously, it's not just money.

31:15It's really like empowerment of like, how do we navigate this next phase? Do I want to be in the forefront? And can I do that in a company? Absolutely. Where do I personally think I'm at the point that I could give more? Is it, and I asked myself this, like, is it the right time or the wrong time in my career and my family's life? Also, I have three kids and I wanted certain like flexibilities as well. Not that I'm not going to be working more for myself, but I think there is a point that I felt like I could actually do better, more good for developers working independently and still advocating for all the things I love, including Ray and AWS and all these things in different ways because I've gained enough that I'm able to combine and bring a different interpretation to things and help people navigate.

32:00Jon Krohn:Yeah, I had a similar journey to you and actually in some ways on similar kinds of timelines where it was also 2015, 2016 that I started creating content. And it was also, it initially started with meetups in person. So doing talks at data science, computational statistics, meetups in New York, that's how I got started with content creation. And then somebody from Pearson happened to be at one of these talks and said, would you like to create a video course? And then that, you know, created this really popular deep learning video course. And then that same Pearson team says, would you like to write a book on it?

32:38Jon Krohn:And then that ends up being a bestselling book. And then I got invited to be on the Super Data Science Podcast as a guest. No way. On the back of my bestselling book. And about six months later, Kirill Arimenko, the original host and the founder and still co-owner of this show, said to me, do you want to start, do you want to become the host of the world's most listened to data science podcast? and yeah. And then now several years later, you know, after doing that for about five years, about four years, I got to a point where similar to you, I had enough kind of different, different clients, different ways of impacting people where originally I got started in the content creation because I wasn't doing it for the insurance aspect necessarily.

33:29Jon Krohn:I was doing it because as an extrovert, it's hard for me to stay motivated to learn when it's just for myself. The weekend comes, and specifically the content that I was creating in deep learning 10 years ago, I wasn't creating that content because it was popular at the time. I was because it was obvious to me that there was a huge amount of potential in deep learning. And for about a year, weekend after weekend went by where I was like, I need to spend this weekend learning about deep learning, but then ended up going to like the pool instead. This sounds so similar to my AWS journey because I was talking about AWS before I ever joined AWS.

34:07And I just, for me, it was like finding other people that were also learning with me on the certifications. And then it was not touched upon enough. Like you could count how many people on TikTok or like Instagram, especially in those areas were talking about cloud computing. And because of that, I met a lot of people that we first met on social media, became friends across the ocean in different countries and ended up working together. you know and it's like how the world is so much smaller also and we think about open source and we think about developer tooling and how it's actually the same thing in a way right where you contribute to different things and you're able to find that community and i think in that way it was when especially like i think right now that exists in ai infra a little bit i don't think enough people are actually let's say on short form video talking about ai infra and the depth that needs to be there it's on x it's on youtube but i don't think it's like i had videos go viral on TikTok that are about AI infra.

34:57And I think there's people assume, but until you don't try and you don't go technical there, people are still there. It's just that you're dealing with different audience and maybe you need to make it something that they could watch at 2am, but they'll learn something. And how much of education is also how you retain something. Like if you're reading something from a paper, you're not going to remember the same way as you listen to it on a podcast or you watch it visually on a YouTube video. You know what I'm saying? it's it's there's different ways people learn and i think um there is a part that i've i was like oh this is what the way i love to learn i love x2 and twitter and stuff like that was where most tech content was but i realized that the women in tech preferred instagram and that's where i was finding my community of like women too and i was in devops and for me that was like i was looking for my people also throughout that and and cloud and i think um the amount of community i found in cloud on those platforms was incredible.

35:52So similar to you also, what is your one lesson also that you feel like today for people kind of navigating that or entrepreneurship, what do you think people should be thinking about?

36:01Jon Krohn:It does seem like a time where you're saying the barriers to entry to be building applications because we can be orchestrating teams of AI agents to be building products for us. If you're not able, especially if you're not able to get that in your day job as somebody in our space. If for whatever reason, your role in a business, you have this very specific narrow niche, or maybe you've even... Actually, it applies at all stages of the career. It could be that you aren't in the technical role that you eventually want to have. You're in a non-technical role. We have a regular listener, Adriana in Germany, who is a flight attendant but she's been studying data science courses for a while and at some point soon i'm sure she's going to be going for you know her first technical role but so whether you're in that kind of situation coming from you know a non-technic you know your your resume up to date doesn't have that line that says you know uh software developer at meta yet so yeah whether you're that kind of person whether you are in a specific niche because maybe you're in a very big corporation.

37:14Jon Krohn:And so you have a specific niche carved out for you. And even though you're really excited about all the new exciting things happening with agents, you don't have the opportunity in your nine to five. Or it could be the case that maybe now you've actually grown really far in your career. You used to be hands-on, but now you're a manager. And so regardless of which of those three situations you're in, which together are probably the majority of our listeners fit into one of those three buckets. Whichever you're in, if you're excited about this new co-gen, agentic moment that we're definitely in where it has never been so easy, so powerful to be building AI applications, production quality.

38:00Jon Krohn:We have guests on all the time talking about open source frameworks or proprietary products that you can be using to get secure, reliable agents into your hands and into potential customers of yours hands. Or potentially, you know, it doesn't even need to be a customer. You could be thinking about how can I be, what kind of project can I do to make a social impact with these kinds of AI agents? And so regardless of what situation you're in career-wise, you can on the side be experimenting. And, you know, there's great courses. My friend Ed Donner has great courses on getting into agentic coding and also fantastic yeah follow him yeah he's he's incredible he has um so we co-founded a business together no way yeah um so he so ed donner you know there's few people on the planet i know better than him uh i i left a very comfortable amazing corporate job in 2015 to join him at a startup that he had just founded in 2015.

39:05Jon Krohn:It was a startup called Untappd. He was the co-founder, CEO of that business. I was chief data scientist, not a co-founder, but in 2020, that business was acquired. And then him and I co-founded another business together. And so from 2020 to 2024, late 2024, him and I were co-founders of a business called Nebula. That is so cool. I had no idea. Yeah. And he's in New York as well. I need to interview you. Okay. We have a lot to talk about here. But yeah, Ed is amazing. And he has about 18 months ago, he made the leap that you and I made in 2016 to creating content on the side. And oh my goodness, his stuff has taken off in a way that I don't know if I've ever seen before from anyone.

39:51Crazy.

39:51Jon Krohn:He has half a million paying students on Udemy for his courses. In 2025, he released about a new 20-hour Udemy course per month. And so whatever aspect of agentic engineering you're interested in, so that's from, you know, engineering the agents. So, you know, writing code to create agents or the flip side of that, using agents to engineer. Yep. I like it. Whichever of those you're interested in, Ed can sort you out. And actually, you know what? I have a surprise for you, Linda. Oh, uh-oh. Because this is, given how much you love Ed, something I've never said on air, even though this has been in the works for a long time, is that him and I are writing a book.

40:37Drum roll.

40:39Jon Krohn:And it's about, you know, so we're basically, you know, with my experience book writing, we're taking his content, you know, these wildly successful Udemy courses. Incredible. and yeah, packaging them into a book format. That's amazing. Well, first of all, we need to do the clap right here. I feel very honored that this was shared here, by the way. I didn't know that. That's amazing. And I think, you know, books, courses are fantastic. And having like, I always say like, you need multiple ways to learn the same thing and having that other format as well. And I'm not, I will need to pick your brain because I think writing well, like well-written books is a whole art in itself.

41:19Jon Krohn:Well, and you may not know this about me, but I am now for, so, you know, going back to 2016, when I started creating content and writing books for Pearson, I'm now the series editor for all of the AI books that they publish for professionals. Oh, no way. And so, uh, when you sleep. Well, and so we can talk about, you know, you having a book for sure. Oh, I will. I will. That's amazing. First of all, I I'm very excited for this book that's coming out. So, and that also, that goes for any listeners who think they have a book ready to write, feel free to reach out to me and we can get you with the - So many skill sets.

41:53Jon Krohn:Well, but it's all about communication, right? It's all about taking technical concepts and being able to, in different formats, like you say, different people learn in different ways, but whether it's a book or a video course or a one minute long YouTube short or Instagram short. People consume in different ways and learn in different ways. I think that's a thing. There actually more and more people who actually need to also learn and we're all also going on that journey right building in public right where it's a very big cultural thing right in tech where you build in public why because when you're at a certain point you're able to explain something that in a year from now you will not explain the same way and so sometimes i always said this to people when they were like i would give uh certification classification courses like i started like an internal one uh when i was like a software developer when i was upscaling to cloud and I was always like one step ahead right but the the reason it was helpful was because the way I explained it then would not be the way I was so deep after that that how I would explain it to someone would not sometimes be the best way for them to grasp it at that time and there is always this part where your perspective is also really important in the way you're educating is like you have a voice in that and not to feel like you know this part where like oh I don't know just make sure obviously everything you're saying is accurate but I think there is a part where it's like learned experience is much more raw when you're able to like be hands-on and freshly explain something after you were hands-on on something.

43:17Even if you don't understand every single thing, say that, be honest about it. But I think you will be surprised how much you're teaching someone that is very much close to where you are. Maybe you're one step ahead of that.

43:27Jon Krohn:Yeah, 100%. Yeah, so really exciting for anyone. Hopefully maybe a little bit of extra encouragement here from us around how doing stuff on the side can lead to unexpected positive consequences. Who dares wins? And just by, you never know. I do encourage you if you, for whatever reason, feel like creating content or studying is something that's useful to you, doing it in public even before you have an audience I think is really important because that knowing that you're gonna be pressing publish on that LinkedIn post or that Instagram video or whatever, knowing that someone might see it. And in the beginning, not everyone's gonna be an Ed Donner.

44:16Jon Krohn:But in the beginning, it's gonna be, you know, maybe very few people looking, but it could be, you know, an old colleague or something. A few people look, you get some feedback. And just knowing that someone's gonna be watching, it forces you to have this extra level of making sure you understand what you're doing. They say the best way to learn is to teach. Yeah, yeah, yeah. kind of puts the deadline for you and you also have a responsibility for someone else but and to add to that i think you find also that the feedback as you were saying and the community so much of it is about like sometimes two people in the same spot as you that you will interact with are worth more than the hundred thousand you know it's never really about it's about finding right the right people great if it's like a hundred thousand people that are also in that but i think even for i'm sure you and i both like it was kind of like that personal branding part and that content part comes from also finding like-minded community and individuals as people as extroverts but this is also for everybody else if you're an introvert this is probably the best time too because uh you could be not having to go to social social uh events and be able to do that from your house and in whatever format that works for you just because it worked for person a if you don't like doing that kind of content you like doing this kind of content do what's right for you you'd be surprised how much like sometimes what you there's not one way for it to work and I'll give you an example.

45:34When I was making tech content in the beginning, I had two young kids at home and I couldn't do talking head videos. So I would do like music with text videos and I would still educate it other ways. Music with text overlay videos. And those ended up really working well for shorts.

45:48Jon Krohn:But in a way, like that was also the only way I was able to film. I was able to take B-roll of what I was doing as much as I wanted to talk on camera. That was just not the format that worked for me at that point in life because I was like hiding in the closet, trying to edit these videos. So I think my point is lean into like also the things that bring you joy in education. So whether it be and start there, don't overwhelm yourself on the 10 steps. Like if you built something, take a screenshot at first and write out a caption. Like it doesn't need to be the full video. If you are nervous, you usually have to build that comfort level.

46:21But I do want to get into that. The personal branding is also really important. And it's not because like branding or to be an influencer, it's to find like-minded individual community because you're building with more people around. It's the same concept as contributing to open source, right? It's like it's open because we're all innovating together and you're much more likely to bring the right things to yourself and opportunities if you put it out there than chase them because you don't always know what's out there.

46:45Jon Krohn:Yeah, spot on. All right, so wow. This has been great. And none of this has been in my plan. Almost 100 % of the episodes so far. So some episodes, and I don't know, maybe listeners can tell in some way or other, but some episodes, we have a research plan. We have an amazing researcher, Serge Massese, who's a great data scientist in his own right and content creator in his own right. Author. I was following him on LinkedIn. Yeah. Brilliant guy. And so he comes up with, you know, a rough conversation flow based on everything that his AI agents can find about guests online. And yeah, so, you know, some episodes I have a plan and I relatively stick to it.

47:29Jon Krohn:In this episode, this entire conversation so far has been on the way. Yeah. Which does happen. You know, sometimes it makes. Yeah. It's a vibe. But I will get to now the first question that we had. So I started by talking about how you'd recently left your staff developer advocate role at any scale. But I want to talk about what you were doing in that role because the open source framework Ray that you were advocating for there is sensational. I mean, we have been AI software companies that I've had for years have been using Ray as a key part of our stack. So tell us about Ray and why listeners need to know about it.

48:09So to level set, if you don't know what Ray is, Ray is a Python native open source distributed computing framework. And what's great about Ray is, right, it's open source, Python native. Spark is probably something that people use that's I think for AI workloads, Ray works really well, especially AI infra. And the reason is, one, you have more multimodal data you're dealing with, right? Video, audio, text. And I think as a genetic AI systems get more complex, right? You need that distributed computing part. Many engineers are not coming to start being orchestrators or anything. An ML engineer wants to just build something.

48:44They don't want to start being a distributed computing expert potentially. And it actually was born that way. and I think Robert would be a good guest by the way on your podcast. Robert Nishar actually, co-founder of AnySkill and co-creator of Ray, they were at Berkeley, Jan Stojka's lab. And they created, because of reinforcement learning at the time, they were ML researchers and they were spending more time on distributed computing and distributed systems over building and working on ML research. And so a lot of it was to help that. But the funny part was that because they made it work for reinforcement learning today, it ends up, I mean, reinforcement learning for a bit was kind of like not talked about.

49:23And now it's like back because a lot of

49:25Jon Krohn:RLHF, right? A lot of the biggest models today, DeepSeek, all that, they're all using RLHF and different variations of reinforcement learning. So I think now it's, and it's not just for reinforcement learning, the library's expanded. So the thing about Ray is it actually works on multiple different parts of the stack. So RayData, RayTrain, as it sounds, for training, RayServe for serving. And it's the same language, Python native. It's fantastic. I definitely recommend checking it out. There's also a lot of native integrations if you use it. If you're using, for example, VLLM and HuggingFace and all that, there's a lot of native integrations.

50:02And maybe if you're doing distributed training, you'd be using RayData and RayTrain. There's a a lot of different libraries pretty much built on top. I think I'm like throwing a lot of information, but for the most part, it's helping engineers not have to be distributed computing experts and writing in the language they're used to building in. And whether you're building an MCP server and you want to serve, you could use Ray, or if you're doing distributed training, you could use Ray to distribute the workload because a lot of this does not fit. The problem you're solving is it's not fitting on one GPU, right?

50:34You have multiple GPUs, TPUs, and all the multimodal data we have in the world, 80 % plus of it right now is important because that's your unlock for businesses.

50:45Jon Krohn:And it's open source, so you can get on it right now as you're listening to this episode, obviously at no cost with the full functionality. Now in your response there, as well as actually many times throughout this episode already, you've used the term AI infra. And I'm not sure if we've used the full word yet, which is infrastructure, as opposed to say, maybe people thinking inference possibly. Yeah, because I swallow my vowels. I'm like from Brooklyn. No, that isn't what I mean at all. This has nothing to do with you. No, I know, I know. But what is AI infrastructure exactly? Like that is something that you are expert in.

51:19Yeah.

51:20Jon Krohn:And how do you define that? You know, that's a very good question. I wouldn't, I don't know if I'm going to define it in the way that like maybe the industry wants to define it. And just like anything, I think DevOps has the same problem, right? If you know the definition of DevOps, people say, oh, my DevOps engineer, but it's really a culture, right? And as it expands, it gets complicated. I think we do that a lot in tech. I find AI infrastructure to be still infrastructure, but the thing that actually is more aligned to the problems that are different with AI ML workloads. And there are a lot of things that happen that are infrastructure-like, right?

51:50Same thing as any other infrastructure that's also not AI, but that you're dealing with specifically with AI, right? So Ray, for example, is built for AI because it's the way it's actually working under the hood. One, because it's Python native and most of a lot of things in machine learning are built in Python, it works better for that. But additionally, because of the way it functions with GPUs, right, where you're kind of not starting, you're not letting it just waste as much money, pretty much, in the way it's filling the workload. Because AIML workloads are, like, there's a lot of variation of issues that come up with AIML that are not in traditional infra, especially like deterministic ways.

52:29Deterministic or not deterministic? Is it like, I think, let me backtrack for a second because I think I'm going very granular, but AI infra in general, if I had to define it, and again, don't kill me here, but for me, how I look at it is like anything that really is also infrastructure specific to AI ML workloads versus like your traditional infra. However, there's overlap.

52:48Jon Krohn:And so traditional, you mean like AWS EC2 instance would be traditional compute infrastructure. Well, that's why neoclouds exist now, right? Like Lightning AI. Yeah, anything, because you don't need as much of the virtual, you know, the virtualization, right? You need direct bare metal access, right? So I think there's a lot of things that are happening that are challenges in AI ML workload. That's why you have frameworks like VLLM that came out that are open source, right, for inference. Inference is the first place you'll hit it, right? Because that's probably the more common place you'll hit it.

53:19Inference is, you know, whether you're doing it for a finance company and you're trying to do batching, batching of inference, or you're doing this as like trying to serve a lot users, you're dealing with inference every day, right? But that's the first place you'll probably see those challenges that are different than regular inference, right? Like than traditional inference.

53:36Jon Krohn:So it's in inference that you're likely to see the first infrastructure challenges. From like a user standpoint. I guess AI, you know, infra engineers have always, you know, I think there's also a lot of new roles all the time. Like you have the ML platform engineers. And then, you know, when you're a DevOps engineer, you might be an SRE or you might be an infrastructure engineer. You might be called the DevOps engineer doing all of the above. And you might be a CICD pipeline person. I mean, every company, again, and I'll say this because there's a lot of debate around it and I've seen it. In a corporation, you might have more silos of what each role does.

54:07In a startup, you might be doing all of the above. And I think we are going to see new terms. We're seeing this now all the time. There's new terms that didn't exist because what we're really trying to do is give it a term to explain the problem. But AI infra is pretty much any infrastructure that I think is really associated with building out the workloads we're using today, which do behave differently. They are much more compute heavy. In the past, we were probably more reliant on having data input output, but today we're really dealing with a lot more compute heavy workloads. And so that's a different challenge than traditional.

54:40So when I say that, I think the compute stack is also changing a little bit. And that's why Ray exists, to distribute that load. You needed Spark for other use cases, But for AI ML workloads, you needed to integrate with VLLM and you needed to also distribute the workloads in different ways.

54:55Jon Krohn:Yeah. So to do a bit of an analogy here. So Spark was kind of like analogous to the EC2 instance in terms of providing you with the infrastructure you needed for traditional compute loads that didn't involve GPUs. But now that we do have GPUs as a key part of serving any AI capability or training an AI model, that it now counts as AI infrastructure. And so open source tools like Ray become useful relative to something like Spark. A NeoCloud like Lightning AI becomes useful relative to AWS, Cloud Compute perhaps. although of course they're also developing. I call it the specialist. I'm actually doing a video tomorrow.

55:42I'm going to do this in the airport where I'm doing an analogy between traditional cloud and neoclouds because I think that each are kind of overlap. Traditional cloud are actually building a lot of stuff that neoclouds do. However, neoclouds are just, they're specialists, right? And they're pretty much really building out what you need for AI infra, right? And I think there is a good analogy there in a way which I will work on. I'll put this video. But there is this part that like you have this problem that's growing very fast and you need to solve it faster because we're moving very fast. So NeoClouds are really solving that.

56:17And you have the Nibis and CoreWeave and, you know, Cruzo and all of them, right, that we're seeing more and more. We'll hear a lot of their names for sure. And I look at like, you know, AWS, GCP, Microsoft is like they do everything in the airport, right? And then you have like that specialty route. That's probably. You're filming this at the airport for the purpose of - Yes, yes, I'm psychotic for this video, yes. Just for the video.

56:37Jon Krohn:You're not flying anywhere. Well, I am. Tomorrow I'm going for a while. Yeah, yeah, yeah. I was like, I take advantage of the situation. I'm like, you know, I should do something in the clouds. All right. Analogies help. Oh, for sure. We did some research on a popcorn analogy that you had. Oh, yeah, a batch inference. Yeah, I mean, wow, now that I've said it a lot, you might as well just tell us about it. Because there is, you know, popcorn. It's like, imagine you're eating popcorn. I think I was trying to compare... I think the video was all about explaining batch inference. And so I think I was sitting there with a pen.

57:12I was like, so when you're talking about regular inference, you would be just maybe putting one... You're prepping your CPUs. Prepping data is usually on CPUs. So you're prepping the data and you need to distribute that. But you put the butter, you put the salt, and you're like kernel. and then if you're only taking one kernel and you're popping it on the pan and then you're, whatever, that's great for getting something in a fast way, right? So you're dealing, that's latency optimized. But then if you need to batch a bunch of popcorn, that's not the best way, right? Let's say you're in finance and you have so much data that's gonna take you years to do it that way.

57:49You might wanna do a bunch of them. So you might go to a microwave and put the full bag of kernels in there. the problem becomes that the microwave is always on and you're paying for it so what happens you pay a lot of money and you need the the pre-processing to happen so it was kind of an analogy of pretty much trying to show how how you need kind of the pre-processing of the cpus you needed the prep of the kernels to be ready for the amount of uh microwave popcorn in the yeah yeah so you're not wasting money um and not you know and not starving your gpus but i like that

58:23Jon Krohn:and the visuals make it easy. So even just like as you were speaking there, I was seeing this single kernel of popcorn and how inefficient that is if I need to do hundreds of thousands of popcorns. Yeah. Yeah, so the visuals there are super helpful. Going back a few minutes to something you were saying, you were talking about all these specialized roles that are developing, which is a natural consequence of so many more technologies, so many more capabilities, so much more demand in the AI space And actually, so something, the reason why I'm bringing this back up is because today, the day of recording, Monday, April 13th, I made a post on LinkedIn that regular listeners might be interested in checking out.

59:04Jon Krohn:Check it out. Because this show, the Super Data Science Podcast, has been around for 10 years. And what was originally one role, data science, and maybe at that time we had like data analyst, data scientist, there weren't that many different kinds of related roles. But now 10 years later, all these AI ops, ML engineer, AI engineer. LLM ops, everything. LLM ops, just, you know, so many different careers. And you could potentially, depending on how you wanted to categorize it, you could say these are all subspecializations under a big data science umbrella. Not necessarily everyone would do that, but I do.

59:41Jon Krohn:But the thing is that we are thinking, the show is almost 10 years old, and we're thinking, is the name Super Data Science Podcast? Does that make sense given how much this field has evolved? And so, yeah, check it out. So I might make this post again, but if you go to my LinkedIn and definitely if you scroll to Monday, April 13th, I'll probably post at least one more time before the survey closes. But if you wanna have a say on the Super Data Science Podcast brand, does this name make sense in this day and age given the stuff that we talk about on the show? So yeah, let us know what you think about the brand, about potential alternative brands.

1:00:21Jon Krohn:We will make the changes based on the feedback that you provide us. So - I think this is such a great topic because the tech overlap role thing has, I think even from a DevOps standpoint, that has always been such a pain, but the speed in which it's going in right now because of like AI ML is like, and it's also, I think the best way to also understand that evolution is like the story behind it. Whether it be like big data and the internet came and then you had like machine learning, you had to make sure that your models also have their own ops problems. And so you have jobs just for that. Yeah.

1:00:57I will weigh in on that post.

1:00:59Jon Krohn:Fantastic. Yeah. I love the name. Well, I'm glad you do. And that's what we're just - It's good for SEO. Well, yeah. And we're not necessarily changing it, but we're just, we're evaluating, We're doing an exercise. And so let us know your thoughts. Speaking of posting and models, you've been doing a lot of posts recently on new exciting developments in the AI space, like open source models, rapidly closing the gap with proprietary ones, which is exciting. And you've also been posting a lot about protocols like MCP, model context protocol, moving toward vendor neutral governance under the Linux Foundation.

1:01:35Jon Krohn:So how will these kinds of developments, open source, accessibility, how will that redefine the competitive landscape? I think especially around, like, I know we were talking a little bit about AI. In general, I think open source will catch up. You have also a lot of contributions that could happen today with Agendic AI in a different way. But I think especially when you have governance with Linux Foundation, Ray actually recently also joined the Linux Foundation. and you have VLM, Ray, Kubernetes, it's like the stack. It's like the AI compute stack, right? And I think it's crucial, especially there's also with MCP joining as well, I think it's crucial for it not to be owned by one company and for it to have governance.

1:02:17So that's obviously, and also from the community standpoint, I think there's processes and all that that are really important. And as far as open source, I think it opens, funny, it's pun, but it opens a lot of opportunity even as we see more people build servers in their house. Okay, using open source models like Gemma 4 that just came out and DeepSeq and all that, I think it allows you to also build in different ways and fine tune and update things and that allows for a lower barrier to entry. As we were talking about a lot, it's democratization. I also am very big on encouraging people to contribute to open source in their career.

1:02:53I think that you also find your community and you innovate in different ways and you're able to learn a lot through seeing the code there too. And yeah, MCP is also a great example. In general, I'm very passionate about open source. I think one, from a community standpoint, as you could tell, I'm a people person. I think the open source, I think the culture of innovation and being able to build on each other will always win in the long run. We have to make sure that's the case. I think also it allows us to all be able to keep up as well.

1:03:21Jon Krohn:But yeah, I was just trying Gemma 4 on OpenClaw. Yeah. It was good. Nice. What have you been doing with OpenClaw? So at first, before Claude got really, I feel like, okay, now I'm kind of more on Claude than OpenClaw. But I think initially I was trying to make it my AI assistant that was just at home and building out a bit of a newsletter I want to do and some automations there. although I would say that Claude works really well for most of that right now for me as I've been building a lot more skills and plug in you know just all the stuff I'm also trying to always keep up and as of recently leaving any skill I get to build a lot more it's very new but there's a lot of building for myself as far as systems I think that's also my tip always first thing solve your own problems and then maybe and then you'll be able to probably share that and teach and uh so lots of skills lots of mistakes too some of my skills were overlapping a little bit so I'm I think the best way to learn is to always build.

1:04:19But now you could also watch your agents work. But yeah, I kind of have been in, I call it the experimentation phase because I try almost like, I try a lot of different things. Gemma 4 I really like for my phone because on airplane my Wi-Fi is really bad and now I could just talk to AI from there. I also have used Olama and put it on my laptop and have that running too. But I think sometimes I don't want to switch the Wi-Fi from the phone to the laptop. and I'm like with kids, so I can't always open the laptop. So got my handy dandy phone.

1:04:49Jon Krohn:And you and I are both, this podcast is sponsored by Anthropic currently. And you've had Anthropic as a sponsor, but we both love Claude genuinely. Love, love, love. It's like, it's one of those things when, you know, a broker of these kinds of deals came to us and said, Anthropic is interested in sponsoring your show. And you're like, great. Great, because this is so aligned. I talk about you all day. So yeah, and then this open source ecosystem thing, agents, tying back to earlier in the conversation, this is so great for anybody, regardless of where you are in your career, regardless of if you just wanna be having some fun, making an impact, learning skills in your free time, or developing a side business, or going off on your own to develop your own business, all these kinds of open source innovations pair really nicely with the agenda capabilities that we have today.

1:05:48Jon Krohn:Now, final career question for you. Yeah. Given that you have now left your role at any scale, you're getting things going on your own. It sounds like you might not have decided on exactly what the name is of your next thing, at least at the time of recording. It may be something that you know by the time this episode is published. But what specifically are you doing? and what kinds of people amongst our audience should be reaching out to you for help? Yeah, so I think first step for me is I'm continuing with the tech education I've always been doing on content. I do have some consulting, especially around DevRel.

1:06:25I'm very passionate about helping companies with developer advocacy and B2B companies on that.

1:06:30Jon Krohn:So in that case there, given all of your experience in developer relations at AWS, at any scale, and then of course all the stuff that you do independently, you will now help enterprises, not just literally with the content creation, which you're also available for, but also setting up the systems, the operations, understanding how to succeed at DevRel as an enterprise in general. And especially around digital communities and building that. I think there's so much accessibility that happens. Developers are all over the world. And there is a part that you could really make a sticky community through social as well as through other ways.

1:07:04And it needs to all tie in, as well as enabling people internally. I think a lot of people, founders especially, are the face of a company. And their brand is actually a DevRel version, right? Especially technical founders. There's a lot they bring that no one else could bring. And that's a really important, I think, part that I also coach on and specialize in and have done also for companies. But I think that has a different art because there's a thought leadership part to it. There's a part that they don't have time also to deal with. scaling themselves. But that is very much a part of the why of their company and not something that's a company page thing.

1:07:45It's more of a personal story. It's their own technicality and it's their own thought leadership that also drives, a lot of the times that's what drives the company. So being able to take that in different ways for people to understand, visually explain it, and scale that message is really also important. So specialize in those kinds of things as well. And yeah, I'm excited to build a lot So I think this is the most exciting time in history to build. As we were saying earlier, I always ask myself what I would do. So I'm building a bunch. First, I'm of course building your typical application. I'm building an app right now that will be just for my family to keep track of my kids' schedules and birthday parties and everywhere in one place.

1:08:23And agentic workflows, I have my morning briefing, I have my, you know, for even like contracts I'm doing, I have my agents and my scheduled tasks that are happening. so using Claude for all that. And yeah, so of course, initially also building a lot of my own systems to be able to scale as well as consulting and we shall see. There's more to come, but I'm super, super excited about it.

1:08:50Jon Krohn:Fantastic. Yeah, very exciting. So great to have you on the show at this important point in your career. Thank you for having me on. Yes, of course. I mean, love to have you on again soon. This has been such a fun episode. but before I let my guests go, I always ask for a book recommendation. What do you have for us? All the chip books. Okay, all of them. Every single one. Yes. AI engineering probably is the first one. I definitely think it's a required read.

1:09:21I know people have different visual learning styles. I think that one was fantastic. So I know you had her as a guest here too. Yeah, I think we had her on the show

1:09:29Jon Krohn:talking about that book. Yeah. So look back at that episode. We should link it back. And I think that is one of the most popular episodes we've ever had. It's fantastic. I felt like when I was trying to get a holistic view, you know, we're always upskilling. And for me, I needed that book. I have it on my desk. Has a bunch of bookmarks in it. Written in it. So thank you. Thank you, Chip. Thank you, Chip. Yeah, definitely recommend. And in general, I think reconnecting with more like human stories. and I personally just love, I love tech and I love everything tech and I think finding our elements of how we solve with technology human problems.

1:10:10I think this is the time. And so I'm even working on trying to reconnect with certain things as well. While I feel like there's not enough time in the day because I am very much in the rabbit hole, like everything from reading the whole paper about mythos to like every new thing that's happening and trying to actually make demos about it. A lot of my passion is also making like hands-on visuals for people to actually set it up and try to help people actually adapt without having to waste as much time or giving them crash courses. So that's kind of my mission right now. Initially kind of take that information and then show you, okay, here's, let me save you time.

1:10:45Here's what you're going to do. And yeah, and I think most important message is be the artist of your own journey and lean into the, what you actually enjoy doing tech. You're able to do that with that today. Don't get overwhelmed. We're all professional students at this point. Just new terms every day, new roles every day.

1:11:08Jon Krohn:And yeah, so for people who want to stay close on the pulse of all of this content that you're creating, we will have links to all of your social media profiles in the show notes. But do you want to list out for us all the various places we can find you? It's a little confusing because my LinkedIn name is Linda Haviv, and my socials are all Linda Viva, which is Haviv backwards, V-I-V-A-H. At this point, I didn't update it for SEO reasons because I actually found out that, you know, I actually show up in quad search. So I'm like, okay. Yeah, so what you're saying there is that you've left. So on Instagram, on various other social media platforms, you have YouTube.

1:11:50Jon Krohn:Yeah, all of them are Linda Viva. Everything except LinkedIn is Linda Viva. So for people who've been wondering why, I've been calling her Linda Haviv in this episode. That is, in fact, her real name. My work name was always Linda Haviv. And her LinkedIn name. But yeah, for finding her on all the other - These were the days, you know, we separated content from working full time. Right, exactly. Yeah. But now you say that you're leaving it with the backwards Haviv, Viva. It was actually my stage name for music. Ah. Well, it's good because it's like life. It's life, yeah. There's something about the ah, you know?

1:12:20Yeah. I don't know what it is. but I was like, you know, it's a good album.

1:12:24Jon Krohn:Yeah, and now it has great SEO and AI agent EO. Yeah, EO. But if you like AI engineering, AI infra, a lot of those topics, and in general, upskilling in this age. Fantastic. We really appreciate you doing all that work and also taking the time away from all of your kids and all of your business ventures to spend time with us and my listeners on the show. Thank you so much for having me on. Thank you. You're amazing. Thank you.

1:13:20Jon Krohn:stack tooling, and frameworks purpose-built for AI and ML workloads, how Ray is a Python-native open-source distributed computing framework that lets engineers distribute training, data processing, and model serving across GPUs without needing to become distributed systems experts. She talked about how open-source models like DeepSeq and Gemma are closing the gap with proprietary models and key projects like Ray, VLLM, and MCP are moving under Linux Foundation governance so they aren't controlled by any single company. We've got links to all of those technologies for you in the show notes. And finally, Linda talked about how building in public, creating content, and contributing to open source are not just career insurance, they're how you find your community, attract unexpected opportunities, and learn faster through teaching.

1:14:06Jon Krohn:All right, as always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Linda's social media profiles, as well as my own at superdatascience.com slash 987. Thanks, of course, to everyone on the Super Data Science podcast team, our podcast manager, Sonja Brejevic, media editor, Mario Pombo, partnerships manager, Natalie Zajski, researcher, Serge Massis, writer, Dr. Zara Karche, and our founder, Kirill Aramenko. Thanks to all of them for producing another ebullient episode for us today, for enabling that super team to create this free podcast for you.

1:14:40Jon Krohn:We are deeply grateful to our sponsors. You can support the show by checking out our sponsors links, which are in the show notes. And if you'd ever like to sponsor the show yourself, you can see how to do that at johnkrone.com slash podcast. Otherwise, please do help us out by sharing this episode with someone that would love to have a laugh and learn from Linda. Review this episode on your favorite podcasting app or with a YouTube comment on the episode. Subscribe if you're not already a subscriber. But most importantly, I just hope you'll keep on tuning in. I'm so grateful to have you listening and I hope I can continue to make episodes you love for years and years to come till next time.

1:15:15Jon Krohn:Keep on rocking it out there. And I'm looking forward to enjoying another round of the super data science podcast with you very soon.