Spark Isn't Going Anywhere. So They Rebuilt It in Rust. — Shehab Amin, CEO of LakeSail

0:00Essentially we rebuilt Spark and Rust for the AI era. And so we thought to ourselves, what if we can turn data pipelines into AI pipelines? And that's kind of the path we've been going down for the last few years.

0:16Welcome to the Data Engineering Central Podcast. I'm Dan Beach, your host. Today we have the CEO of Lakesail with us. How do I pronounce your name so I don't butcher it for everyone? Yeah, it's Shehab. Shehab, okay. It's nice to have you. I've kind of been watching Lake Sail and watching you online and super interested to talk to you and just hear about your background and talk about whatever we want to talk about, Spark and Lake Sail and Rust and Data. I'm curious to see where you think the future is. Can you just give yourself a little intro to everybody if they're not sure who you're here? Yeah, well, first of all, it's an honor to be here.

0:53And, you know, I've seen your podcast just like explode over the years. So it's, you know, it's really an honor to be here. So, yeah, I'm Shahab. I'm the co-founder and CEO of LakeSail and the co-creator of Sail. It's an open source engine, Apache 2.0, and essentially rebuilt Spark and Rust for the AI era. And so we thought to ourselves, what if we can turn data pipelines into AI pipelines? And that's kind of the path we've been going down for the last few years. Yeah, it's pretty awesome. So when I first, I think one of the, do you know Tyler Croy? Yeah. He's the one that had mentioned, I think he was the first one I heard Lake Sail from.

1:28And when I heard that it was like a Spark rewrite and Rust, I'm like, say what? Is that even possible? I don't know. It doesn't seem possible, but apparently y 'all did it. Yeah. You know, we've gotten that a lot. It's recently become possible thanks to the Spark Connect protocol. Sure. you know, where there's a big effort separating client and server. And so you can kind of take those protocol buffer files and replace the server. Yeah, I would definitely want to talk more about Spark Connect because that is like, I feel like I don't hear very many people talking. I was just surprised when it came out.

2:02I thought there'd be more people talking about it. I never really hear too much, maybe because it's behind the scenes. But before we do all that, and I just ask you about your background. I mean, obviously, we'll get to Lake Sale and you're the CEO of it. But I'm curious about like early you, like how did you get into tech? Where'd you grow up? Were you like the classic nerdy kid? You know, did you do computer science? I'm curious about your early years. Yeah, for sure. So yeah, I was born and raised in the Bay Area. My parents, they came from Egypt. So I have an Egyptian background. I've been writing code since I was nine or 10 years old.

2:36Oh, wow. You know, so big, big computer geek. I was really into video games. And so there's the way that I got into programming was there's this computer game called MapleStory. You may have heard of it. And it's a fun game for kids. And, you know, there's this thing called private servers. I really wanted my own private server. So that's how I got into software engineering, you know, writing C++. Wow. Yeah. And, you know, I stuck through it throughout, you know, middle school and high school. I actually went to one of the first coding boot camps in the world and then got my degree in computer science from UC Berkeley as well.

3:12Awesome. So what year roughly were you at school? Like, so I can date you. Yeah, yeah. I graduated Cal in 2019. 2019. Okay. And sounds like, so you were writing code by the, that sounds like for a long time before you got to actually school and learning. learning yeah yeah i think i think it's been 19 or 20 years now okay and what were some of those you just mentioned c++ was there anything else back in the day that like caught your like techno i'm curious like the early technologies was there any databases any other languages yeah a lot of my sequel um you know i guess i guess stack overflow isn't a thing anymore but it definitely it definitely wasn't a thing 20 years ago either uh so yeah a lot of a lot of my sequel a lot of C++.

4:03I did a lot of web development I'd say, you know, like maybe 2013 to 2015 and then didn't really find that very interesting. Like PHP, I'm with you. I kind of learned coding like with the LAMP stack and MySQL and little PHP and I never really liked, I never liked the web. Like, I don't know, just something about it just, nah, it wasn't that interesting. I don't know. It seemed too icky for me, you know, HTML, CSS. I didn't mind doing the PHP, The back end is okay, but any time I had to deal with the front end, I'm just like, this is not my thing. Yeah, there's that meme of Peter Griffin with the blinds.

4:43It's like CSS. You're always trying to pull at it, try to get something to work. Yeah, I'm with you. It wasn't my thing either. I stuck with the back end. Got really deep into databases, which is led up to what I'm doing now. did you do any database stuff besides what came after mysql did you ever go to like sql server or postgres or did it kind of a little bit everything uh a little bit of postgres mysql interestingly enough mysql scales really really really well uh i would you know i'd say probably in the last five years postgres caught up to a lot of things mysql had um but uh the test with MySQL is really cool.

5:25You know, distributed MySQL. Did a lot of real-time replication. So, you know, processing data from the transaction log, MySQL, and processing with Flink or Spark. And if you want to, you know, put it in a search index or a graph database or whatever have it be. So that's, you know, that's where I've done a lot of my data processing. Sure. What were some of your first experiences and, like, stuff you worked on in jobs when you got out of school? Did you end up at like a big place, small place? What was like your first job, real job out of school? Yeah, yeah. I was a startup guy. So I've always worked at random startups.

6:04I started a startup in college actually called Traderie. Did financial modeling, trend prediction. It's actually still around. It manages a very small amount of money for friends and family. And then I worked at various high growth startups, which is where I met her on my co-founder. Sure. Did you, were you always kind of in the data space, big data? I mean, is what, like, so what year, 2013, 2014? So Spark was kind of just taking off at that point, right? That would be like Circo Spark getting big, sort of. Yeah, I would say that I didn't really dabble with Spark and Flink maybe until like 2020.

6:45Okay, so that makes sense. Yeah. um but before then just a lot of a lot of my sequel uh a lot of i did a lot of php as well um you know which which i did not find very enjoyable um c plus plus a little bit and then when was when did you start dabbling with rust because i was going to talk about like still being written in rust but when did rust come into your life yeah so i uh fun fact haran and I only first learned rust when we started building SAIL. So 2020, yeah, the very beginning of 2024, end of 2023. That's funny. Tell me about the inception of Lakesail. Was it like, tell me about the idea.

7:28Did it like kind of happen over time? Was it like sudden? How did that all come about? Yeah, it was kind of sudden. So we started Lakesail in 2023, initially just doing managed infrastructure for Flink. I'm sure you worked with Flink. It's really kind of a pain to stand it up and get it to work. And so we built a little business out of that. And as LLMs were getting really popular, at the end of 2023, I thought to myself, wait, why is no one doing AI workloads with Spark and Flink? There's a great interface for Python functions. Why does no one use it? Why is it not effective? That led down a rabbit hole.

8:10I found Alibaba had tried. There's an Alibaba project called Deep Learning on Flink, and they abandoned it. Wait, why did Alibaba abandon this? What's going on here? Why can't one of the biggest companies in the world get this thing to work? And so we dug a little bit deeper, and the answer was straightforward. You know, Spark and Flink and other traditional compute engines, they're JVM-based. And JVM, as you know, doesn't have great interoperability with other programming languages. and so python runs out of process uh and you don't have low level access to hardware so you have surday on every batch that you want to have your python do some computation on and then you don't have low level access to hardware so you can't you know leverage gpus or or whatever else um so you know it's end of 2023 and beginning of 2024 and we have this flink business and i go to her on i'm like hey like i know i know we just spent all the last eight months building this Flink business.

9:08But this thing I discovered, I don't know where the future is going to lead. I don't know what it means, but what do you think? And he was all in. He didn't even hesitate. And so from there, we both were like, all right, Rust. You know, Rust has memory safe, which makes it a blessing if you're building a data-intensive application. But it's also as fast as C. And so also a blessing for building a data-intensive application. So Haran came up with the idea of Spark compatibility. As we were evaluating the ecosystem, we saw a very long graveyard of compute engines. And so we took a step back and we were like, well, you know, Spark is really, really sticky.

9:53I think over 90 % of the Fortune 500 use Spark. And Haran's like, Spark Connect just came out. What if? what if you know how hard can it be which is famous last words how hard can it be if we took a Spark Connect server and replaced the server and that's that's what we landed on and that led us to Data Fusion sure and you know getting really close to the Data Fusion community which is phenomenal what do you think so I kind of grew up in I started working with Spark I don't know like a long time ago like before Databricks and it's weird to me because there was almost like two worlds of data back then right you did have like the my sequel postgres a little bit sequel server is huge for most large corporations right oracle was kind of like and then you had this transition period where things like big quarry were starting to come out spark like emr and aws sort of like managed spark but back then spark was nasty to work with i mean i don't think most people realize because they just open a notebook these days and And back at the day, you have to like set up servers, you know, configs, memory limits, make sure the driver can talk to all the cluster nodes.

11:09And it was just like, it was not fun to work with, which has always surprised me of like, why was, why is Spark still sticky? And I guess it's sticky now because a lot of people have made it easier to use. But I was always kind of surprised at the stickiness of Spark. I don't know if you have any opinions with like Python or SQL. And like you said today, it's still, I mean, there's lots of options out there, but man, Spark is still, like you said, it seems like it's still like the go-to for most places. It seems like. And yet at the same time, I feel like it's, it used to be not easy to use, right?

11:45I mean, depending on the data sizes you have now, it can be not fun debugging Spark. And it's so expensive to run from my point of view, right? Yeah, I mean, I think you've kind of nailed every point there. You know, the people who've lived and breathed the pain know it, deeply know it very well. I think the number one thing is Sparks Battle Tested. You know, 2009, it's been 17 years, and they've had a sufficient amount of time to think through a lot of really non-intuitive problems that other engines still haven't had time to think through. And we only figure that out at Lakesale because we've read the code base so many times.

12:31And when you go through the entire Spark test suite, some interesting things come up. So I think the battle-tested part is one part of it. The other part of it is at truly large-scale, petabyte-plus scale, there aren't many engines that can compete with Spark there. Flink can, but Flink is even more atrocious. You're right. And, you know, I guess Ray, but Ray is really more focused on distributed Python. And so there hasn't, you know, Dask tried, but doing, you know, compute in Python, pure Python, isn't really a good idea. Yeah. Yeah, I've seen over the years, I've seen like Ray, like I remember when nobody knew what it was a couple years ago.

13:21I've seen a lot of tools try. I'm curious your thoughts on, like, it seems like I've seen, like, the reaction to Spark was, like, DuckDB, Polars, Daft. Yeah, it seems like there has been a shift in the community. And it seems like it's still heading that way of, like, I don't know, people are starting to realize that, like, man, it's really expensive to run Spark. at scale and it's oh by the way like compute is compute engines have changed i don't need to have a cluster to even run hundreds of terabytes of data data you know because of tools like rust and data fusion there's things called like you know it's lazy it's streaming we can compute you know i don't need a cluster for 50 gigabytes of data anymore um it still seems like some people in the community know that and some people don't.

14:19I feel like we're in that weird still transition where people are just using Spark because it's there and they still haven't quite caught up to this idea of these new tools like you talk about Data Fusion. I don't know if you've seen that as well. Yeah, yeah, definitely. I think, yeah, it's interesting. I think DuckDB is doing a really great job informing kind of the community on, Hey, you can do a lot with a single machine. And I think we owe DuckDB a big thanks for that. Another part of it is I think if you are an organization, adding more tooling can sometimes be daunting. And sometimes it's just easier if you have an architecture and a framework built around one thing.

15:11You know, just kind of continue using that. so there's I think there's that too yeah are you also good are you worried about the I don't know trying to like be very trying to say this a certain way it doesn't come off the wrong way that doesn't make people mad it's probably impossible but it seems like I look at Spark and Spark today isn't especially with like the lake house with Iceberg Delta Lake and just Spark in general we've got huge players like Databricks who are doing their own things with Spark, right? And it's, I don't, Spark is not going to be dead, right? Spark's going to be here for a long time to come.

15:53A lot of people use it, but it seems like it's changing in the sense of like, now where it's like, when I talk to somebody, are you talking about Spark, Spark? Are you talking about Photon or Databricks, Spark? I'm just, it seems - Are you talking about EMR or Fabric or - Yeah, it's like an interesting time for Spark, right? Because it's, is it going to stay like it has been? There's now a lot of big corporations involved, you know, just things seem a little like anything could happen, right? I don't know. Yeah, yeah, that's something we've thought about a lot. And I don't know that we have the right answer.

16:29I know that we have an opinionated answer. And the opinionated answer that we have centers kind of around lake house formats. I would say that we were very ignorant to what lake house formats meant and how deep lake house formats went when we first started building Lakesale. Sure. And, you know, for a while there, we thought we could use some library or some package to, you know, offer Delta Lake and Iceberg support. uh after some time we we realized that having proper full read and write support for a lake house format requires really deep integration and it's critical it's critical to the overall data infrastructure right it can't like it's like critical it's not a side thing it's like well it needs to work and it needs to work right yeah it needs to work it needs to work right and it goes deep it goes all the way from the the semantic analysis to the planning to you know to partition, pruning, you know, to your catalog and metadata management, to your physical execution end-to-end.

17:34And so I think Spark, interestingly, kind of solidified its position in the world because of how strong of Lakehouse format support there is, right? The Lakehouse ecosystem is really, really strong at Spark. And I didn't know this until a Microsoft employee had posted about it, But Lakesail, or rather Sail, was the first engine to rebuild Delta Lake outside of the JVM. Sure. And we didn't, you know, and when an employee at Microsoft told us that and, you know, posted about it on LinkedIn, we're like, whoa, Lake. And so I think there's a long way to go for the whole ecosystem to build towards lake house formats.

18:13And I think there's a really interesting opportunity for the community to come together and, you know, think about a lake house format around maybe Parquet or Arrow. So it's kind of unifying and modernize the stack. Sure. Yeah. Do you, how has, because I'm assuming it complicated my life because I was using these tools before data catalog was the thing. Have data catalogs like through, is that like a curve ball? It must be a curve ball that sort of like comes in from left field, especially for someone like you where it's like, oh, I can just deal with Spark Connect. I can just deal with Delta or Iceberg.

18:52And that's like, yeah, and no now. Because now apparently within like the last, it seems like within the last, I don't know, two years, it's like data catalogs, whether it's Unity Catalog, whatever, went from being like sort of on the peripheries where now it's like, no, you have like a data catalog that's now like on top of everything else is now central to the... Yeah, I could have sworn like three or four years ago no one was using catalogs. Yeah, it's very interesting. You know, I think we've come full circle with the ecosystem because, you know, I'm sure you know this, but for maybe some of the viewers who don't, a little bit of a history lesson is, you know, back in the Hadoop days with MapReduce, you had this really complicated architecture with all these different, you know, fragmented pieces that you had to stitch together.

19:49And it was like this Frankenstein monster and, you know, spark comes in and it's like, wait, what if I plug in, you know, for map reduce, you know, switch out map reduce and plug into the ecosystem and I collapse the stack. And, and, you know, there's a big applause, like simplifying a stack, collapsing the stack. But now you got all these different pieces again. And it's getting more and more and more, and it's, you know, it's seeming like history is repeating itself a bit. I think we've questioned, and DuckDB has done an interesting work around this with Duck Lake. We've questioned why you need an external catalog for metadata processing.

20:29Why can't the engine do metadata processing? If you have an engine that's built for the largest scale of processing, why do you need to offload your metadata processing somewhere else? And so that's kind of the next phase that sale is looking at.

20:45Yeah, it just complicates everything, right? From like testing to like, I don't know, I remember when first Iceberg first came out and there's like this Iceberg versus Delta Lake stuff going on and everybody's doing like Hello World's an Iceberg and all of a sudden they're starting to realize like, oh, I have to, I need a catalog. Am I going to use something local? like you said it complicates things it adds another tool to the stack that some engineering team has to manage and pay for it's got to work well with all the tools and now you have package differences and things being updated a lot of vendor lock-in most of the open source like Unity Catalog open source and I guess even like the Iceberg REST open source I wouldn't say they're very production ready.

21:39I think maybe Polaris is getting there. It's in JVM, which is kind of unfortunate. Interestingly enough, a lot of companies I talk to are still in Hive Metastore. Because it's open source and it's solved end to end. There's no, you have to sign up for Databricks or Snowflake or something to unlock the core feature of the catalog. Sure. how this is kind of a question from left field but how important is and how how much did the last year to the rise of agentic coding of llms how much how important was that to like lake sale like building lakes on how important it is how would how does it play a part in your day-to-day operations for your engineers working are you when you're working on lake sale is it like what's your viewpoint on you know because you got everybody on all sides you got people that are unbelievers people that are i'm uh you know running gas town over here building everything are you somewhere in between tell me about your viewpoints and how lake sale approaches the agentic coding yeah uh so that's a great question uh we're definitely believers i would say that we started to find value in agentic coding at the beginning of this year where we found the models a lot more capable.

23:07When you get outside contributions, it's really, really, really hard. Really hard. And we get a lot of them. And it's amazing that people want to contribute and be a part of something and improve something. And so what we have to try to do is We have to try to figure out how to keep these people motivated and keep them feeling encouraged while helping guide them. One interesting thing I found is I think we're kind of uniquely positioned because part of our developer setup requires downloading the Spark code base and patching it because you run the entire Spark test suite. and so in my you know in my uh in my prompt i have go you know for every single task you do go to this directory and ensure exact parity with spark and so if i want to get a contribution from outside contributor i'll go to my you know if i could tell it's clearly vibe coded i'll go to my my clod or my codex and i'll uh you know give it the pr and say hey you know review this thoroughly ensure parity with Spark and it'll come up with like a list of 30 things after three hours.

24:23And then I'll paste that as a first pass. Internally, I found that Fable is really good with some architecture planning, but not actually writing code. Fable is really bad at writing code, I've found. And so if you want to do some architecture planning or whatever else, Fable's really good at that. And then I personally don't put like quad or fable on autopilot mode. I do the manually approve thing. Sure. And I know that's very inefficient, but if I'm in meetings all day, because I don't write as much code these days, then in between meetings I can kind of see what quad or codex is trying to do, approve or deny.

25:07But we still do a lot of manual coding. Sure. We just find that there will be a lot of edge cases and bugs and incorrect assumptions otherwise if it's not tightly controlled. Yeah. Do you see yourself generally optimistic or pessimistic about the future of software engineering? Because I feel like we can't put the genie back in the bottle like it's out. And humans are humans. and they're gonna i mean you're always gonna have a subset of engineers who love coding like an art form right and they're gonna love to write their own code and that's good and be awesome but at the same time i feel like well the genie's out and always putting pandora's box back together i don't know as someone who like works at a startup and writes a lot of code it's like i I don't know.

26:03I feel like torn because I had to learn rust without any help. I had to fight the borough checker and do all this stub my toe. And I maybe it's just me being old and grumpy, but I'm like. Writing coat by hand, like sweating, getting angry, working on something an entire weekend because you just had to figure it out. There was no one to help you. Spending an entire weekend on two lines of code. Yeah. All right. Like that sort of I feel like that's done. and I mean it is what it is like it's not that I don't like to go to Claude and be like where's this bug what did I do wrong like it's who's not gonna like to do that but I feel for the younger engineers because like how are they gonna pick up those skills and maybe they will I don't know maybe the assembly to see guys said the same thing I don't I don't know yeah yes I I think about the same thing all the time i think it's really really really hard to learn uh if you know if you kind of have a genie at the touch of your fingers as you as you said uh unless like you're the most disciplined human of all time uh yeah i fully agree with you i think it's going to be a lot harder for the younger generations but maybe you know maybe uh it won't matter maybe you know future you won't need um you know you won't need to know how to do that and then just the theory and the concepts is all that matters because you know today if you're if you're um you know an engineer with a little bit more experience and you you know you understand you know proper system design and you know architecture and and you know writing code at scale um it can be a superpower because when you know when quad or codex output something you can immediately tell no this architecture designs full of crap or no it's okay this is solid um i think uh with hiring in the younger generation i think what a lot of big tech is doing right now is not right um you know there there there's a there are a lot of layoffs that are increasing every month and it's not right and so um you know lakes hill at least doing everything it's power as long as I'm steering the ship to make sure we can hire people and keep people employed.

28:20If we continue the trend of laying off everyone, it's going to be a bleak future. And I don't hope for that. I agree with you. It's sad. I think it's short-sighted. I don't know if they're using current world situations as an excuse to, I don't know, just lay off people because they can save money, invest in data science, but I don't know. It's all complicated, but I agree. It's sad. And I really feel for all the people. It's like a bad time. You know, you get laid off and it seems like it never, like you said, it's never ending, right? It seems like the last two years, like every three months or something.

28:58It's just like you think it's going to end and it's more people and it feels super sad. I wanted to talk about a more, a happier subject. I don't know. I would assume a lot of people know what Arrow and Data Fusion are, but maybe not. I'm always surprised. It seems like Data Fusion in particular is always, not for me, but for a lot of people, it's been kind of like under the radar. They may have heard it in passing. Same with Apache Arrow. They probably feel like they've seen it on their LinkedIn feed or on YouTube, but a lot of people, it seems like, still haven't interacted much with those tools.

29:39Can you give people a high-level, what is Data Fusion, what's Apache Arrow, and why is Lakesale built on it, and how is it different from the past, and what makes it so attractive? Yeah, so Apache Arrow is a columnar in memory format. It is really useful because it allows you to leverage vectorization in SIMD, which allows you to essentially leverage the hardware for extremely fast and efficient processing. It also allows for zero copy execution. And so you're not serializing or deserializing data, which, again, allows for really fast execution. It was created by Indy Grove and Wes McKinney, I think, who were the two co-creators of Arrow.

30:27And then Data Fusion is a composable query engine in Rust. Well, I'll just take a step back. So Arrow is a format, right? And so it's implemented in many languages. There's Arrow C++, there's Arrow Rust, there's Arrow Go, there's probably Arrow Python. So Data Fusion is a composable query engine written in Rust. based on Apache Arrow, also created by IndieGrowth, and heavily pushed forward and contributed by Andrew Lamb as well. Man, Data Fusion is really a blessing to the data engineering community. It's so composable that you can break it down to bits and pieces and just take the tiniest piece of it and use it for your project.

31:17And because of that, it supports such a wide kind of widespread of companies out there, including Lakesale. So we're really heavily involved in the Data Fusion community. Every pre-release of Data Fusion is actually tested on sale to ensure correctness and catch bugs and whatever else. I think Data Fusion it's hard to I think it's hard to state its impact because its impact nobody talks about it I feel like I mean here and there but I feel like people don't realize that everything they're using is being built on top of Arrow and Data Fusion yeah especially Arrow and even more so today Data Fusion for sure I think because it's most of the time you can run data fusion out of the box and run it as a query engine.

32:14But most of the time people use it as building blocks for other engines. And maybe that's why you don't hear about it as much. Yeah. And it's sort of like related to this topic. Tell me about what you think. I don't know. This question is about like what I would call classic like batch processing versus streaming data pipelines. And we've sort of seen this sort of quasi shift in Spark. You've got like SDPs coming out in Spark that were released not that long ago, declarative pipelines. And I feel like there's always been this, I would say like more a decade ago, it was always like two worlds, right?

32:55but you had Kafka and streaming, and you had this whole other world that was not in batch. That's what people call that. And it seems like those two worlds have been coming together a lot, and I feel like that's because of things like Data Fusion and Arrow. And it feels like there's this push in the community. I'm curious how Lakesail has approached it. How do you approach streaming versus batch workloads? Do you just write Lakesail, and do you differentiate between true streaming and what I would call batch? And how do you, because it was, you know, we're talking about different infrastructure behind the scenes, different, complicated.

33:35How did you approach that problem with like SAIL? Yeah, that's a great question. So with SAIL, we, you know, we wanted to unify batch processing, stream processing, compute intensive AI workloads, and also ad hoc workloads. and interestingly enough we achieved stream processing through spark's uh uh structured streaming api which is uh it's actually micro batches yeah micro batches funny enough uh we were able to do proper stream processing through it um today spark as you said has you know sdp and and other real-time streaming APIs that they're adding.

34:19Stream processing, I would say, is definitely a buzzword right now, it seems. It seems like it's being used in a lot of cases where it doesn't need to be used. Sure. And there's probably, you know, I would say that there aren't as many use cases for stream processing where you really, really need it. From an architecture standpoint, from a distributed processing architecture standpoint, because I think that's really where the difference between batch and streaming kind of is important. The architecture is actually the same. If you're doing batch or stream, the architecture is the same if you have the right foundation.

35:02By the right foundation, I mean it comes down to shuffle. So in traditional batch processing with shuffle, which is when you're exchanging data between nodes, That writes to disk or memory or some storage. Writing to disk is slow, but it's great for resiliency. So when you're doing stream processing, you can't write to disk. You want to send data node to node, right? So you want to do something called like a pipeline shuffle where it's all done over IPC, Arrow IPC. And so Arrow in particular made stream processing really intuitive for sale because we were able to use Arrow IPC to send data between nodes and not have to write to disk.

35:50And that also supports ad hoc workloads as well. And so we have a unified shuffle format where depending on the workload you're doing, you can do blocking shuffle or you can do pipeline streaming shuffle. Yeah. I'll leave a link in the notes I recently did a review of Sail and Lake Sail on my sub stack just totally unrelated by myself to heard about I've been meaning to get to it I was impressed with the because I test out a lot of tools right when they come out people that's kind of like my thing I'll just go try new things I was impressed with the developer experience out of the box with Lake Sail because I've tried a lot of new tools and you'd be surprised a lot of times people's websites just don't work.

36:32Their north doesn't work. Just deploying it just doesn't work. So I was, kudos to that because it worked. And I, you know, the first time, which is amazing. It never happens when I'm testing out it works the first time ever. Thank you. Yeah, we saw your review at Lakesail and we were like, we were freaking out. Everyone was so happy. Like, it was, it made everyone's week. So thank you for that. Yeah. How do you, this is kind of like a high level. How do you, how does LakeSail, how does Sale approach? It's a busy data world out there, right? There's new tools everywhere. Everyone's got a million choices to pick from.

37:12Everything's shifting. AI just, you know, every company's an AI company. Just like, I don't know, seems like a crazy world right now. And how did you, how do you reach developers? What's your approach? How do you grab their attention and especially data people? Because I feel like it has to be easy for data people. And what I mean by that is like a lot of data people that work in data are not necessarily software engineers. A lot of them come from other places, right? There might have been a financial analyst or who knows. I just, you know, I see data people as being more, you know, it's like you're talking software engineers, But then you also have a lot of people who don't have that background who maybe need, let's say they just they want SQL on top of their data because that's what they can do.

38:06And it seems like a difficult job to take the complexity of something like, oh, we support the full Spark protocol. And you just you have to hide so much under the you have to hide the P under so many covers to make it like this button click. because that was my experience with sale it was like it wasn't like oh i gotta go meet under the old oak tree at midnight and throw salt over my left shoulder and do like you know what i mean it was just like a couple button clicks that's just it works it seems like yeah difficult thing to do well thank you yeah we we spent a really long time developing the uh you know lake sales platform probably like two years um and uh i was really hoping it would resonate with people because we were building something for that long which is generally kind of you're not advised when you're in a startup to spend that much time on something uh so we really went all in on this and i really designed it as what i wanted as a buyer so i've you know i've been a buyer uh for a long time of databases or data infrastructure, whatever you have it.

39:17And whenever I wanted compute in my own cloud, I wanted control of the data, I wanted control of the compute, some engineering team from whatever company I was buying their enterprise software from would have to spend six months setting it up. It takes forever. And so we thought to ourselves, what if we can eliminate all steps? You just click a few buttons and we fully manage infrastructure in your cloud and your data. Nothing goes over the internet. It's all on private subnets.

39:55It was hard. It was a lot of work. But I think it's important because ease of use is so important. And yeah, there's, as you alluded to with AI development, there's a million companies out there. There's so many people building so many things. And so I think if you can focus on the user experience and making people's lives a little bit easier, that can sometimes go a long way. Tell me more about the, this is sort of related. We talked about it a little bit before, but the other thing that people don't realize, it seems like we sort of talked about it with the Lakehouse format, storage formats, catalogs.

40:33We alluded to this, but there's so many tools in the data space that are just, they're there, like sparks there, right? like sale supports the spark protocol api because it had to right because spark is there and everyone's using it and you've got things like on a data platform you probably have some orchestration like apache airflow right like everyone is using it and how do you decide which tools you're going to support and how does like how do you approach that if i want to call it a problem i call a problem of like you're building sale and it's like well half my clients are you know like 80 percent of the people are using Airflow and they whether I want them to or not they run most of their pipelines there so I need to be able to you know provide an easy operator like because data platforms are also not just about the compute itself but typically in a real world production environment right you've got many complex sort of data pipelines doing all different things maybe on different systems have you tried to tackle that problem with sale of like how do I pick tools that I know I need to work with or integrate as well or do you just try to build like a good foundation and hope that you know people can figure it out yeah great question uh so on our platform that was you know think thinking about orchestration tools was really important and so one thing that we did from day one was uh versioning so job versioning and job and job run sure uh which is kind of like the core piece you need if you're going to use like a prefect or or Airflow or Daxter or whatever else.

42:07On the open source, it's a lot harder. We had this interesting split between half the people who were interested in Iceberg and the other half were interested in Delta Lake. We're a small team. We're a small team. Man. I don't envy you. I mean, I get it. It's just like, and nobody knows where it's going to fall. Like, which side is going to fall on? Who's going to win? Yeah, I think both are going to win and both are going to stay. And so we went to support both.

42:46Luckily, there's a lot of overlap between Delta Lake and Iceberg. When you get to the catalog, the catalog was really interesting. So, you know, one thing that Spark has done right that other engines haven't done right, at least that I've seen is catalog operations. So if you're doing like a create, update, delete, whatever catalog operation, you don't want that to happen in your planning phase, right? The logical plan, the semantic analysis, the kind of the understanding and the like, hey, this is what the execution DAG is going to look like. Yeah, kind of simplified terms there for some of the viewers.

43:28You want that to happen in your physical execution. when the processing is actually happening. A lot of engines do it in the logical phase, which is not good because it's not meant for execution. So one thing that we had to kind of re-architect around and rebuild around was building our own catalog interface and Rust in order to support that properly. Then there's new DBT.

44:01I've avoided dbt I mean it's me too I have avoided it yeah personally I I don't understand it either I've never I've never used it but it's so popular everyone uses it it's so like that's another thing everyone uses sequel and everyone now wants to say I use sequel with dbt they just it's a cc ctl is probably right they just bought into this I gotta use dbt and I don't Yeah, and so we have dbt sale, and that was another thing. Okay. And then catalogs, Unity Catalog, Iceberg REST, Hive Metastore. Man, just so much going on out there. Hive Metastore, really, that would hurt the most. Whatever's going on there is screwed up, man.

44:54has it been difficult on the side or not so much and what I mean by that is as a whole like Azure AWS GCP is that at all like throw wrinkles in building data platforms is it kind of like at this point as long as I can throw up images somewhere in a container it's for the most part similar or is that still like a does that give you headaches that sort of I'm on GCP I'm on AWS on the open source no because on the open source you really just have to support the the storage so s3 or uh azure or um i forget what google calls whatever yeah gcs whatever uh and then there's hugging face and there's cloudflare um that's that's pretty pretty simple on the platform so our commercial offering, that is a pain.

45:46So we support AWS today, and soon we'll support Azure. Sure. And I don't know if we'll ever support GCP. Yeah, don't. It's probably not worth it. I haven't really found anyone on GCP. Yeah, I would say, you know, the ecosystem's changed a lot. Five years ago, I didn't know anyone on Azure. Today, like 40 % of the people coming to us are on Azure. That's more than I would have guessed, but I guess it makes sense. It seems like it has to do with the size of the organization too, right? Sometimes Microsoft just has their claws and things. That's just the way it is. Right, yeah. It also could be, I think, Databricks has done a lot of deployments on Azure, and Databricks and Microsoft kind of had a big partnership there for a while.

46:41So that also might kind of attribute to it. But definitely for a startup, you know, that's kind of a pain. And so now we have to, all right, well, let's plan out Azure. And, you know, thankfully we have a lot of infrastructure experience and we've done this rodeo before, but it's kind of a heavy lifting for a small company. Sure. Has anyone come and asked you, hey, every other tool I use has some sort of like agentic sidekick, plug in, MCP, da da da da da, whatever. Are you ignoring all that? Or are you sort of like on the fence? What's the future of sale when it comes to agents? And are you just kind of waiting it out?

47:25Are you starting to build? Are you trying to support stuff? I don't, whenever that means, whether it's NPC servers or who knows, I'm just curious, kind of like, are you like, I'm thinking about it or working on stuff? Oh, not really. Yeah, we have an MCP server. Actually, we weren't one of the first ones to have one and the Spark one is actually based off of ours, which is pretty cool. An interesting thing about SAIL is the server startup time is instantaneous and we support single node and we support distributed compute. and so if you have a single node workload and you want to do salespark run it will do a one shot Python script execution so it'll take your Python script start a server execute it return the results and then tear down the server all in one shot that's free for anyone to use in the open source along with the MCP on our platform we have this really interesting feature called sessions and so sessions allow you to tunnel into your compute from anywhere and without exposing you to the internet.

48:30And you get kind of like a unique token that's access and time scoped. And you can generate that token and give it to an agent and give it to Claude on your local terminal and it'll have access to your compute. Or you can use it yourself. It's very useful for your own workloads if you don't want to be inside your private network and have to connect to the private network to execute or even on the platform. So that's how we've thought about it. I think that we haven't seen agentic workloads, at least that I'm aware of, I could be totally oblivious to it, take off in the data space as we've seen in other places.

49:04And I know that's maybe because of compliance and regulations. And if something's wrong with data, a lot of people get in trouble. So I'm curious your take on that. Yeah, I don't know. It seems like the Wild West, right? Yeah. I don't know. just recently at work I've been building some, I would just call them like quasi agentic data pipelines where you're like, you're doing something in airflow and you have like these just normal tasks, right? Oh, it's compute or just whatever, very deterministic. And then, you know, and now we have a request from the business where we need some sort of agent call or on call, which is very non-deterministic.

49:41And it's definitely this weird world where like people, I feel like it's sort of dependent on the business. And what I mean by that is like the end users, what they're demanding, what they want to do. And it's, I mean, I feel like the business always lags a little bit behind technology. And now they're starting to like, now just your average marketing, a product person is like starting to use Cloud and starting to realize and they're thinking and they come ask, hey, can we? You can have access to the prod database. I want to do this. And it's like, it is a weird world, I don't know what the future holds, right?

50:18Because it's definitely only going to increase those people, like the end users and whether they're internal of the business or external where they kind of starting, because it's starting to become part of the culture, right? People are just using ChatGDP to like, hey, I need to fix my toilet. How do I do that? So this is like becoming part of the culture and then it's like starts to come into the business and people start wanting certain features and wanting to integrate agents in certain spots. And I don't feel like the data, even though there is a lot of like agentic tooling, it hasn't, it does not feel like the data community or even the software community at large has really like solidified on like, how do we integrate these non-deterministic things?

51:04To like this other, this old stuff that we've been running that's totally deterministic, right? I don't know what that is, right? Because you're talking about frameworks and tools and how does that work? I know Airflow released some like what they call, what they call them, like AI provider operators. So you're starting to see kind of like sort of stuff take shape, but still seems like the Wild West, right? Yeah. Yeah, I'm with you there. I think that everyone's talking about using agents in production, which means that no one's using agents in production most likely. Yeah. I think as cloud, you know, as the cloud became really popular, infrastructure engineering became really relevant.

51:48And so we'll probably see a new class of engineering that's an infrastructure type role, but for these kind of internal agent frameworks or architectures or systems for organizations. And so I think that might be an interesting role in the future. yeah um i know we're rolling up on the end of the hour here end of time but i wanted to ask you how it's kind of sort of related to ai but how do you are you reaching developers and people and finding people for sale and like sale have you found that was like the explosion of ai content it's like harder to reach people how do you are you approaching like reaching data people are you like kind of like on the open source side doing stuff just trying to do the best job you can and if they find it they will come how's that experience been for you in the you know the day that we live in how do you reach people find people whether it's data people or customers curious yeah it's gotten significantly harder uh going to lots of events i think is really important connecting with people in person sure is really important um we got really lucky in the last few months google search ai started recommending sale sure uh but obviously there's nothing anyone can do to control or make that happen uh i think high i think high quality manual written posts do really well agreed uh and we have we have a blog post that to this day is still making the rounds around the internet it's our how to build a sequel parser and rust um uh or like you know the the post that you wrote about lakes on all your other posts there you know you can tell that a lot of time and thought and effort and love was put into it and that kind of you know when you read it it doesn't feel like a chore when you read a lot of ai content it's like a chore almost right um uh surprisingly i you know going back to the basics and and and putting on a you know a lot of heart a lot of love into something i think um appeals to people yeah i agree yeah that's all we can do really i totally 100 agree with that um if people want to find more about you where can they follow like sale sale and github are you on linkedin where to i can leave links to all this stuff obviously in the show notes to wherever people are consuming this yeah what is your what's the future the last question's like where's like if you can whatever you can share like what's the future for sale lake sale what are you excited about that's coming and where can people like find you and follow you yeah uh so lakesale.com github.com slash lake hq slash sale for the open source uh where they can find me i'm in san Francisco always, uh, all the data then San Francisco today.

54:51I'm in New York. Uh, I'll be in Amsterdam for PI data the week of September 7th. Uh, and so I'm, I'm constantly traveling for events and I, if you're enough, I'm coming to your city, I'd love to meet with you and talk with you, uh, where Lake Salem sales going. Um, it's a really interesting time. We've spent, you know, a really long time, maybe two and a half years, just heads down building. and we're starting to see some really interesting validation from some of the largest companies in the world, which is really exciting. I think if you are thinking about feature engineering, so you want to do feature engineering on top of your data pipelines, that's kind of our North Star.

55:35And we're seeing a lot of people with these data pipelines that process everything about the business, right? Every day at midnight, anything that the business knows goes through that data pipeline. What's interesting about that is that's great for an AI pipeline. And so if you want to turn your Spark pipelines into AI pipelines and start doing feature engineering and embeddings and kind of understanding of your organization, that's our future and that's where we're heading. Awesome. Yeah. everybody go check out the links and the show notes to Lake Sale and I'll link to your LinkedIn and everywhere else stuff like that go check them out they do awesome stuff I've checked it out myself and impressive and I'm excited to see where you guys go in the future I appreciate you joining me today it's fun to talk yeah thank you it's an honor to be here you know it's a highlight of my career to be on such a prestigious podcast so thank you so much for inviting me and thank you for the time no problem at all thanks well have a good day