The Future of Data Engineering in the Age of AI | Erfan Hesami

0:00Erfan Hesami: I mean, because we can't put the AI back in the box. It's here. It's here to stay. People are going to use it. I mean, it's just the way it is. And it makes me think about.

0:15Erfan Hesami: Welcome to the Data Engineering Central podcast. I'm Dan Beach today. I'm your host today. And today we have Efron. No, I get it right. Erfan. See, I knew I was going to mess it up. I get used to it. Don't worry. Yeah, it's no surprise there. I messed it up as usual. So you run, if people don't know who you are, you run Pipeline to Insights, a very big sub stack. I'll leave links to all your stuff in the notes of wherever you're listening or watching this. You can go in the show notes and you'll get links to the sub stack. It's a pretty nice sub stack. And then are you also, you're on LinkedIn?

0:51Erfan Hesami: Are you trying to do LinkedIn too like everybody else or trying to crack that net? not very active as much as sub stack but yeah i'll try to be active as well so we can yeah i i think sub stack is a great platform and because i started on linkedin and it's gotten i don't know i got started early and i don't the algorithm it's all messed up and i'm i'm super happy for sub stack it's a great platform especially for data creators and people want to write about tech it's uh been cool but yeah why don't you give everybody just a quick overview of who you are, what you're doing, etc. Yeah, thank you so much, Daniel, for inviting me.

1:29It's a pleasure speaking with you in this podcast. My name is Erfan Asami. At the moment, I recently promoted to LiveData, an AI platform engineer. I want to back to the history of my career. I did my bachelor in IT engineering. I'm not sure if this course exists anymore or if that changed. And then worked for two years as a software engineer and decided to continue my education in data science. I moved to Australia in 2017. I was my master in data science business analytics here. and I've got my first data role as data analyst at audience insurance and then moved to tech company, Xero accounting software.

2:26And got another data analyst role there and another tech engineer. And the curiosity that I have about the whole data lifecycle make me think about, all right, let me learn more about how these systems generated and gradually shifting to data engineering. And at the AFL, Australian Football League, and currently at AirMaster, like leading data platforms there. That's awesome.

2:59Erfan Hesami: Yeah, I have a pretty similar background because I did. I got into data by being a data analyst, the same as you. got into as an analyst and then went the same kind of that route into the analytics space and then kind of like you i went more into the back end of data engineering etc were you like growing up were you kind of so you got your degree um obviously in it computer stuff whatever did you were you like into programming as like a young kid like what why what brought you into like tech if you want to call it that in the beginning when you were younger was there somebody in your life was it something you worked exactly a good question my siblings honestly they i my brother's assistant they're all very in computer science engineering and in our home always someone working with computer or assembling cases or so on i remember one of my brother has a small business like assembling computers and cases for families.

4:07Yeah, so I exposed to it, and then I started learning, working with it. And when I was actually programming, when I was at high school, mostly, we got some classes and then tried to be creative, just experience how it works, problems we can solve and gradually start and i remember i start with c and c plus sure to school and then other languages do you think that uh what did you so you obviously run

4:46Erfan Hesami: a big substack called pipelined insights when did you decide to start like sharing things online talking about this is that relatively like new to your journey where you always kind of little bit into it? Yes, it's interesting, like a story. First, it was around the same time that I kind of started shifting to data engineering. So it was something that I wanted to push myself to learn more. Okay, now I've moved to data engineering. Then I figured out, yes, I have the title, but I still need to learn more. Yeah. And, uh, the idea of soft, like itself came from when I was in a conference data inch by, I'm sure if you heard that dependent data engineering conference in Australia around two years ago, I guess where Joe race, Zach Wilson, they were like one of like the guests there talking about soft stack a lot.

5:53And same night when I came home, I said, all right, let me explore what soft stack is and what, let's start writing there. I see people like my writing and so the content I create them. That's how I actually started. I'm so very glad I did it because it helped me a lot in terms of learning, because if you want to write, you want to read a lot, and if you share it with other people, then you need to deeply read about it, understand the fundamentals. And, yeah, I think the last two years, Substack helped me a lot in terms of my thinking.

6:31Erfan Hesami: Yeah, I would agree. Substack and writing in general, right, especially in tech, because I feel like tech people aren't the – if you do something wrong or you say something wrong, they're going to tell you. You know, everybody's got an opinion about every language or framework or how to solve a problem, especially in data. But I agree with you. I don't think people realize I'm a fellow substacker. Obviously, there's a lot of work that goes into it. People just think that, like, I don't know, people ask them. I'm sure people say the same thing to you. They want to know how you started. How can I have a substack?

7:02Erfan Hesami: It's a lot of just commitment, right? You got to love what you do, what you're writing about. it's a lot of time people probably don't really get how much time we put into it right you said you're reading docs you're playing with tools you're experimenting so if you're going to write about it you're right you kind of gotta you gotta learn it and play with it otherwise you know it gives you something to say for sure exactly you need to dedicate a lot of time and commitment but it's worth it. I mean, at least for me. You grow very fast, faster than anyone else. So we had kind of the same, you mentioned before you started as an analyst and went to analytics.

7:45Erfan Hesami: And I kind of had that same trajectory of starting as an analyst, analytics, and data. And this is kind of jumping ahead a little bit, but do you feel, I mean, obviously you're working with AI a lot every day. is there i mean because a lot of i'm sure you've found this a lot of the things you learned were because you did those things right because you were an analyst and you had to do a certain task or an analytics and you were making dashboards i'm sure doing analysis talking to end users whatever um with ai coming into the picture and it just everything's at a prompt are you worried about the future of for like the other generation that wants to get in and learn because they They don't necessarily have to stub their toe or, you know, I don't know, try it.

8:32Erfan Hesami: You know, I mean, like you said, you learned writing C and C++ and you just did that whole trajectory. And I feel like with AI, it's been kind of shortened, right? That window, you don't necessarily have to take that path anymore. Exactly. Yeah, there's something I've been thinking about it, if I not say every day. But most of the time during the day or during the week, at some point, as anyone else, I don't know what the future exactly looks like. Even the engineers in big AI companies, like I know recently one of them resigned, somehow they're thinking it's not under control at some point. But what I can see, like try to convince myself is I can control what I can control.

9:26So for future generation or like juniors, I can say it's hard to enter the data engineering market or data market. But it's doable. I think universities also or the courses that they are doing or boot camps, they are updated. So they can expose to latest technology and they can bring those knowledge to business. However, they need to shine. They need to, because the competition is high, they need to find a lot of ways to shine. Build continuously. And I suggest use AI to like showcase what they know, what they can achieve. And as a hiring, I'm not hiring at the moment, but if I want to, I mean, if I see any juniors is that they know core fundamentals of data engineering and continuously learning AI.

10:29I would definitely give a chance to them and I'll try to learn from them because we need to also provide an environment that we learn together. And why not a young junior who recently exposed to the latest, I don't know, technology and ideas can also support what we are doing.

10:55Erfan Hesami: I noticed on your Substack, you write a lot about AI, like specific tools, like very in-depth. Clearly, you're an expert. You use this stuff, build stuff all day long. It's obvious from your content that you know what you're doing. Do you think that there will always be like data engineers and AI engineers? Do you think that will merge? And what skills do you think are like complementary from like a data engineer? like where do they cross the ai data engineer like where do they cross overlap and then what skills have you seen like on the ai side that are maybe new to like so if somebody's been a data engineer for 10 years and what are some of the new skills and things that you've seen and tools that are what i guess would be called like new new to them or maybe unfamiliar to them Does that make sense?

11:42Yeah, really good question. And I ask this for myself as well. Based on what I see, I believe some responsibilities, they kind of like their boundary are getting close for sure. And I think both sides, a good data engineer, need a good AI engineer need also to be a good data engineer and at some point data engineer also need to learn about AI systems, how agents works, how model works find a way to even automate some part of repetitive tasks that they have it can be unique to them but at least they learn about it how these models evolve, improve their limitations. So both sides, they need to learn.

12:44And what I know, and I can say to data engineers, that all these AI models, agents, whatever is happening, a good data needs someone to engineer that. And that is the job of data engineer. However, in some companies, they might in future upcoming, they might expect someone who can do both developing an application and AI agent and also do engineering. So the more that their task, like I can say some basic documentation, some basic ETL will be more automated with AI. they will have more time to spend on learning AI engineering or doing AI engineering as well. So that's what I suggest, to think about it for data engineers.

13:37Yeah, it makes sense.

13:39Erfan Hesami: I've seen, like at my day job, I've built plenty of agentic tools and customer-facing stuff, and it seems like clearly data is a big part of those, whether it's a chatbot or any sort of reg system. I clearly data is fundamental AI engineering, whether it's, you know, data storage, but yeah, just get in, right. You have all this stuff about, you got to get the data in a place. It's got to be clean and, you know, in the correct formats and vectorized all this stuff. I felt like some of the pieces that I would call were new to me, quote unquote, not totally new, but I feel like in AI engineering, there's a very big emphasis, at least from what I've seen on more of like, you probably get exposed to more of the front end of the stack, maybe than we would have classic data engineers, right?

14:29Erfan Hesami: just writing pipelines and worrying about the data where now a lot of these AI toolings end up, you know, it seems like you got to know a little bit more about infrastructure, right? You got to learn Terraform. You got to learn how things are deployed. Probably have to learn a little bit about the web technologies, probably a little bit. It just seems like, yeah, there's definitely some new stuff there, right? For people to learn. And I think it's totally pick, you know, like you said, But a good data engineer can pick up that stuff pretty easy. But, you know, like you said, you're going to have to go.

15:03Erfan Hesami: If you don't have that stuff at work, you're going to have to go look for it on the side and learn it yourself, probably. Otherwise, it's going to be hard to pick up. Exactly. Like, you can be a data engineer at the moment. There is no, you're not working on directly on AI engineer project. But think about what repetitive tasks you can automate using AI for yourself to make yourself more efficient. It can be anything. Even, for example, from my side, I give reports at the end of the week to my manager. So I try to kind of find a way how I can make it more efficient. With less time, I can kind of use AI to automate it.

15:46or communicating with different stakeholders with different background. I think that's something we can, like, instead of always going to chat, and say, this is, here is general manager of data products, how I communicate. He is not technical, for example, but he knows this. So what if you can create an agent for this, for different people? And then, you know, something that I can say data engineers can find a way, to do it and gradually it helps them to learn more so they have sent to see the limitation it helps them all right now whether i need to use open source why i need to use open source or i use i don't know uh the close one what can i run it on my laptop can i write it on close so it helps them to think more broadly in terms of hardware in terms of it like it comes when you

16:44Erfan Hesami: going to start yeah yeah do you think that the modern data platforms today that most people are using have they caught up to this agentic world we live in do you think there's where have you seen problems and what i mean by that is right you were an analyst you worked in analytics engineering etc you've got this explosion of the ability of agents to write sql and just you know human text give me this query you know I want to know something about this customer or tell me about the sales this month and this sort of like make the agent go do it does that sort of stuff give you heartburn do you think there's more room for improvement there are you ready to say like yeah we're we're good to let the agents take over this sort of stuff or does the human have to still be in the loop where do you think we're at I think they're catching up the primarily use It's small, I think for human, but they kind of catching up.

17:42Uh, yeah, as you said, initially like data platforms and create dashboards, someone look at dashboards, creating reports, creating, I don't know, queries. Um, but now, now we entered to a new era. I think also this modern data platform is start catching up, but now

18:07Erfan Hesami: Now what is important is more about the context of data, how their lineage, their metadata, what does this mean in terms of business. And so these are the things I think this modern data platform try to, I think, improve. And at some point they're doing that database, for example, one of them, when they introduced use Unity catalogs and then now, Jamie, lots of great things. I think Databricks are working on that. Some of them are maybe in private mode, but I think if we really need to like use AI and benefit from it, we need more automation and the systems, like each components need to connect at some point with each other because then it's going to be hard like for example yes i think there's something called zero ops that data is developed in that one they have zero ops or zero devops sorry if i i don't remember yeah this is in private we always check the pipeline see if it's fair then go and check the lineage data it needs to be fixed It will create a PR for it and so on.

19:32It means we are getting closer to that based on what I see Databricks is doing. But can we do it ourselves if we don't use Databricks? I mean, it depends how much we automated our data, how much data agents, how much context it has to think about. But at the end of the day, even still, I think today, I can say I won't let them do all the work, especially if the work is dangerous and it's very harmful to undo it. I will definitely consider review before.

20:17Erfan Hesami: this is a good segue into the next topic because it gives me a lot of heartburn to even especially when you're making agentic tools that interact like you said with data and they interact with infrastructure and talk to us about like what people don't understand about how importance governance is when it comes to agents because this stuff like keeps me up at night when on building agents to do a thing, right? Because it's like, okay, what does this, what tables does it have access to? What are the, you got to be doubly sure, like what kind of permissions does it have? Read only. It's just like, it brings a whole nother layer and that's just like the data piece.

20:58Erfan Hesami: And I mean, I don't, agents can do anything if you let them do it. So governance is, it's a big deal now. Exactly. I'm surprised that I see some companies try to justify the cost of the investment on data governance, at least in Australia, when I attended recently an AI and data conference. But yeah, in terms of access, you can grant access to a person. Don't check what they access to, what they can do. but that person can think about all right if i have this access probably i might not use it because it's too much access and i pay track but as you said ai agents they don't care they if they have that i said they go and do whatever they want i mean and continuously can make mistakes and it can be very harmful that's why it's very important what it's not that you create credentials or service account for them and give it to them.

22:08So we need to be very cautious about what we give access to them. And also the access is one thing and what they can use, how they use it is another thing like that. Yeah, of course they can use data, but how much they can use it or how, what decision they can make out of it. That's also important, which I think, again, as part of governance or the business people or the company can decide. And I always need to monitor this. Agents can make mistakes. In traditional governance, you might put some policies as a document and review quarterly if you're very active work once a year. But now, you never know.

22:59Yeah. what agents are doing so we need to have the history of what we keep monitoring them as they

23:08Erfan Hesami: evolve and yeah i think that uh all that means is like there's job security for us right because there's so much i mean we're just at the inflection point of companies onboarding these tools and like we're talking about like we're talking about governance and that's not even the building the tools themselves that's just like governance of what they can do and then like you're talking about observability and monitoring how do we know what's going on and what's happening and somebody's gonna have to do that stuff do you think that people care about cost yet because all the stuff we're talking about whether it's maybe we're talking about tokens at the developer level but you know maybe we have a model serving endpoint running in databricks and we're paying for that computer tokens or have you seen are people is it still early enough where people don't really care they just want the technology they don't care what it costs or do you think that people are starting to think about cost as far as like oh maybe open model maybe i shouldn't be paying anthropic so much yeah i think recently in last two three months there were a lot of emphasizes on controlling costs as like some big companies lay a lot of people spend heavily on ai and they didn't get the result and they again start hiring people and at some point they understand okay maybe they don't need to follow each cutting edge model that all these companies release maybe they can use one of the compatible AI and they just, the models and they just have a compute for inference and so on so yeah I think it's a bit, there are a lot of discussion around that and people changing the way they are doing.

25:02The cost is very important. They are not cheap. Now, cost optimization, especially when working with AI models.

25:13Erfan Hesami: I feel like a lot of people in a lot of places have just been sloppy because it's been like the wild west. It's just like nobody cares. Use all the tokens. Let's just get on board. We want to do agentic stuff. and now it's like sort of like a reality setting in that like oh this isn't free like you know there's such thing as open source models there's ways to cut token costs for developers with tools like caveman or whatever right it's just it's funny to see that switch from like we're all in to like okay like a healthy bit of skepticism of like oh we probably need to take this seriously like any other tool you know what i mean yeah yeah one one other thing i wanted to ask you about because I've struggled with this myself.

25:57Erfan Hesami: Like recently I was working on a project that, so you have like, you've been working in data a long time, right? So like we have these classic systems like Apache Airflow, right? I was working on Airflow recently. It's the, you know, everybody uses Airflow. And it's like now we have these sort of requirements where from the business, driven by the business usually, right? Because they want something or, you know, product, whatever. and I had to put you know basically inside what we would think of as like a very deterministic very classic data pipeline that we've all worked on for a decade and now all of a sudden we have this like agentic thing that we're supposed to like stick in there and then like I don't know if these old tools have caught up of course Airflow's released you know AI provider stuff so these tools are catching up that's not what i'm saying but have you experienced this sort of like weird world where like it's almost these two worlds meeting sometimes and it's kind of it's hard to fit them together sometimes he's very deterministic old school data pipeline whatever and you've got this new agentic non-deterministic thing exactly like in terms of data pipelines like I think we always need to ask this question data processing usually has some input and output usually one answer so if we know that the input that it's receiving one answer is a join I don't know it's a data quality so this kind of stuff I don't think we need to like use it we can just use the same fundamentals or same logic for it.

27:44But what AI can help, and I've been exploring here, is reasoning around the outputs that it generates or the logs that it generates. If, for example, at some point it failed, generates some logs, yes, maybe an agent can help to understand the problem. Summary for me, like suggest potential areas that I can look at, but still I prefer to look at this suggestion and then decide before make any changes before like, especially in production. So I think we don't have to use AI for everything, especially as a data engineer, But anything that's not, AI should not generate data for us. We should use that data, the reason, reasoning behind the data, everything around that.

28:49But the output always should be deterministic for data pipelines.

28:55Erfan Hesami: What's your take on, it's kind of in the same vein, what's your take on like agentic, like code being, like your data pipeline code being all the PRs, everything. What is your take? Where did you draw the line? Or do you have a line of do you care if people just say, hey, Claude, write this data pipeline and Spark and Snowflake, whatever, and how much review do you think should be done? Do you think we should be doing just handing off the pipeline work to the agents because they can write code just as good and they can write tests and what's your take? and how do you approach that? Yeah, very good question.

29:39I've seen people have agents for generating codes, reviewing codes, I don't know, give feedback and then... So it's an interesting word, but yeah, generating code is getting cheaper and cheaper. Previously, when you think about writing codes, you think about whether I need to add this framework, this layer, this component, or this component. But now, When you ask AI, it can generate loss of quotes for just simple things. So that discipline, that monitoring, that standard is important because it can easily make things complex.

30:23Erfan Hesami: Yeah. It's working for sure. You can review it if you have time to review a lot of things or you are willing to do it, of course. But I think even if we want to use coding agents, we need to exactly define what we want in terms of the architecture, in terms of the system that we are designing. And even if it generates or decides the design for us, ask the assumption, validate the assumption on what base it generates this, for example, how they relate it. Or ask it to give a high level of architecture of its design before progressing further. And then you're kind of lost somewhere that you don't know even if something broken, how are you going to fix it?

31:18even the agent is a lot when it's generated a lot of course i don't think agencies easily can like find it effectively or fix it effectively so i think it's good to have uh fresh the fundamentals always uh and uh yeah it's it's dangerous right i mean because i feel like ai

31:47Erfan Hesami: especially for developers will just make them more of who they already were and you're right it'll just take you down a road over engineering all that's tech debt right every single people forget that i feel like in the rush of agentic coding which i do and it's fine and it works but people can't forget that every line of code that's generated and goes to production that's like a break point that's tech debt that has to be monitored and maintained and it's when when code becomes cheap you're right then it's like at some point it's going to catch up with you right if you're just like you said it's complex pipelines or complex infrastructure whatever if you're just kind of like blind like you're right because it's going to generate more code than anyone can possibly review you better know what you're doing because if you're just shipping it shipping it shipping it you're right at some point the complexity of something breaks something bad happens at production who knows right and and it's still on the human to like go fix the problem but will they be able to you know um it's a strange world and i feel like i don't know if it's you know i don't i feel like humans just like humans generally i don't know if we i don't know if most humans care you know our organizations i don't know if they think about that i mean they might say they do but i don't know they might just jump on the bandwagon and deal with the outfall later possibly you know what i mean exactly maybe caring about the results now i'm not thinking about future but 100 percent agree and interestingly it was just yesterday though uh sorry just at this but on reddit in on ai agent one of these channels someone said i'm non-technical.

33:36I just created an application for a company and it was working great until it stopped. And then now I don't know anything about programming. And where should I start? Should I learn Python? Should I learn SQL? I'll try to find it in the screenshot. And I was thinking, okay. So.

33:57Erfan Hesami: That's what happens. Yeah, this is, because I think one of the most important, I mean because we can't put the AI back in the box. It's here. It's here to stay. People are going to use it. I mean, it's just the way it is. And it makes me think about, if we think about, and I'm curious your view on this, if we think about data engineering, like even five years from today, like I wonder what a data engineer's job description will look at five years from now. And part of me thinks it is like what you're talking about, like a lot of systems architecture, maybe a lot of understanding. It's one thing to understand like a little function, but to understand like how things are deployed, like what VPCs are, you know, what a container is, just like these systems, the system architecture way, this high level understanding of like what something actually looks like from beginning to end in its totality is different from saying, like you said, it's like, Hey, Claude, make me this app.

34:56Erfan Hesami: Like, that's not the same thing. Those are two different. It'll do all that stuff. but then there's this whole other world, right? This sort of system, this architecture stuff that's baked into these assumptions. I'm curious what you think data engineering will look like if you had to guess in five years. It's really hard to predict five years even, I don't know, based on what the engineer has been the latest design engineer in one year. We don't know. But what I can see is some part of our business normal duties responsibilities will be the automated some basics some of them already automated like creating documentation basic tests the scripts uh but um another way to ask the same question would be like if someone came to you say you're say you had a younger brother and he came to you and he's like i want to be a data engineer would you still tell him to like what would you tell them to learn would you say learn sequel or would you be like yeah i probably wouldn't start there maybe i don't know i don't i don't know the answer either would you say maybe spark or is it like i mean i think i would suggest look whatever these ai models at least for now it's great if you want to just like test for the word judgment that a lot of you nowadays use want to judge the outcome so how you want to do it you need to know the fundamentals you need to know like how things works but yeah it can work you can design you can like build pipelines automate but how you want to say in long term it's scalable like in terms of finops like in terms of cost if you're working for a company.

36:50So how you can justify these things? Will that AI, I mean, at this stage, can do it for you or you need to decide the system, the design of the system? So that's why I emphasize on like any other, like you or Doris or someone, on fundamentals, definitely. and not fundamentals of data engineering. I mean, besides that, learn as much as you can about AI, AI literacy, I can say, how this system works, how this model works, and then how you can use it for your work to be more efficient and not replacing your creativity, your judgment, but in a way that can help you like move things faster in a most effective way.

37:50I think that's what I can, but learning, you pointed out SQL, SQL, I think, yes, AI can create SQL and you can run it. You can see the result and the result can look all right.

38:02Erfan Hesami: So I have noticed, I don't know, I'm just curious your thoughts on this. I've noticed that. So AI is like the great level somewhat is a great leveler, right? We, yes, we have to learn fundamentals, everything, but clearly like it can upskill people, juniors to do things they didn't may have not been able to do before. They were at more difficulty with. So it's somewhat like leveled the playing field. Have you found that non-technical skills have become more important? And what skills do you, when I say non-technical, do you have, Have you noticed like, oh, maybe being able to talk to like the product group is even more important than it was before because, you know, code is cheaper now.

38:45Erfan Hesami: So maybe these non-technical skills are like super important in a job market where it's tough, right? Yeah, really good question. Now that things moving fast, there will be more opportunities to work around business. what does that mean is we need to have the business skill which is communication self-order management like even for if I want to back to like governance there are still some tribal knowledge in people's head which I think at some point we need to communicate with people and capture them and for their process now that we can do more automation if you want it's really important how we communicate with people people this will become very important and around this topic i had a live session with colette malteni who was talking about the human aspect of technical wars and she emphasizes on like human being a human empathy and this kind of stuff I think is really important having good relationship building a relationship and at some point if you work in corporate some politics I can say it's important work dynamic in general it's now become more important yeah I've definitely seen that especially as this code's gotten cheaper you need to be able to talk to the business stakeholders.

40:26Erfan Hesami: You need to really understand, especially if, because, you know, projects are, they go a lot quicker, right? Before where maybe a project, the main blocker was like the code generation, all this stuff, and it's clearly not the case anymore. So the main blockers are humans usually, right? It's like getting, understanding those business requirements well, understanding what the needs are, and then combining that with like your technical knowledge of the systems. I mean, that's fundamentally what a lot of engineers do. And I think that's even more important now, even when people are talking to Claude about doing something right.

41:00Erfan Hesami: If you're not able to really communicate well, the environment you're working in, your expectations, all this, like you said, this tribal knowledge you had in your head of, oh, I work at this company, and this is the type of architecture we have. These are the systems that our pipelines run on, right like all that if you don't have that good communication baseline it's going to be much more difficult because a bigger part of the job is going to be talking to people or to clod i guess if you assume clod is a person i wanted to um also ask you about your sub stack how long have you been running your sub stack now did you said i catch you you said two years october 2024 It was my first post.

41:47And in eight months, I got Best Seller Patch, which I did, honestly. And yeah, initially I was working on very core fundamental data engineering principles inspired by good books like Jory's Fundamentals book and so on. And also I started sharing, create a series. It was a long series, sharing all the experience I gained while interviewing for data engineering role because of the transition with people. And then all of a sudden it just gets very popular and people start paying attention and supporting a space subscriber, which I appreciate. and just recently last recent year tried to also upskill myself more in AI last two years and just share what I learned every now and then I attend some conferences try to bring what I learned also do some live self-sex session with like-minded people

43:07Erfan Hesami: Do you think, what do you, as I'm curious as a fellow sub-stacker, do you, at least in your mind, do you think, do you know why you were so successful on sub-stack? Do you think it was like consistency? Do you think it was just like the passion for what you do coming through? Have you thought about it at all? good question I didn't expect to grow this much in the short that I had in my mind but I think the passion definitely if you like what you do I think you'll get the outcome and consistency for sure every week at least I think I've been posting almost every week if not more to post but at least one post every week since 2024 for so that's impressive so i think you also do a lot of the soft stackers that are successful are same they spend a lot of time that you can sacrifice a lot but i think what you get from is very enjoyable uh in terms of the people the support you receive that's awesome to hear about your sub stack doing well i mean i i think some people don't realize how much work goes into to doing a Substack, well, like you said, dedication and showing up every week.

44:26Erfan Hesami: I think a lot of it, I agree, is passion. It's hard not to, you know, you look and compare. I don't know for you, but I'm always watching other people on Substack and comparing myself and seeing what they do. And I'm tempted to use AI more, but I've kind of stayed away from that, I feel like. At least personally, I like to, I just like to, like you said, have my passion come through to talk about something that I've actually used, like you said, like going through your data engineering interviews or whatever, the things you're learning, I find the same thing. People kind of resonate with sort of real content in that sense.

45:00Erfan Hesami: Especially with AI coming in, I think that having that honesty and that passion is probably what's going to make us keep growing in the future, I imagine, hopefully, anyways. Great. It's just about our learning. The first thing, like we can use AI, I mean, sub-stackers can use AI to generate content but at some point for people I think like you and me if you want to really learn stuff we need to study and then if we can explain it simply then it means we understand if we cannot and let just leave it to AI to do it and just you know so I think we need to think about us our learning as if you want to use AI for writing.

45:47Erfan Hesami: Sure. Yeah, I got a couple of fun rapid-fire questions for you. Like, you just have to give me answers. So Databricks or Snowflake, what do you pick? Databricks, for sure. Me too. You never know. If you had to pick, like, Cloud Code or Copilot. Are you a Cloud Code person or Copilot? Cloud Code person. Cloud Code person. That's good. Are you like a batch guy or streaming guy? Do you like batch detail better? Do you like streaming? Depends on the use case. But most of the works have been working with batch or near real time, not very streaming. But, yeah, I think it depends on the use case. That's awesome.

46:35Erfan Hesami: Well, thanks for joining me today on the podcast. I'll leave a link in the notes to all your sub stack, your LinkedIn. So if anybody's watching, just go to the show notes, wherever you're watching, consuming this, listening to it, and you'll have links to Pipeline Insights, Substack, and everything. I really appreciate talking to you today. It was super fun. Thank you so much.