AI Is a Mirror: MLOps, LLMOps & the Future of AI Engineering | Maria Vechtomova

0:03Data Engineering Central Podcast Host: Welcome to the Data Engineering Central Podcast. I'm Dan Vecht, your host. Today we have Maria with us. How's it going? Did I pronounce your name right?

0:10Maria Vechtomova: Yes. Well, Maria is the easy part. My surname is much harder.

0:15Data Engineering Central Podcast Host: Yeah.

0:16Maria Vechtomova: And there's Marks. Yeah.

0:18Data Engineering Central Podcast Host: That's funny. He's always there. Can you introduce if somebody is new to you and has no idea who you are on LinkedIn or your book or something? Can you just like give us a high level, like 10 ,000 foot view who you are?

0:29Maria Vechtomova: what you do what you've done yeah yeah of course so i'm maria my surname is which tomova uh well if you are russian speaking it would be easy for you to pronounce it's not then probably hard because my surname can be pronounced in many different ways i learned um i've been in data in ai for a very long time i think for 12 and a half years almost 13 years and I started as a data analyst. I think yourself as well, right? Because data science didn't really exist back then, so it was just analytics. And then I moved into data science and started using Python instead of R. And then we started deploying our first models to production behind APIs.

1:13Maria Vechtomova: And it was unknown back then. We were just starting doing all of that. It was super fun. And then I got into MLOps because you need to have these endpoints reliable, keep them reliable. And you also want to scale it to other use cases and also teach other data scientists how to do similar things. So we basically started MLOps platform. We called it Model Factory. And we had our version of MLflow kind of. We reinvented MLflow. It didn't exist back then. But we had our own experiment tracking using SQL Alchemy and we were storing it all in Teradata. Wow.

1:56Data Engineering Central Podcast Host: Yeah.

1:56Maria Vechtomova: And we had our, yes. And we had our little dashboard and the whole deployment was also through CICD pipelines and we had Kubernetes workflows, like, you know, it was really pretty cool. And, you know, if you look, it was in 2016, 17, like who was doing it back then? No one. No one. Exactly. Exactly. So I have been doing it for a while. And well, in the last years, I use Databricks a lot. I would still say I have more experience in Kubernetes than in Databricks. If I look at years specifically, but Databricks is something that I just happen to use a lot. And you see that everyone has Databricks, I think, today.

2:39Maria Vechtomova: but people don't use it in the right way I find and that's how I got into teaching people how to

2:45Data Engineering Central Podcast Host: do it right and writing a book about it as well that's awesome what's the book yeah it's called

2:52Maria Vechtomova: MLops with Databricks and it focuses on two types of use cases regular ML and more like agentic workflow LLM based workflows and how to deploy them on Databricks and that's what we also do with our clients really. So we also have a consulting company now started about a year ago. So moved from roles in large corporates and more towards doing our own thing. It's going very well. It's very busy. Sounds a little scary.

3:24Data Engineering Central Podcast Host: Is it scary in the internet really?

3:27Maria Vechtomova: Well, yeah. I mean, you have to be a marketer, right? You have to like, you can't just be ML.

3:33Data Engineering Central Podcast Host: Now you have to be like advertiser.

3:35Maria Vechtomova: Well, I guess I've been doing it for a while. So we were doing trainings for already a couple of years on Maven on the same topic, really, MLOps with Databricks, LLM Ops with Databricks. But, of course, it's very different selling to individuals than selling to enterprises. It's quite a different skill. But I guess through thought leadership, you get your champions and that helps are you a databricks champion and well i'm not a champion uh i'm an mvp but i'm not

4:12Data Engineering Central Podcast Host: a champion yeah i'm an mvp too recent just recently like in august yeah i saw that congratulations yes um it's fun group but yeah those champions must be the real deal like it must be like the cream of the crop or something you know yeah i guess i think you have

4:31Maria Vechtomova: to be a partner to become a champion and there are some people who write about the differences about champion and MVP but I never had an ambition to become either MVP or a champion but I got into it somehow

4:48Data Engineering Central Podcast Host: I honestly like I was surprised I didn't know I was surprised about the MVP thing when I got I had no idea that someone nominated me or whatever because a lot of my content is I write about Databricks a lot I ever use Databricks a lot. And sometimes I'm not very nice. Yeah, me neither. So I was kind of surprised. I go, does this mean I have to be nice now? You can kick me out. Can I go back to your early days as a data analyst? Because I didn't study computer science. I got into tech as data analyst. Did you learn programming super young or something? Go to school? How did you end up being a data analyst way back in the day?

5:26Maria Vechtomova: well I studied economics and econometrics and we used R a lot so it was you know a lot of macroeconomic modeling things like that and I didn't kind of like economics I think never really liked it I got into it through Olympiates and yeah I just went with the flow I guess but then I just knew I didn't want to be an economist but I want to do something with data so I really like that our part so data analyst was something that was as closest to it as possible

6:05Data Engineering Central Podcast Host: and that's how I got into it yeah kind of the same thing got into data like SQL did the data analyst thing that's kind of when at least I'm guessing we're circa the same time you know like business intelligence got kind of large yes yeah yeah time data big data big data high yes yeah yeah it was uh it almost like feels sad today like looking at where we're at and then looking back on the past i don't know it's it's just like it's a different world nostalgia yeah yeah yeah i just don't want to go do something back then we actually had to like go vim into some cessation to some server and like yeah yeah you don't have to suffer anymore that much which is i don't know is that good or bad depends i don't know i think you learn a lot through suffering i would agree with you but that's probably what every generation programmers say you know like the assembly the c guys are probably like yeah yeah like what you're just doing python you know it's not real suffering correct yeah that is interesting though because i eventually got into data engineering and i kind of remember like those wild west days of like machine learnings getting popular but like you said the community the are the tools didn't really keep up so it was like i specifically remember one job i worked at a startup and it was like a dozen data scientists and they like plopped me into the middle of them it's like one data engineer like I'm going to fix everything.

7:39Data Engineering Central Podcast Host: And it was just what you described. It was like the Wild West. There was no like, people just like wrote whatever code they wanted. Didn't commit it anywhere. It's just like, oh, I just run this script that calls five other scripts that run five other tools on some server that I made myself. It's all good. You're just like.

8:00Maria Vechtomova: Yeah. Yeah. Well, we had a server actually, and we had a key to the server. It's just standing in the room. yeah it was our first automation it wasn't we didn't have clouds back then you know yeah it's

8:13Data Engineering Central Podcast Host: uh we've come a long ways um yes and i still feel like it's early days because i've been using databricks now for i don't know a long time and it's still like as ml flow i remember when they first were like started pushing that project and it came out but it was still kind of like I still feel like you're right. People typically, they don't think about running experiments. They don't realize that they can log things as assets. And I still feel like there's catch-up that needs to be done.

8:44Maria Vechtomova: Oh, yeah, for sure.

8:46Data Engineering Central Podcast Host: Do you see that? I mean, since you're consulting, do you see that still people just have no idea what they can actually do?

8:52Maria Vechtomova: Oh, yeah. Yeah, I see. I see a lot. So, I mean, it's basically AI is a mirror. You can write a lot of code fast with AI. whether it's going to be good code. Yeah, it depends on how skillful you yourself are, mostly. I still believe that. And, well, I just see a lot of spaghetti code still, a lot of code that is not reproducible, really, even though there are CICD pipelines. If you deploy something, you actually need to know what data created that artifact, right? right? What code version, where exactly can they find that? And yeah, you need to know like where that endpoint was called from as well, right?

9:39Maria Vechtomova: Like all these things you need to know. And very often you can't find it.

9:45Data Engineering Central Podcast Host: I've noticed this. It's still a problem. I still feel like, especially we can talk like, if I think about like a feature store, for example, it's still it's still kind of the wild west of oh yeah you know who's in control of what features when how do you depreciate features can you can you not because right they're referenced and like again like you said how many times have i seen a notebook that trained a model some artifact that they just wrote to s3 some weird spot called it like 1 2 3 4 5 version 6 And it's like, you have no idea what features we use to train this model. Yeah, how we got there.

10:25Data Engineering Central Podcast Host: It's still a problem today.

10:27Maria Vechtomova: Yeah. Or also deployments that are ripped out of each other. So you have the whole pipeline for training the model, and then there is a whole other pipeline that does the deployment and treats it as a black box. and then something goes wrong on the front end side that is using that endpoint and the whole deployment cycle, you still have no idea how we got there because it's a whole other team that did that. I see that a lot still as well.

10:57Data Engineering Central Podcast Host: Yeah, every once in a while I'll read a blog article that's talking about some machine learning thing and they talk about how they're tweaking features and rolling back models and blue-green deployments, testing two models at once. and i'm like man if you don't know what goes behind the scenes you're just like i'm when i read some of those i'm just like i'm almost questioned if it's real because i've seen like what most places look like and i'm like you're either the smartest people in the world or you're

11:24Maria Vechtomova: lying because i don't know well or it was just in one place that you could implement it and maintain it well i mean we had really nice deployments ourselves but how sustainable is that because sometimes you leave a company and then the whole team leaves the company as well and then someone else has to maintain it and all the knowledge is lost and then it's wild west again so

11:50Data Engineering Central Podcast Host: do you think that MLOps is changing with the Gentic world we live in so like I think about Databricks offering you know LLMs, paper token you can just like call a can model like in a second

12:05Maria Vechtomova: or you know

12:08Data Engineering Central Podcast Host: how is Do you think it's, have you thought about that? Have you run into any problems with like, I don't know, do people, do we need sort of, do you need ML apps with LLMs, like versions? What model are we doing? Props?

12:23Maria Vechtomova: Oh yeah. Well, I actually see like LLM workflows, you know, much harder problems really than if you look at machine learning models. There are more, there is more non-determinism that comes into play. So I think observability monitoring part is the worst part of the whole ML of stack. People just ignore it because it's too hard a thing to do. And for ML, it at least is clear what you're looking for. If it's a classification problem, you look for one score or whatever you're looking for. For regression task, you look at MSE or whatever matters for your use case. and you could at least do some kind of estimation there are ways to do an estimation of error and that's what you can see okay the performance is probably deteriorating even though you don't have the ground truth yet so there are mechanisms for that and you could explain that through feature drifts for example or i don't know sharp values like there are different ways you could see why things are going as they're going.

13:34Maria Vechtomova: For LLMs, you don't know what you're looking for, just to start with. So you had a set of examples and you decided, well, this is a great metric that we need to look for. And then a new example comes in and it looks perfect according to the metrics you have identified, but you see it's clearly wrong because you have not thought about that other metric that now you see clearly you need to you know account for so the whole evaluation is non-germanistic what you're evaluating for changes over time for you know a lot of use cases really because of these

14:09Data Engineering Central Podcast Host: edge cases that come in that you just didn't you know see before but also how do you even know

14:17Maria Vechtomova: the ground truth um the ground truth in regular ml is pretty you know static clear what the ground through this for most of the cases. But for LLM, it's very vague because also people don't agree with each other. And that's a very known problem in NLP applications that I even saw in 2018. We had a chat classification problem that we needed to route the chats to the right agents, a call classification problem. and we had predefined labels that were defined by some business people. And we saw that in some labels, people were disagreeing with each other most of the times. And the overall agreement was like 75%.

15:05Maria Vechtomova: So how can agents perform when the labels are vague, just to start with? And I think that's what people underestimate quite a lot. spending time into defining problem very well defining the labels very well and making this you know this process well defined and you do need mlops practices to do to do this correctly it's just it adds extra steps that you have never had before but in terms of you know regular components that you have in mlops and llmops they're pretty much the same you need something for orchestration, you need some for data processing, the context engineering, you need something for model serving or LLM serving, you need something for capturing your calls, like endpoint calls, like what was the input, what was an output, you need some pipeline to do evaluation in production, you want some kind of alerting on that, like all of that is very well known and i think that's also what really accelerated uh the speed of current developments right because it always has been there well at least for last couple of years definitely it has been there it just got a bit more complicated and the knowledge is i find still

16:28Data Engineering Central Podcast Host: lacking in people yeah i feel like i've been working on a couple different agentic tools and some of the things that have caught me off guard or i guess especially in like the l think about llm ops is a lot of times now we're not just talking about like just an endpoint we are probably but now you have like all these tool calls right that are yes custom or you know maybe somebody installed it from somebody else or they you know if you're on databricks who knows maybe you wired up your own spark connect and then go into certain you have these tool calls which add just like these layers of complexity and like how do you yes like how do you separately do you treat them are they together are they different and then also like the adversarial approach of like well yes a lot of times some sort of front end is kind of like open to whatever and you do you need to know what's going on because someone yeah rate limiting who knows trying to be nosy do bad things it's just like which typically i would say in the past if i think about machine learning 10 years ago you didn't you weren't really like those you really were worrying about some of those things right it was more very defined to workflows and now it's

17:46Maria Vechtomova: we were naive who knows we were naive i think in a way or like the world wasn't you know that far in that way of thinking and back then yet but it doesn't mean that the problems didn't exist they definitely got worse today because as you correctly mentioned there are what also made it more complicated is all kinds of integrations that you need with other systems that you you know never saw before so there is like big governance issue and doing it correctly is really really hard um and um so you also mentioned traces i think well traces is how you could also capture these tool calls and lm calls and all the steps really within your applications and i find it funny because traces are you know it's not a new concept no and suddenly it becomes like lm con on LLM ops concept.

18:44Maria Vechtomova: But I mean, I actually tried using traces for regular ML and I think everyone should, by the way. I think it's brilliant. We always wanted that feature and it never existed. And now finally it does.

18:58Data Engineering Central Podcast Host: It just shows you how much people were actually paying attention to that stuff before, which means like not much.

19:03Maria Vechtomova: They were ignorant. Yes, indeed.

19:06Data Engineering Central Podcast Host: I actually, today I was working on like a chatbot that has some tool calls, whatever. And I actually, like a plug for Databricks, I wired up the app to ZeroBus, Databricks ZeroBus. People don't know what that is. Like kind of a new version of Kafka sort of. Like streaming endpoint and like obviously a lot of tracing, logging, AWS systems. So going into CloudWatch and stuff. But it was super nice to be able to stream what's happening, like questions, responses. Yeah. You know, time taking all that stuff, stream it right into a zero bus, straight into a Delta Lake. And then.

19:43Maria Vechtomova: Yeah. That's amazing. I love this feature.

19:46Data Engineering Central Podcast Host: It makes, you know, this, you know, alerting, it makes all sorts of stuff like easier, right? Like alerting what's going on.

19:53Maria Vechtomova: Yes. A hundred percent. And it's not just makes that easier. I think it makes platform controls easier. So previously everyone could just create, well, still they can, they can't really disable that. but yeah people will just go and create experiments in the workspaces right and how as a platform if you have many of these workspaces would you even ever know what's in those traces right and now if you provide as a platform the experiment in unity catalog which is basically like six tables right that you would see under a schema and you as a platform can define permissions on those tables, so only with that SPM that you grant to a team, that only then they can generate those traces, populate those traces in their catalog, then you can also monitor, are they really using it well?

20:47Maria Vechtomova: And I think that's very, very powerful, and that's what we are now pushing for as a platform implementation. I think it's just such a big change in observability.

21:00Data Engineering Central Podcast Host: yeah the other thing that kept me up at night like working on this sort of stuff is you mentioned it a couple times like the governance and permissions it's so i don't think people realize how helpful it is to have everything under one like say it's under one databricks environment got you in the catalog just especially when you have like some of the stuff i was working on were hooked up to tools like directly on top of data sources and just it gets you i don't know what it is but it just keeps me up at night thinking like i don't you know what kind of grants does this service principle have it can read only these columns it can't do anything else because it's again that's not a new problem but it's like maybe a different representation of an old problem like

21:44Maria Vechtomova: it got worse you're like it got so much worse i don't want to be the person with like where it

21:50Data Engineering Central Podcast Host: goes bad and you know somebody does something they're not supposed to yeah no exactly and

21:56Maria Vechtomova: well as i said i think everything is accelerated like the problems that we mentioned in our conversation they always existed but now everyone is paying attention to those finally so actually i really like that we are in this ai hype um because it makes it really better finally those problems that we were you know talking about forever they're being solved so you call that ai hype so i have to ask you are you generally pessimistic or optimistic about the future of

22:31Data Engineering Central Podcast Host: say you're a data engineer today somebody's watching this they just graduated they want to like are you like would you be happy for them would you be kind of like scared for them in this environment i mean everybody's laying off people ai i don't know yeah it's scary times

22:48Maria Vechtomova: I think very uncertain and scary times I would say but I am positive overall about that so I called it hype but I don't think it's really hype it's actually something that generates the value and is really changing the way we work but yeah right now everyone is just doing many things with AI that don't make sense right that's that that's the hype part about it so it's just everyone is experimenting with pretty much all kinds of stuff and about most of them i think well you know a regular ml would be much cheaper and better maybe or maybe you don't even need any ml or agency maybe you just need the pipeline so and you know it's like you get one hammer and you want to try to solve everything with that and i still see too much of that really do you personally you Do you have your favorite harnesses?

23:48Data Engineering Central Podcast Host: Are you like a Claude person? Do you still write hand code some? Do you just Claude everything? Do you have tools that are your favorite? Yeah.

23:56Maria Vechtomova: Well, I used to use Claude a lot. And then I decided to try Codex. And I'm not going back to Claude.

24:06Data Engineering Central Podcast Host: Okay. Why is that? Because I use Claude mostly. I don't think. I've tried Codex maybe once. It's been a while, though.

24:11Maria Vechtomova: Yeah, you should try again. Well, I did previously and it wasn't that impressed and I like Claude much better. But I do prefer now Codex over Claude. Well, I think it's more the way how it's more mostly about the models, but also partially about the harnesses around it. The way how it writes code or the way how it generates documentation. It's just so much easier to get it in the way that makes sense to me. and I don't get irritated about it's still not doing what I want it to do. So it's mostly that, I think. And it can be different for different people, right? So I just figured out, for me, it works for now the best, but it can change, of course.

24:58Maria Vechtomova: It's so easy to port out everything to another tool today. They are very much replaceable. So it took me, I don't know, half an hour to fully be fluent in codex. after being on cloud code. So I don't know.

25:13Data Engineering Central Podcast Host: Yeah, let's try it. I tried. I've been playing with Omnigent too a little bit, the Databricks open source. That's been pretty nice. Just being able to have like their poly feature where you can kind of hook up two different models, have them argue about stuff. Yeah, I think at the end of the day, I feel like AI just makes people more of who they already were. Whatever kind of code they wrote before, it pretty much just amplifies them to do more of what they already do. Yeah. it's an interesting interesting world

25:44Maria Vechtomova: yeah for sure

25:45Data Engineering Central Podcast Host: have you been I wanted to ask you just because I'm curious and I'm a creator myself how long have you been like creating content on LinkedIn and other places for a while

25:56Maria Vechtomova: let me think when did you start writing online I guess you know yeah I think three years three and a half years ago or so

26:05Data Engineering Central Podcast Host: so not too long then fairly

26:08Maria Vechtomova: no not too long

26:10Data Engineering Central Podcast Host: What made you do that?

26:13Maria Vechtomova: Well, I just saw a lot of content. I was just following people on LinkedIn mostly, and I disagreed with a lot of them. And I felt like there are too little people that write about things that actually know what they're talking about, and that prompted me to do that mostly. Yeah, I think there are a lot of great creators, by the way, like a lot of amazing people. and that I also got to know through LinkedIn. I also really like your sub stack, by the way. I even recommended some of your articles about testing with PySpark. I totally agree with the approach. So yeah, I think -

26:55Data Engineering Central Podcast Host: Is LinkedIn like your main, would you call that like your main platform that you focus on mostly?

27:00Maria Vechtomova: Yeah, I mostly am on LinkedIn and I try to focus on different platforms. I like I use YouTube, also on sub stack. we had a publication that we you know spend a lot of time on with Marvel SML Ops I don't do that anymore but yeah I don't know I need to figure out still like what works for me also now with with a consulting company and all these things I'm doing it's just not really feasible I think to dedicate that much time to it and LinkedIn it's really you know very easy for me today I used to spend hours and hours on writing posts in the past. I know like forever it took. But now I see everything as content really.

27:46Maria Vechtomova: Out any conversation I could easily make like five posts. So it just, you know, goes naturally to me already. Whether I do it or not, it all depends on how much free mind space I've got on a day. But posting is not that hard, I find.

28:07Data Engineering Central Podcast Host: did your book that you wrote open up a lot of doors so that like have you like been told come speak at our conference have you done conference stuff yet talks and stuff because of the book

28:18Maria Vechtomova: um well i've been speaking quite a lot at conferences generally speaking also before the book um so and i think i would i i do less of it now just because i'm so busy but I do some book signing events so we it just launched right the book just launched in August the end of August and it became available in the US mid of September so it's just really really recent and I did book signing and spoke at signals conference which is SRE conference it was in Berlin it was really nice and I will be at the World AI Tour in Amsterdam will be also signing book not speaking there and i will also go to la uh also do book signing there the world fancy fancy yeah indeed no i just happened to be around so i thought why not why not come by

29:15Data Engineering Central Podcast Host: would you write another book was it worth it do you think

29:21Maria Vechtomova: well it's a huge amount of work i think anyone who thinks i can write a book they definitely can but everyone vastly underestimates the effort that goes into writing a book i thought like come on i can write the book it's just you know how how much harder is it than writing a bunch of blogs right so i think it's good not to know how hard something is before you you know get into it and then it's too late to stop you know you have to you have to finish it was that was that interesting

29:50Data Engineering Central Podcast Host: with the like was it kind of hard having somebody like the book publisher they like come on what's next chapter or was it not that bad?

29:58Maria Vechtomova: No, I think I was the hardest judge on myself mostly. Yeah. So you just have this mental pressure. You need to finish this thing. But I do think, you know, a book like that just didn't exist. And I wanted it to be, you know, to be out there just to teach people how to do things properly on on databricks but it goes way you know beyond databricks it's just general practices on you know what mlops is about why it all matters and why do we need to do things certain way um and yeah i just i just haven't seen a book that does that before um

30:42Data Engineering Central Podcast Host: though i feel bad though like the databricks man they change like you probably were like midway through a chapter and they change the name of something yes yes oh my god yeah that that that

30:55Maria Vechtomova: i had so many times i really was so frustrated you don't want to know um yeah it was pretty bad but i think like mlops kind of froze in time in a way right because they data rig doesn't release new mlops feature like ml specific features has you know so mlops chapters that i wrote they they were pretty stable so the only thing you would change is that okay there is a new runtime new versions of libraries stuff like that and but i have really nice testing process around it so it's basically all with the i just bump libraries run all the bunch does it all notebooks run and it's all fine so um you know that was not that hard i also have uh like a nice tool to regenerate diagrams and stuff like that so that was also pretty nice so i made my my life much easier uh with that um but llmobs chapters oh my god i rewrote it four times so i would rewrite it again today uh it's uh exhaustive really exhausting to um to deal with that but yeah they change that stuff i mean because i've been playing with their ever since they started releasing ai features

32:11Data Engineering Central Podcast Host: there's Genetic stuff, Genie, all that stuff. Man, it just, it changes. You learn something, it changes. I might be working on something and then somebody's like, hey, that's not good. I'm like, what? Oh, they changed it. Dang it. You know, call it one thing, it's another. Man, it's not surprising.

32:26Maria Vechtomova: But it's not just Databricks, right? It's like the whole world. It doesn't matter what tool you're using. It's all changing so fast. And I'm sure it's going to take another couple of years before we converge. So I think we had like a great period of acquisition for ML Ops for a couple of years. Like everyone was acquired by someone else, right? So a bunch of tools disappeared or changed a lot. And so I think we are really stable now with ML features, generally speaking. But for LLM Ops, there are just so many different companies, you know, focusing on specific parts of the whole cycle. or something else yeah or reinventing the wheel the things that we already had forever you know now it just add a gigantic to it and it's now very cool yeah you see that a lot and of course i'm not sure like what of that gonna survive but the practices around it uh they will stay so and that's the part of the book that is very stable i think because practice is like common sense it doesn't really change, right?

33:40Maria Vechtomova: So what makes sense is that you need to start with evaluation very early. You need to have human in the loop. You need to have certain processes for monitoring. You need to have certain processes for, you know, tagging, all of that. It's just not going to change. None of that.

33:55Data Engineering Central Podcast Host: Do you feel like, do you ever have the classic existential crisis where you're like, I haven't written code by myself in like a couple weeks, I'm going to forget everything. Do you worry about that? I worry about that sometimes. So then I would still write some rust by hand just to make myself feel better. I can still code by myself. Or do you think it's more like what you talked about? As long as you've learned the fundamentals, you've obviously been around a while doing things. So you think that those are more important anyways at the end of the day rather than just like the generation of the code or whatever.

34:30Data Engineering Central Podcast Host: It's never that important anyways.

34:34Maria Vechtomova: Yeah. Yeah, well, I'm not that worried about it, honestly. I indeed think that certain practices, they just, you know, and understanding those practices is probably the most important part in it anyways. I do care about specific code quality. I think it comes from, you know, background of having to maintain things in production. And if something is broken, you actually need to understand why it broke. And I guess good luck figuring it out if you didn't write that code yourself.

35:08Data Engineering Central Podcast Host: Claude, fix it. Please help me.

35:11Maria Vechtomova: Yeah, exactly. So I don't know. I think if your team is maintaining a very high value application, that if it's down for one hour, it costs you tens of thousands of euros or whatever, well, you really do need to understand that code as a team. Maybe not every single line of code, but you need to... well, the code structure and the structure of your repository need to make sense so that everyone knows where specific logic is and where it could potentially have been broken. So I think it really helps.

35:48Data Engineering Central Podcast Host: Yeah, for sure.

35:50Maria Vechtomova: And what you see a lot today is that, well, a lot of code is indeed generated automatically. It's not per se a bad thing. the fact that a lot of code reviews also happen automatically also don't think it's a bad thing but what is important given those two things we need to have really robust testing something that we have been preaching forever right now we finally understand why it's so important which is a good thing again so we simplify our life but we also need to have this maybe I wrote a post about it on LinkedIn as well. You need to have like maybe bi-weekly or monthly sessions with the team to actually understand where the code is going.

36:36Maria Vechtomova: Because the world around you is changing as well. Other teams that you work with, they also adopt some other practices. And overall, the code may diverge so far from the idea because of AI-generated code that you need to, you know, have the sessions to bring it back to this common understanding. and then it also improves right because that common understanding it also gets written down so that ai doesn't make this mistake anymore so you basically um that internal team knowledge you

37:10Data Engineering Central Podcast Host: you know materialize it yeah it totally makes sense um what do you think would be say it was like can you come up with three or something they've got like a new you know engineer coming and they just graduated they want to get into machine learning etc data at large what skills do you think are valuable or what would you tell them to work on does that make sense what i'm asking because i think before we would have maybe i don't know obviously they would have had to spend a lot of time just like learning to code or python or something it's like well i don't know put that on the list anymore do you have because i'm assuming you see as like a consultant you probably see a lot of different teams and see like what is still missing i guess or what they still struggle

38:01Maria Vechtomova: with today common sense i would put common sense it's number one because i've seen so many applications it just don't make any sense uh like you have an application that has 100 requests per day and you have an end point instead of a batch process why why do you have that right so things like that and the idea is oh yeah well it's for the future maybe at some point uh i will need that you know because it's scalable this kind of uh arguments yeah i just hear it so many so often

38:39Data Engineering Central Podcast Host: yeah it's like very common i think is i think it's something common for engineers i've always seen that throughout my entire career there's always everywhere i've worked there's you always will find like a really smart engineer who's like probably the smartest person i've ever met at that point and they probably are right they're just amazing can write anything they're whatever they just know every language they're the best but even 10 years ago it didn't make them the best engineer on the team just because sometimes it made them the worst it could make them the worst because maybe they're hard to work with maybe they don't like talking to the business so they just ignore them and they're just caught up in their own all they care about is just writing the code that's it and it's like yeah i guess like it's good to write good code you should be good at what you do yes but at the same time what good is it if it's this that's all you care about it's totally disconnected from the business from the team yeah yeah for sure yeah so that that was another one

39:40Maria Vechtomova: so you said three things indeed so these are exactly two things that i wanted to mention so i think common sense to start with but also soft skills um that become increasingly important and business sense so these three things um i think would make any engineer much stronger and much more useful for you know any kind of use case yeah you mentioned it before when you talked about like

40:06Data Engineering Central Podcast Host: the batch and streaming thing and it's like being able to like talk to the product to marketing whoever to the cto maybe you know take these ideals from the business and right you have to have this business sense and we need to deliver something that works it's cost effective whatever and then you have like a lot of times you have like engineers technical team and you somehow have to they're never going to be aligned because like you said the engineers are going to be like well we need an api or an endpoint for that we want to do all this stuff we're going to do the new stuff because it's fun and it's like well but if we have to deliver it at cost on time and like under complexity like you said using common sense now maybe we shouldn't do that um that's like it sounds easy when you said out loud but i feel like it's like you said it seems like the most

40:52Maria Vechtomova: common thing missing is the common sense yeah yeah no i agree but well i do think actually it's very often coming from business interesting enough saying that they need something scalable and uh well then it becomes an engineering problem that's a requirement business requirement to make it scalable and no one really asks why sure yeah that's what i see totally get it um

41:19Data Engineering Central Podcast Host: what do you see the same thing happening with specifically with like llm ops do you feel So like, it's such a new, it's so new. Everything's so new. It's changing so fast. I even, the other day I wrote, I think I was writing a blog specifically about like Databricks model serving endpoints and like kind of your different options, paper token provisioned.

41:43Maria Vechtomova: Oh yeah.

41:44Data Engineering Central Podcast Host: Like governance and kind of like, it was impossible because like everything I thought I knew from like three months ago, like didn't apply, you know, like, so it's like AI gateway. Now it's like unity gateway. Oh yeah. Everything just changing quickly and the features. And it seems like a hard, hard area to like know what the best practice is maybe just because it's changing so fast.

42:09Maria Vechtomova: And I don't know if that, I don't know.

42:11Data Engineering Central Podcast Host: Like you said, there's so many companies that seem to be sprouting out of nowhere, like trying to like resolve these same problems, agenting everything. I don't know. It's a bunch of snake oil at the end of the day, right?

42:24Maria Vechtomova: some well yeah some of it i think it is but the like i think the problems get recognized correctly i think the biggest problem today is still how to make people work to people and agent work together efficiently through different teams and i don't think well many companies haven't figured it out just with humans just to start with right yeah it's it's really hard problem and now you want to amplify it with adding agents on top of that. And yeah, I don't know. I don't think this problem will ever really be fully solved, but at least you can add some transparency layer on top of that with some tools or with some practices.

43:09Data Engineering Central Podcast Host: For sure. I got to ask some questions that are more for me than the audience just because do you have any advice? For me, it's like a new Databricks MVP. Like you got anything I should or shouldn't do to be successful as a Databricks MVP.

43:25Maria Vechtomova: Am I a successful Databricks MVP? I don't know.

43:28Data Engineering Central Podcast Host: They didn't kick you out, right?

43:32Maria Vechtomova: I don't know. Yeah, well, I do quite a lot of things with Databricks. I guess I wrote a book. I guess maybe you can write a book if you're interested.

43:42Data Engineering Central Podcast Host: That probably buys you like a couple years worth of goodwill, right? A book. Yeah, well, I hope so.

43:50Maria Vechtomova: I do post things about Databricks still quite a lot because I just happen to work with Databricks daily. And I just see things that are useful for what we are doing. And it could be useful for other people as well. So when I see something like that, which was useful for me, I also just post about it. But yeah, I don't really have advice. But I do really like the people in the program. I think they're really amazing. Smart people.

44:20Data Engineering Central Podcast Host: welcome to the team do you if you crack the linkedin code obviously like everyone's complaining about linkedin and their horrible algorithm killing them you clearly have you've you've got it figured out i got secrets that you're not going to tell us well i don't have it figured out i don't

44:39Maria Vechtomova: have like hundreds of thousand followers as some people do uh how many i have 81 000 or something

44:47Data Engineering Central Podcast Host: like that yeah that's like yeah i'm pretty sure that counts as a lot it counts as a lot well not

44:53Maria Vechtomova: not not massively a lot like some people do but also my niche is very small right so i i write about very specific things and very few people focus on you know that specific subject that i do and i find that subject still most interesting and still missing in you know a lot of conversations and right now everyone is experimenting already production you know problems are coming to people's minds but it's still very rare i find so it's i think another year or two and everyone will be really focusing on the production side of things again so i keep preaching about it obviously um yeah my my goal has never been you know to grow really big audience i just wanted to share what I'm doing with people.

45:44Maria Vechtomova: Your passion

45:45Data Engineering Central Podcast Host: comes through for sure. You know what you're talking about. Like you mentioned, there's a lot of people that don't. But clearly, you have experience and your passion comes through. I think that usually counts a lot in content. At least I've found that it seems to resonate with people better if you actually have done something and care about it or struggled with something. Usually someone else has too.

46:05Maria Vechtomova: Yeah. I agree with that. But I think you know it's in the end those two types of conversations you want to have and that's how i also found a bunch of people that i really respect and that clearly understand what they're doing as well and it just creates community for yourself with the same similar minded people

46:28Data Engineering Central Podcast Host: sure before i run out of time today where can people so just i'll leave links to everything in the show notes of wherever people are consuming this what do you want them to know about your book obviously we talked about that they can get it on amazon probably i'm guessing yeah i think so i

46:47Maria Vechtomova: amazon depending on where people live but also local stores even target was selling my books

46:53Data Engineering Central Podcast Host: oh that's a big deal that's cool target and you said you have courses on maven yes you said yeah

46:59Maria Vechtomova: we we run courses on maven i'm not sure when this episode will come out but from 12th of october until the end of November, we have last cohort of our LLMOPS course. And that's something that we are not going to do anymore. We still run actually custom cohorts. So what we do, yeah, the reason why we're not going to do that because just don't have that much time anymore for that mainly, but also because I find it boring to teach the same thing too many times, exactly the same thing. Well, it's never exactly the same, though, because, you know, the features in Databricks change a lot. So every time is really quite a lot of new content.

47:44Maria Vechtomova: But still, around the same topic. So I'm shifting usually. So we had the MelOps course, LLMOps course. We will be launching agentic coding, agentic engineering, like harness first type of approach in January. but yeah we do a lot of this corporate training which is customized towards the setup that people have so we teach databricks for this maven course but within the organizations if they got other tools we incorporate that into the trainings as well and also the specific setup of the platform that people got that's usually something that we combine with the consulting that we do got it and

48:25Data Engineering Central Podcast Host: And you're obviously on LinkedIn. I'll leave links to that. Very popular. You have a big following. I'll give you the stamp of approval. Giant following. And what was the name of it? If people want to look up the consulting, what was the name of it?

48:37Maria Vechtomova: It's called Cauchy. People sometimes call it Couchy. Well, Cauchy is actually a French mathematician. You may have heard about him. Yeah, I don't know if math, surprisingly. Yeah, well, he's very famous. I think one of the most famous mathematicians. And he also invented gradient descent. Well, he basically came up with the foundations of gradient descent. And without that, there would be no modern AI. So we thought it's a great name. And it was also available. The domain was still available, most importantly.

49:13Data Engineering Central Podcast Host: That is true. Yes. That's the most important.

49:16Maria Vechtomova: Yes. And we also got a very cute logo, which is a keyhole shape, which is also used to prove another Kausha theorem that has nothing to do with AI, but I just thought it's a fun thing.

49:27Data Engineering Central Podcast Host: That's awesome. Yeah, I'll leave links to all that in the show nights for this when it releases. But I appreciate you joining me today. It's super fun to talk. Hopefully we'll do it again. And I'm glad I did join you as a Databricks MVP. I love following your stuff. You obviously know what you're talking about. It's so fun to follow what you're posting on LinkedIn. So I appreciate you joining me today.

49:47Maria Vechtomova: Thanks. I appreciate you as well. Thank you for inviting me.