Building Modern Data Platforms Without Legacy Bottlenecks ft Andrew Jones

0:00You're trying to build this new world while you're still having to support the old world. The Data Engineering Show is brought to you by Firebolt. Firebolt is a high-performance, open-source, analytical database. It runs as a single binary on your laptop and scales to hundreds of nodes and petabytes of data in your cloud. Whether you're dealing with real-time analytics or batch processing, Firebolt lets you unlock the full potential of your data without without compromising on performance or cost. Hello, everyone, and welcome back to the Data Engineering Show. Today, I'm super happy to have Andrew Jones on.

0:34Andrew currently is a staff data engineer at LocalStack and has a long career in data engineering, building data platforms across companies like GoCardless, Arm, and a couple of others as well. So great to have you on the show. Do you want to quickly introduce yourself, Andrew? Yeah, sure. Great to be here. Thanks for having me. yeah so like you mentioned i've been building platforms for a while now i've always been building platforms really since i started initially with software engineering and those kind of platforms and the last 10 years was helping data platforms that's really our passion is and yeah as i mentioned i joined local stack recently maybe a couple months but we are a company that provides emulators for aws and snowflake i'm not working with products themselves i'm again building the data platform on the back end very cool nice do you want to give a bit more context on kind of what local stack does for listeners who haven't heard about it and also then what the data platform teams within local stack do sure so local stack we provide emulators our most mature one is for aws and we emulators are now you develop against aws services locally so you have a local stack we also work on the snowflake emulator so that's available as well and it's same sort of thing where you can develop against snowflake locally and the idea there is You can avoid all the cloud costs that you have when you're trying to develop against cloud, avoid having to stop in an environment for your engineers, and work much quicker because it's all local, it's all instant, it's all emulated.

2:00Very nice. And then that's usually just for me to understand, that's usually for development workflow. So it's not like someone would then also use local stack for their prod environment. For prod, they would still go to AWS or Snowflake, but it's kind of about making that fast, easy kind of development lifecycle, basically. Yeah, that's right. Yeah, for local environment, for CI checks, things like that. Yeah, don't use it in production. Use AWS for that. Okay, very cool. Nice. All right, that's exciting. It's also interesting that you're offering one for Snowflake. It by itself would be an interesting podcast episode of how hard or easy it is to actually build a Snowflake emulator.

2:37Yeah, definitely. And I'm sure everyone listening who worked today with ours knows that the warehouses can be quite expensive to get up against. and that's one of the things we're seeing from our customers is that we want to make it quicker because again, where it's going to be slow, but also reduce the cost associated with development and CI checking against data warehouses. Yep, makes sense. Okay, very cool. So tell us more about the data platform teams at LocalStackNet, right? Like what are you working on? What parts of the business do you help power, etc.? Yeah, so I'm going to be new, but as a team, we are moving away from a model where we had a few data engineers or data analysts and there would be basically involved in all data work for everyone going to do their analysis working with people in business all the time working on data collection from our emulators telemetry and user analytics we get and taking it from start to finish and owning the whole stack end to end and now we're a growing company we're realizing these data analysts these data engineers they become bottleneck we've only got so many of those but yet we want more people in the business to be able to use data whether it's our go-to-market teams or our marketing teams or our sales teams and learning how our customers are using the product, our product teams, looking at how the product is being used and what features are being used, what issues they might be having with our customers.

3:54We want to enable them to be able to be more self-serve and make use of the data and add more telemetry to our product without having to involve a data person each time. So the reason I've been brought in, the reason I've joined, the reason why I'm excited to be at LocalStack is to help us transition away from being centralised data team that's enrolled in all data work from start to finish to more of a data platform team that enables other people to make use of data. Right. Okay. Very nice. So take us through the stack maybe, right? Like what do you use Snowflake then? Kind of like what other technologies do you use?

4:27What type of data do you collect, et cetera? Yeah. So we have quite a lead step because again, we're quite a small company and we're still growing. We've recently started using Snowflake as our data warehouse. And that's nice because we can dog food our products and get good feedback there but we also use a product called tiny bird which is i managed click house and that's quite cool we've been using that for a while and we use that for a lot of real-time analytics as well a lot of data lands initially we're still trying to work out exactly what it actually looks like say this time next year but at the moment we're thinking we'll have a data warehouse in snow fake that does all the business porting if it will manage and maybe tiny bird or click house or something similar, but is a bit more self-serve, whether we can get started, record our own data, do our own analysis.

5:15But if you're an engineering team, you could do it, come in their own loop, have their own speed. That's still with a bridge between them both, still in a short context, it's in a time machine in both, particularly looking at maybe agents using data as well. So different parts of a stack, maybe with different use cases, but still joined together somehow. Okay, gotcha. And then at the moment, data basically flows from those real-time systems out into your data warehouse for more like offline and batch analytics? Yeah, exactly. At least for the emulator data, really the telemetry and views analytics.

5:47We also get data from CRMs and things like that and they also come in more batchy overnight and they'll join together so we can do, so we know our customers, we can do revenue reporting, those kinds of things. Okay, very cool. Nice. And then so take us through the journey of like building out a data platform and making things more self-serve, right? Like, are you guys powering all the analytics with agents now? And like, you're basically like giving Slack bots to people in the company. How do you work with also people that are differently technical, right? Because I guess like an engineer who wants to look at product analytics is quite different from an account executive who wants to understand like the pipeline and their territory or something like that.

6:29Yes, I'm quite lucky. I've managed to be involved in building out a day platform from scratch a few times now. And obviously each time it gets a bit better as I'm learning. This time, one of the things we're thinking of, well, actually for all the times, really, often we're thinking about the different personas of our users and our stakeholders and customers. So what do they want to achieve and how do they want to achieve it? So, for example, an engineering team, they're quite technical. They want to collect their own data and do so with more autonomy. and also want to be able to report some of their data and they saw their data dashboards.

7:02And that kind of data might be things like A-B testing results or engineering velocity metrics or more telemetry and stack traces from a product, those kind of data sets. So that's useful data, interesting data, but it's not ever going to be the highest priority for a data team where we're looking at things like revenue reporting and customer health and things like that. So for those use cases, because they're quite technical, the engineering teams or their product managers in the engineering teams we're looking at how can we give them with self-tourning so they can produce data and store it in a way that's still well managed and still able to be drawn to other data sets across company but with autonomy over how it's structured how it's stored how it's managed how it evolves over time and do so without having to become experts in say tiny bird or snowflake whatever they're basically using and then make it easy to consume data as well via a dashboarding tool or some description so for the interesting use case it's kind of the end-to-end flow but with autonomy and less focus on joining the data but more focus on going yeah end-to-end as quickly as possible without being experts in data management whereas for people business side they're obviously thinking more about consumption of data i don't want to be too involved in collection they want to give their requirements and say i really need data about when our customer signs up and when they start using products how long does that take those kind of things so i've got requirements in terms of what analytics they want but they're not going to build it themselves but yeah they're business stakeholders so we want to make sure that they are close to the engineering teams they can give those requirements directly not going through the data team each time as a botanette that will be very supporting but we're not making decisions we're doing all the work ourselves again don't want to be botanette but then it's how do we make sure that the data provided by the engineering team for these use cases lands in the right place, the right structure, that meets their requirements, and then works with the tools they want to use for consuming the data.

8:57So that might be a BI tool or just can dashboards. It might be an agent where they're kind of talking to the data, as it were. It might be putting data in the CRM so they can see it in HubSpot in our case. So you load it by customer. You can see all the mentoring about HubSpot in HubSpot around the go-to BI tool and back and forth. So there's much more focus on their persona as a consumer and making sure that they have the right access and the right data available to them at the right time in the right place. Right. Okay. Very nice. That's cool. So what are the main challenges as you and the team are building this out?

9:32Yeah, there's a few challenges and it's the same for many data teams. One of the biggest is you're trying to build this new world while you're still having to support the old world so you can't just drop everything you can say okay six months we're going to go away and build a new data platform it's all going to be self-serve and brilliant when it's done but in the meantime you get nothing that's not a good sales company so how do you free up your time where your current team does a lot of support a lot of analytics yourself and it's all very time consuming for your engineers your data engineers how do you somehow reduce the time they're spending doing that so they spend more time building this platform that will eventually lead to a more self-serve model where you're not always struggling for time and you have a time to build these things so that's one of the challenges and we're doing that by trying to automate some of the simpler and more torsion requests we get so for example this week actually we've had a hackathon and one of the things i've been doing is looking at how we can automate some of our more standard support requests and a lot of makes for access to say tiny word for example and you know people ask on Slack, go back and forth what I already need and then we go and make changes in the database in Tiny Bird to give access management.

10:44And that's a very automatable flow. Like you can have a, what I managed to do or what we've done this week in the team in the hackathon is just like a Slack bot agent where I can have a conversation with the user, work out what permission they need, then create PR and then structures code tools. So it gives them right permission and we just merge it at the end. And if you get confident, you don't have to merge it either. It can be completely self-set. And that's about 15 % of our requests support tickets. So I think it's important as well to make sure you're solving the right problem and using data to solve that.

11:13You know, for a data team, so we should be perhaps not what we've reached. So we know it's a good portion of our time is spent just doing these kind of access requests. So if we reduce that, reduce our support by 15%, that will feel like that's more time for development. Makes sense. Okay, cool. And then how do you as a data team basically like leverage agents to build out that new data platform faster and all of these things? Yeah, so like many people, I'm sure we're using coding agents and that helps a lot in terms of being able to do things like I just spoke about, like a Roo's Hackathon project.

11:49So, for example, I mentioned how we're using infrastructure's code to do active management for Tiny Bird, but actually Tiny Bird doesn't support VSAM box. It hasn't got a telephone module or anything like that. So we had to kind of build it. We had to look through their CLI and the API goals they're making. and it's not that obvious. And doing that, I could have done it myself, but it was taking me reads properly. But obviously with agents, no, you can move a lot quicker and by basically a day of work, a day of prompting, to get that's how that worked. Obviously, you've got to be a bit careful with agents and make sure you've got the right guardrails around it with production.

12:22You don't want them deleting and removing users without oversight and without the right guardrails. But you can move a lot faster. You can build this internal tooling much quicker than they could before, which means you can automate more and more than they could before. That's really helpful. okay very nice cool nice so zooming out a bit from like your concrete data platform challenges right like as you think through building this out now versus the times you've built out data platforms in the past is like give us your view on like how the world is changing how analytics is changing which things maybe actually aren't changing after all yeah it's a good question because i think some things are changing but some things are much the same as they've always been And I think one thing that's changing is just the importance of data and how reliable data needs to be.

13:07So I'm not sure this is always true, but people might have said before that the data team, the owner producing analytics for finance and the business, if it breaks for a day or a few days or a week, it doesn't really matter. If it breaks weekly, it doesn't really matter. And I'm not sure that's always true because it's not a great decision to be in data team if you're always fighting and fixing things. Because see why people outside of that team might fall back to be okay. It's a full trade-off. But now we're using data, not just for internal processes. A lot of companies are using data to build customer features.

13:44And when you're building features for customers, you can't have them just fail once a week because the pipeline failed because something changed upstream. For example, you have a higher level of reliability and more disciplined approach to the deck pipelines. And I think that's a change we're seeing a lot. People want to use data, either directly or by AI and agents to create product features that are differentiated by the unique data a company has. But increasing level reliability, a new requirement, the increasing requirement we have, changes how we think about how we build tape pipelines. Not just how I build it ourselves and applying more discipline and more software engineering style workflows to it in terms of observability and deployments.

14:26and rollbacks and things like that, but also thinking about where our data comes from and how reliable back data is. But there's no point me investing loads of time in my data pipeline, making it really reliable, really easy to observe, great alerting, I'm going to be on call for it. If the data I depend on is not reliable, it's kind of extracted from the database, like a data source, as people say, there's no one on call for it, there's got no SLOs, it's got nothing. like if thanks for no reliability guarantee is there i can't really guarantee that my day is reliable if i depend on it so i think people talk about being there's been shift left but you have to think about whole entering flow and if you want the output to be reliable then your bit's got to be reliable but also serves a bit upstream which means you have to talk to those people you have to explain to how important it is you have to place the right processes and interfaces to make sure that reliability flows end-to-end very cool nice awesome it's been quite interesting to see because I remember when we did the show, I don't know, two or three years ago before engineering changed so much.

15:30We were talking to a lot of people who were building out data platforms, thinking about data contracts, semantic models, all of these things. So many of these technologies are still relevant, but also changing a lot. I think semantic models are actually a great example, which are now actually way harder, I think, than they were in the past because A, agents can potentially partially generate them. B, agents need them to also ground their context and knowledge of the underlying thing so it's been interesting to see kind of the space transform there over the last couple of years yeah exactly yeah these emetic stuff's interesting that's something if i'm honest i haven't quite worked out yet in terms of how we manage context across our company and how we make sure you get the right context to the right agent at the right time that's something i'm thinking about a lot i'm doing a lot of research on but i don't think anyone's figured it out quite yet people talk about context layers and knowledge layers and i'm kind of hesitant to introduce more layers i think in data we tend to like bring more layers in but maybe there's a case for this one but data contracts like you mentioned that's something i've been working on for a long time actually now and what my book was about which came out a few years ago and that really came into play is because of the things i was talking about a minute ago about reliability so when i was working at go canvas we were using our data via ml as we called it at the time to provide features to our customers and that's where we realized the data wasn't reliable enough to do that to be good enough customers and that's where it started looking at data contracts to create a better interface between the upstream services and the data pipelines and ultimately vmr pipelines to ensure that reliability flowed end to end and that's really where it all came from and i think data contracts are getting more and more traction because more and more companies are doing the same sort of thing they're trying to use data for customer facing features or for high important features and realizing that if you want I'm about to be reliable.

17:21I'm about to be of the right quality. And you need to go upstream and make sure it's fixed there as well. Right. Perfect. Cool. So as we wrap up to the show for today, Andrew, any closing words, thoughts, anything you're really excited about for the rest of the year? Yeah, I think one of the things maybe to close on, actually, is kind of what I spoke about a minute ago when we talked about context layers and context. And the way I'll admit it, but I don't have all the answers that yet, and probably a lot of people don't. So I think often we read things online on LinkedIn or Substack or we listen to podcasts and things like that.

17:56And it feels like everyone always has the answer. But I think the thing is changing so fast that we're still working a lot of these things out. And I think one of the best things we can do is be open minded as we see all this change. Maybe ensure we're not just blindly following assumptions we had before in terms of how we manage data, how we define data ownership and data governance and really take the opportunity as things are changing to just question some of the assumptions we've had before and rethink exactly what we need to do to solve problems we have today not just do what's done before because that's how we've always done it so for example data governance a lot of data governance has been built around manual processes and documents and checklists but that's quite slow moving it's quite hard to validate and maybe these days with agents with platforms with data contracts with other tooling and processes we have maybe that's an area where we can rethink how we're doing it and be a bit faster and be a bit more dynamic than we were before while still maintaining control that our debt protection officer will need to have and maintaining compliance with regulations and things like that so they go on this another thing about a lot but it's just another example of trying and be open-minded, trying to admit the areas you don't know yet and just, yeah, keep learning and drive a ride.

19:15Nice. Awesome. Cool. Well, it was so great having you on the show, Andrew. Thanks for dropping, Guy. I'm excited to follow your journey at LocalStat. Cool. Thank you. Thanks for having me. Of course. The Data Engineering Show is brought to you by Firebolt. Firebolt is a high-performance, open-source, analytical database. It runs as a single binary on your laptop and scales to hundreds of nodes and petabytes of data in your cloud. Whether you're dealing with real-time analytics or batch processing, Firebolt lets you unlock the full potential of your data without compromising on performance or cost.