Data Science & AI: Getting Buy-In and Demonstrating ROI | PagerDuty’s Sanghamitra Goswami

0:00Each phase should have a deliverable. It might not be a specific product or anything. It is okay to be imperfect, but let's have a goal. And when we have a goal and when we have a plan at each phase, then big foundational work might seem achievable. Although you cannot see an ROI at one and two, but since I know I can give you that ROI back at phase six, so this is very important. In that case, you'll be more convinced than me going to you and saying that, hey, I don't really know how to go to phase six, but we need to do phase one. How can you drive developer productivity, lower costs, and deliver better products?

0:41On June 20th and 27th, Linear B is hosting a workshop that explores the software engineering intelligence category. We'll showcase how data-driven insights and innovative workflow automations can optimize your software delivery practices. At the end of the workshop, you'll leave with a complimentary Gartner Market Guide to software engineering intelligence and other resources to help you get started. Head to the show notes or linearb.io slash events to sign up today. Welcome back to Dev Interrupted. I'm your co-host, Connor Bronson. Today, I'm being joined by Sanghamitra Goswami, Senior Director of Data Science and Machine Learning at PagerDuty.

1:16Sanghamitra, thank you so much for joining me. Thank you, Connor, for inviting me. Honestly, it's my pleasure. I think we could really benefit from your expertise as someone who has such a deep understanding of the approach that data scientists have taken to developing AI models. And, I mean, frankly, AI is all the rage right now, right? Everyone's talking about it. Everyone has opinions on what it is. And so it's important that we level set with the audience a bit and have the opportunity here to pick the brain of a data science leader like you and understand how engineering teams can translate this and leverage AI models or LLMs in particular within their org.

1:54So let's maybe unravel some of the strategies that champion the role of data science, machine learning, AI teams, and help our audience understand how to navigate this emerging future and ever-expanding landscape. Why don't we talk a bit about the history of LLMs, what they are, and why they're so important? You know, Connor, it's been a crazy time now. Nine months back when Chaya Jubilee came out, PagerDuty leadership, they told me, Mitra, we need to do something with LLMs. And it's crazy the way the world is saying that, hey, we want to do LLM. Let's have a feature that uses LLMs. So what are LLMs?

2:31Large language models. they leverage foundational machine learning, AI, deep learning models to understand natural language and to give answers so they can talk to us like a bot. If you look at the history of NLP, it's based on NLP, it's based on the NLP models that we have built out. If you look at the history of NLP, in 1948, Shannon and Weaver, the first paper came out. And during that time, we didn't have the computer storage that is possible now. We couldn't really have a lot of computational power. So it was not possible to always run these large language models because they require large amounts of text.

3:14However, from that start, where Shannon and Weaver in 1948, if you fast forward, I would actually mention there is another milestone where the transformer architecture got introduced. And attention is all you need is the paper. So if I look at these two milestones and how the landscape has changed, the computational power, GPUs, so with everything in mind, now is the perfect time that we can all reap the benefits of years of research and computational power and engineering in that sense. So this is the perfect time. Absolutely. It's really interesting to think about how even just a few years ago before we realize that we could paralyze processes within AI model development using GPUs, there just wasn't this speed of development that we've seen on AI models today.

4:06I know. So I'd love to kind of talk about how teams can actually leverage AI or LLMs within their tooling. What would be your advice to engineering leaders who are thinking, hey, I want to start using AI to extend the capacity of our product, how should they kick off? I think let's start with AI, just AI. Someone wants to do AI with their teams, okay? That is a huge challenge because if you look at all the leaders in the industry, how do we see if something is successful in the industry? We have to get some ROI with all the endeavors, right? And data science, AI, it's an experiment. So when we start building it, it is always not clear how this is going to show up.

4:53For example, in AI models, we try to build a model, we experiment it with historical data, and then we take the model out in the market. And then it starts getting all this new data from our customers, and our model is giving answers live in the runtime. So it's very difficult. It's an experiment that we are running. And leaders should be very careful that they know that what are the risks of running these experiments. That's number one. Number two, they should be very careful on how they measure adoption or how the results of these experiments are being used by end users. Once these two pillars are in place, I think it's easier for leaders to measure value and define ROI.

5:41Without these two, we can't do AI or we can't do AI successfully in a company. So these are the two main, I would say, pillars if you want to do AI. Now, there are other organizational challenges as well. For example, democratization of data. Everybody talks about it, but it's a difficult job. Data is always in silo. There are different channels. If you look at marketing, there is social media, there are emails, there is YouTube or other social networks. So, so many different channels and you have to get the data together. If you look at logistics, there is carrier data in the ocean, carrier data on road, in air.

6:29So, it's just that if we want to measure any process, data is always in silo. And there are huge efforts that is needed on the side of the person who wants to do AI to do a successful AI. That's number one. And we have the architecture. We have the solution. We know that we need a data lake or we need a source of data where all the data can be consolidated. But it's a huge effort. It's a huge foundational effort that always doesn't show direct ROI. So you have to convince, as a leader in AI, you have to convince your leaders that, hey, I'm going to do this foundational effort. However, you might not see a direct ROI right now, but it's going to, you know, executives talk about this flywheel effect, but it's going to create that flywheel effect at some point.

7:24So that's the discussion they need to drive. Absolutely. I think it's really important for us to understand both the potential and the risks of AI. And I don't mean risks of AGI and this world of crazy cyber villains created by AI. That's fun to think about on sci-fi. Sure, we can talk about that. But more specifically to your business, there is a challenge where if you don't put these foundations in place, there's major risk to how your business will present itself, whether that model will hallucinate. And this is where it comes down to these foundational data science concepts you're talking about.

8:00Is my data siloed? Do we have the right training data? Is that training data validated? Are there issues with that data set that are going to cause long-term issues? And when I've talked to other data science leaders like yourself, that is one of the things people really hone in on. So I'm glad you bring up this foundational piece because a lot of leaders are getting pushed by their board or pushed by their C-suite of like, oh, we need to get AI in the product. But if the data that you're feeding in to train the model isn't data that is actually validated and maybe peer-reviewed or checked on, there are major risks that you put in play.

8:36Yes, absolutely. You need to know what your data can do. Without that, garbage in, garbage out. You cannot save yourself even with an LLM. Very well said. I think it's challenging, though, for a lot of leaders to get buy-in on that foundational work, as you pointed out, because there is an immediate ROI. So how should engineering leaders start to get that buy-in about ensuring they actually do all the steps needed to be successful? Yes. I think we work in an agile world. So we should have a plan. We should have a plan with different phases. And I always say this to my team that each phase should have a deliverable.

9:18It might not be a specific product or anything, but it should have a goal. It should have a deliverable. I don't know if I read about this Wabi Sabi. It's a Japanese guiding principle. It talks about continuous development and that imperfection is good. And I say this to my team that Wabi Sabi, let's do something. It is okay to be imperfect, but let's have a goal. And when we have a goal and when we have a plan at each phase, then big foundational work might seem achievable. And that is very important. As say you are my boss, Connor, and I'm talking to you and I'm giving you a plan and I'm saying that, hey, phase one, two, I have a goal and I know I can go to phase six.

10:05and at the end of, although you cannot see an ROI at one and two, but since I know, I can give you that ROI back at phase six. So this is very important. If I give you the plan, in that case, you'll be more convinced than me going to you and saying that, hey, I don't really know how to go to phase six, but we need to do phase one. So if I'm an engineering leader who hasn't had deep experience with data science or AI, and I'm thinking about how do I build this phased approach, What would be the general steps you would advise? Or is there a resource where leaders can go in and say, hey, let me look at, I don't know, a template to start applying to our specific use case?

10:45Yes, I think there are many if you look at Google, like how do you use Gather ROI for data science projects? But I would say before we do that, it is very important that in any organization, there is a product counterpart with a data science engineering manager. I believe there should be other people championing data science rather than... To get buy-in. Yeah, to get buy-in rather than the data scientists themselves. So it is critical that you have a friend in the product organization because they can look at the product holistically from a top-level view and they can help you go ahead. What would you say to people who are having trouble finding that champion or picking the right champion?

11:26Convince your boss. Convince your boss that you need a product partner. A single person or a group of data scientists can't always do everything. You need to have someone else beyond that organization who could champion for you. Sometimes an executive can play that role too. But with having some time and common goals and everything, I think it's critical that data science organizations have a product partner. And I'm sure this creates some translation challenges across the company as you're trying to bring in these other stakeholders and get people to buy in because you know you need the support.

11:59but maybe the goals across those different organizations can be different. How would you try to solve that cross-organizational transition challenge to get these champions? Well, I don't think the data scientists can solve it by themselves. What they can do is they can say that, hey, I understand it's late in your roadmap, and I can be ready on my part, and I can make it easier for you to understand and access what I'm developing. But I do think executives play a very important role here because they need to drive alignment across different teams of the organization. Let's say engineering team A and engineering team B both wants to do data science, but they don't have time because their roadmaps are full of other projects.

12:47Whatever the data scientists do, it won't convince them, right? So the executives need to prioritize that. Or a product partner who can look at it and say, hey, this data science feature, if we do it, this will drive huge ROI than a small change or than any other feature that we are taking out this year. So we need to have executive alignment on roadmap across teams and also some other champions. But what the data scientists or data science organization leaders can do is they can think of, okay, here are some benefits of empowering data scientists and data engineers so that they can write code well.

13:30this is i'm going off a tangent because you know data scientists come from different backgrounds and they are always not the best software engineers so they need support from data engineers and they need to productionize their code write production level codes so what the leader in the data science organization do is make sure that the organization is empowered to build something that is very easily accessible and can be taken by the engineering team and the engineering team doesn't spend a lot of time building that or understanding that. It's interesting because you're talking a lot about these change management concepts, frankly, of getting organizational alignment, building up champions within the org, making sure you get buy-in so you can showcase that ROI, ensuring you have these phased rollouts and a clear goal for each step of your process.

14:20What if you're having trouble getting that kind of buy-in? Are there ways that data science or engineering teams can leverage currently available AI models or tooling to showcase the ROI and then create that buy-in? Yes, there are also many tools in the market that they can use to show that ROI. But it's a little bit difficult. Once again, I think it's an experiment. That's how I see data science. So it's not depending on how much customer, how much data your customer have. You know, some customers might be new. Some customers might not be storing data very well. So the experiment that I have run on my end might not always be great when I run it with real time data with all my customers.

15:09So having those risk factors, figuring those out before release or having a slow release so that you can talk to your customers and figure out. Connor is a great customer because we have five years of data and our model is going to give very good results. Whereas Mitra might not be a good customer because she has only six months of data. Figuring those out and what is the fraction of your customers will give good results. So the risks, I think, thinking those ahead of time makes a huge difference. I mean, we've talked about this some on the show before, about the importance for leaders of understanding the risks of even exciting opportunities.

15:46And I think you bring up a good one, which is it's really easy for us to over-exaggerate the impact of AI on a particular customer or on a feature or product, where maybe the realistic truth is that it's going to take time for it to develop because you need that data integrity to actually build up. because you need more customer data. How should you go about communicating with customers about what you're able to do with AI as you build your program? I think building trust with customers is key. At PagerDuty, we have a process called early access where our product is not fully built out, but we are in the early access program.

16:26We have a prototype and we can ask our customers to use it and give us feedback. I think that feedback is critical. They can tell us that, hey, it's giving great results. They can tell us it's giving very bad results. So then we know and we can improve. So this early access program is very useful. How are you leveraging AI at PagerDuty? We do a lot of AI. So we have five features. And when I say features, these are features which have different models in the back end. So we have five AIOps features. and our AIOps, which used to be an add-on for our full IR product, now is a separate BU. So we have a lot of features in AIOps, noise reduction.

17:10I was just talking to someone who mentioned that it's always a problem when you have lots of alerts. And we are talking about security camera, like Google camera. You keep on getting alerts and then you are lost, right? So the same thing happens when people use PagerDuty. People use PagerDuty when there is an incident and you are getting alerts. And if you get a lot of alerts, if you're inundated by alerts, you don't know which one to go for. So we have very good noise reduction algorithms and we use AI to build those noise reduction algorithms. That's a super smart use case because that kind of cognitive load, it really makes us start to tune out.

17:48I mean, like, I'm sure we've all been guilty of this, maybe with our email sometimes. It's like, oh, so many. Okay, I just got to get through these things. And it's so easy to miss something that might be important if you're not really staying on it. And so that's a great example of how you can leverage the power of AI to assist your customers. Are there other ways that you see PagerDuty leveraging AI in the future? Yes, we have root cause. So I'm talking about non-NLM AIOps features. We have root cause. We have probable origin. And what we do with these features is during the incident, during the triage process, we try to provide information to developers who are looking.

18:27We're trying to figure out what has gone wrong. How can we figure out how can we resolve the incident faster? So we have a suite of features on that end. On the LLM side of things, we have three new features that are coming out. These are our first gen AI features. We have a summarization use case. I think this is a very good use case. And one of the ways I always say, once again, going off a little tangent, I always say that if you want to do LLM, find a good use case. And I think this is an awesome use case. So during that incident, developers are trying to solve a problem and say that, OK, I'm resolving the incident.

19:06But even during that phase, they have to update their stakeholders or external companies who are waiting for information about the incident that is going on. that's a very difficult job. Developers who are always in the back end, they need to write up an email and, you know, divide their attention between solving a problem and drafting up an email. So you'll give it to the Generative AI platform because now they can do it for yourself. And those conversations are already there in Slack, in Zoom, in Microsoft Teams. So why repeat it? Ask your Generative AI model to write it for yourself. Smart.

19:43So I think this is a very, very good use case, It's powerful and empowering to the developers who are using PagerDuty. And you mentioned this idea of ensuring you find the right use cases or good use cases. Yes, very important. What's the approach that you think people should take to that? What is the problem that you are solving for? How would you provide relief to your customers or the end users? What would they find useful? That's the key. And that's true for LLMs too. You want to find a use case, but is your use case solving the problem that is being asked by a lot of your customers? And that's just good business advice, period, right?

20:21Like solve your customer problems. The phrase like make your beer taste better has been popularized recently. It's a great example. We say in Peter Dutty, champion the customers. Great way to put it. I'd love to just get some more general thoughts from you about your viewpoint on AI in general, where the industry is going, the explosion of success here now that we have paralyzed GPU models, we have years of them working. Obviously, there's been an explosion in the public consciousness of the ability to leverage AI. As you pointed out, this all goes back to old papers. This goes back to old sci-fi novels, frankly, where we talked about these ideas.

21:01Now we're seeing them come into reality. What are some of the things that you're excited about by the current AI revolution that's happening? I think I'm seeing very good use cases. One use case that I really loved, I am big on Instagram. And, you know, I was looking at the photo editing capabilities. And you can just take out a person that you didn't like. So if you've got an ex-boyfriend or girlfriend or something like that. No, think about it. I want my picture before the Eiffel Tower. and I don't want anyone else. And I can do that now with AI. So I love it. I love some of the applications that are coming up.

21:44This is a fun one, but there are very useful ones if I look around. Recently, when I was going to the Chicago airport, they did a facial scanning and they didn't actually scan my boarding pass. Not LLM, but still it's so cool where I'm just walking and there is a machine who is catting my face. I get some privacy concerns there, I have to admit, but it is very cool. Yeah, it's just so cool. Well, thank you so much for taking the time to chat with me today about this. It's been fascinating to dive into your thoughts about AI. Do you have any closing thoughts you'd like to share with the audience about either how they should approach LLMs or what all of this change means?

22:25I would say data science leaders fight for space. You need to do more. Think of a good use case. Ask your executives for a product partner. Try to prove that the features you want to develop, that the use case you are vouching for, you want to build for, is going to solve a customer problem. And that is needed. Write up. I think writing is very useful. And give it away for people to consider, take feedback, be vocal. I would say that. Well said. Sanghamitra, thank you so much for coming on the show. It's been a distinct pleasure. if you're someone who's listening to this conversation consider checking out on YouTube we're here in the midst of an incredible lead dev conference here in Oakland and I think it would be a ton of fun for you to see us having this conversation live on the YouTube channel so that's Dev Interrupted on YouTube check it out and once again thanks for coming on the show Thank you Connor, thanks for the invitation