Uber’s On-Call Copilot with Paarth Chothani and Eduards Sidorovics

0:00At Uber, there are many platform teams supporting engineers across the company, and maintaining robust on-call operations is crucial to keeping services functioning smoothly. The prospect of enhancing the efficiency of these engineering teams motivated Uber to create Genie, which is an AI-powered on-call co-pilot. Genie assists with on-call management by providing real-time responses to queries, streamlining incident resolution, and facilitating team collaboration. Parth Chothani is a staff software engineer on the Uber AI Gen AI team. Eduard Siderovich is a senior software engineer on the Uber AI platform team.

0:37In this episode, they join the show with Sean Falconer to talk about the challenges that motivated the creation of Uber Genie, the architecture of Genie, and more. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

1:05parathan edwards welcome to the show thank you sean for having us thank you hi yeah thanks for being here you know i'm really excited to talk about you know genie this on-call co-pilot that you guys were involved in at uber but maybe before we get there let's have you introduce yourselves just so since there's both of you, people can hopefully learn your voices. But let's start with you, Parth. Who are you? What do you do? Yeah. Hey, everyone. I'm Parth here. I'm a backend infrastructure engineer on Michelangelo, which is like the SageMaker equivalent in Uber. And I've been here at Uber for four years working on distributed systems, generative AI, and core ML problems.

1:45And before that, I was at AWS, also building chatbot-like solutions, and also at Microsoft, working on Teams and those kind of products. Awesome. And Edward, same question to you. Who are you? What do you do? My name is Edward, and I joined over a bit more than a year and pretty much started right away to work with Barton Genie and also part of ML AI platform team. Yeah, and before that was pretty much working in some startups, training mostly some deep learning models. And partially for a year also working in the same manufacturing company and doing some MLB stuff for them. Awesome. So let's get into Genie a little bit.

2:26Can you explain what this project was and sort of how did it come to be? Yeah, I can maybe take a first shot at it. So at Uber, generally, there is so many platform teams, you know, supporting many engineers across the company. and there is a lot of like tooling and all of that that gets built to support all the engineers and to make sure that infrastructure is very highly scaled right and as part of that there's many many like support forums specifically slack is a very popular one and people generally engineers will come to slack for help and what you know we also went through as part of like our background and whatnot is like there was a lot of pain that we as engineers faced when asking for help from other support teams or other platform teams and this was a recurring pain across the company so that was something that prompted us to think like how can we solve this kind of a problem and that's where inception of genie started that we wanted to have an automated solution which can look at all the internal knowledge sources and be able to answer questions that use customers across the company, engineers across the company can take help from and really improve their efficiency.

3:42So, I mean, I think that this is like a super common problem that like many companies all suffer from. And it's certainly as you scale any organization, it becomes more and more of a problem where you end up with all these sort of data silos that exist or in the Slack world, they're like chat silos of one-off bespoke conversation that's happening where someone gets help. And then inevitably people ask sort of same types of questions. It's hard to surface that in a uniform way. And it becomes this kind of like death by a thousand cuts to like, you know, basically every company like suffers from this and certainly at scale become a real hindrance.

4:16So I totally get that. I want to get into, you know, Genie's architecture a little bit and like how you built up the project. So can you talk a little bit about what's actually going on? What is the user interaction and then sort of what is happening behind the scenes to support that user interaction? I mean, I'll just maybe start with the user experience of it. And pretty much, assuming I have a team and then we have, we maintain our own NGWiki and we have our own help desk channel. So we want to customers, we want to onboard Genie. So how it happens is pretty much, we have like a platform, Michelangelo.

4:51Then you go there, you create a project, you specify the ngviki which you want to use and then everything else happens on our side you pretty much it creates the pipeline you run the pipeline on backend it pretty much scrapes the data embeds everything and stores it and yeah there's another backend service which actually is queried when someone asks a question being that on other end there's a let's say a Slack bot and then it calls this backend service which gathers from which channel it is queried and then sends the question to Alala with the provided context. Yeah and just to add to that basically like our goal has always been a simplified very collective user experience where people can come in they just specify their sources and then boom that's it and then everything else is just one click setup for them to be able to, you know, use Genie in their own Slack channels or in their own UIs, whatnot.

5:55Yeah. So that's our being not star experience that we have been always trying to build towards. So me as a user, I point this to some internal wikis and knowledge bases. And then this Brad pipeline kicks off where it's going to go and essentially parse those, presumably go through some chunking process, create embeddings, land that in some sort of vector store. Can we, you get into details, how does that pipeline work? What are the sort of steps and components of it? Yeah. So underneath we use a lot of big data technologies like Spark, which helps us be able to take a lot of internal sources, be able to generate embeddings on the fly, parallelize basically with different, let's say we have executors being, taking different chunks, trying to create embeddings either through in-house models or third-party models, embedding models.

6:43And then even for something like when we want to push the data to a vector store, we have like workflows and open source technologies that we have used something like Cadence, which is a uber grown workflow system to be able to ingest data into vector store at scale. And there also we have Spark behind the scenes to be able to take all this data and do a faster ingestion on the fly. Yeah. And when I come in and I select these sources, how long does it take to essentially generate the vector embeddings to a point where I can start actually interacting, going through sort of the user experience, being able to get my questions answered by this copilot.

7:20Let's say there's a different thing. Like one is when you onboard yourself and because of there's some, like you have to wait for approvals and whatnot. So it can take like a day, for example. But if you, let's say, update your sources or you completely remap your sources, whatever, It doesn't matter. Run the pipeline, but yeah, it takes maybe today I run one and then took 15 minutes. Then it completely updates the sources. Yeah. Why have it where it's sort of a user configuring these sources for their specific needs versus sort of more of like a wholesale pipeline that is scraping everything, building one vector representation of all these internal knowledge bases and wikis, and then presumably being able to use the semantic search to attach the right context for the user when they're interacting with the co-pilot?

8:08No, that's a fantastic question because that was very much like our first thought as we wanted to build something like that. I think some of the things we learned also, so we were trying to explore some solutions like Glean, which could support something like this out of the box. And what we kind of found was that the way we had configured Glean inside Uber, it was very, very individual access oriented and there was not something like public data, which was all scraped for us already. So that was one problem that we kind of surfaced very, very early on. And then I think we also found like when we are more focused on ingesting sources, which are, let's say, more hand curated, more filtered, the vector store always does a better job at surfacing that information and also the accuracy is much higher versus when we experimented like we tried ingesting all for example ng wiki and the accuracy of the answer seemed all over the place for a given use case so we felt like not the best of the performance either so we try to find a sweet spot where we can enable people to bring in their sources but then again make it more like a magical user experience so that it's more UI driven.

9:26People don't have to do much. And that's what we have been building towards. Right. I just want to add that it's also the use case driven. In our case, we have a help that's channel from Michelangelo for our ML platform. So people are coming there to ask specifically about Michelangelo. So it's better to narrow down only to Michelangelo and not to surface anything what someone else have written about Michelangelo, which might not be updated. Maybe they once wrote it, they're not updating, but now because of their outdated information, they will surface the wrong answer. So we might also kind of eliminate that.

10:03Got it. Yeah. So it's sort of like an easier way to get the performance and accuracy that you need by having people sort of self-select into how they want to constrain the universe, than try to programmatically essentially figure out what the right context is going to be, because you're going to end up with a lot of noise across these different potential internal wikis. Yeah. Was there consideration around essentially, you know, scraping everything, but then for each sort of chunk, keeping the representation of the source? That way, when I'm selecting my sources, I don't have to go through the generation process of the pipeline based on sources.

10:37I'm essentially just subsetting the existing set of sources and embeddings. Yeah, we definitely wanted that kind of experience to start with. I think we found some infrastructure gaps where we pretty much don't have all this internal sources in an offline store that we can just pretty much take and create embeddings in the background. so we found some limitations there and which is where we went with the next best experience which is like let's just create it on the fly also the other thing is there is a lot of wastage if we just were to create everything behind the scenes the reason being that it feels like almost every team runs its own processes and own style so some teams prefer like michael angelo for example is very wiki eng wiki driven we will be very meticulous in updating wikis with all faqs and all user documentation and some other teams seem to not have that discipline which is where again enforcing something like this means like if we were to just blindly do everything we pretty much might waste a lot of resources also and not have much business gain either versus letting people choose i think then we have given them the capability to refresh knowledge which means like Once they know what they need, then they're able to refresh it at their own pace, which pretty much is like a second best experience of like what you're talking about.

11:59So if internal information gets updated in any of these source information, do I need to go manually like do a refresh? Or is there automatically when these updates happen to the actual like internal knowledge base or wiki, it kicks off this pipeline to do the updates automatically? I mean, it doesn't detect anything so far. It's meaning that initially it was only manual. Like if you update and you want to update, you go ahead and click. But then now it's also orchestrated in a way that it's like a cron job in a way that you can update it daily or whatever cadence you prefer. Okay. And then what model are you using for generating the embeddings?

12:39Yeah. So we have two different flavors of models. We have some of the third party models, which are open source and which we have hosted inside. those are some options and other options are also like you know third-party models which open ai and other providers also give so i think we have those options but generally we preferred the ada embedding models from open ai to begin with and those have worked reasonably okay for what we have been trying to test with so far yeah and what are you using for vector store yeah so we We have a homegrown solution right now for VectorStore, and we are trying to move towards other better VectorStore solutions.

13:18But that homegrown solution is what we call a SIA, and that's a solution that we have been working towards. And we're trying to embrace a new technology, as we called OpenSearch, as we are trying to kind of become more open source compatible. Was that something that existed already at Uber, or was that something you built specifically for this project? Yeah, so the technology did exist, but the technology existed for more for like the typical search, which is more text based search. I think when we started the project, there's like more of like company started realizing that there is a much better, bigger need for VectorDB store hosted solution, right?

14:02In-house hosted solution. So then the team, one of our sister teams, they spinned up infrastructure to be able to host a managed VectorDB solution. So it was more like I think we learned as a company there was a need across for GNI solutions like this to have a very nice, highly available infrastructure for VectorDB. And in terms of both from the pipelines and also like sort of the user experience interacting with the copilot, How is that essentially built to maintain reliability and durability so that parts of essentially this pipeline don't end up breaking or going out at some point? Yeah, I can start and maybe Edwards, you can chime in.

14:46We, as part of any production internal or external applications, we have a highly available monitoring system. so as part of that what we do is we make sure we have alerts on the back-end apis that surface responses to the slack channels or any uis that we are supporting for genie so that's obviously part one we also look at logs to make sure like if there's nothing obvious that is going wrong and then edwards can probably chime into more of the evaluation and what we have built around to make sure customers learn about how their applications how their channels and uis are performing against our backends and the whole end-to-end yeah i think it's one of the solutions is that we constantly receive feedback meaning that when we genie replies there's like a pop-up of you can reply with the emoji saying okay is it good it's resolved by genie or it's not good enough so this is also kind of keeps always a feedback loop for us to know like okay if something is good or not good yeah and then we kind of built some evaluation on top of it being that one of the more like interesting solutions what we did is that we thought okay that kind of genie has a interesting perspective on the documentation because like when you as engineer you write the documentation you think you know what people need to know but typically it's not true and And when customers ask a question, it means typically it means that something is not covered in documentation or they were just lazy to check the documentation.

16:25And what we actually built is that we check what answers were not good, meaning that if the answer is not good, it means that either RAC components were not good or actually the documentation was not there. So, and if the documentation is there, then we, like with another, like LLM the judge, we try to suggest what is missing in the documentation. Like it's kind of summarizes all the unanswered questions and tries to point out where it should be added and what should be added. Like specifically, how to run this pipeline, how to debug, it kind of points out. Obviously it doesn't know how to do it because it's over internal knowledge, but yeah.

17:09It helps. And it's actually, yeah, some users actually kind of acting on it pretty well. And just to add to this, like basically our idea is give people these tools that Edward was talking about, where they can pretty much figure out some of the high level themes around what documentation is missing, where the bot might be underperforming. They have at least a headway to figure out how to improve their channel quality. In terms of the feedback loop, is that primarily for you to sort of monitor performance and also give the team some insight into where maybe the documentation isn't meeting the needs, essentially?

17:48Or is some part of that also factored into sort of the learning cycles of the actual copilot? So if I know that the response wasn't good, I can take that into account the next time I generate a response to a similar query. So that is more of a first one that, yeah, it's to hold us accountable that knowing how good it performs and to make sure that we also motivate customers to update their entry keys and yeah, to point out what is missing. But it's also, yeah, now it's, we kind of adding more on top of it, that means that you can help to update the documentation right away. I think Parth can maybe explain it more, but this means that you can update the FAQs and then it will go eventually to the knowledge base and it will help to answer the question later on.

18:41Yeah, it's more like building a loop where people find out what is missing. They add FAQs to the documentation and then we have the refresh knowledge pipelines, which pretty much take this FAQs, refresh it. So it's like a quick feedback loop. and we actually found out inside even Michelangelo that there were parts of our documentation which are outdated. We didn't know about it. And then the bot surfaced some answers and we were like, how did this happen? And then we took initiatives to clean up documentation. And that was like a quick feedback loop that without even looking at our evaluation reports, we found out immediately that, hey, there's this problems within our documentation where we give conflicting information that we ourselves have not reconciled.

19:23Yeah, I mean, I think that's super valuable because I don't know, any significantly large organization I've ever worked for, the internal documentation usually is like, can get really horrific over time. And just, you know, there's not a huge incentive a lot of times to keep those things up to date. So it can really fall out of date, but they're really valuable, especially for new people, because they don't know where to get those answers. And the only option you have is internal documentation, or you end up having to message somebody and get that sort of bespoke answer. Can you take me through sort of the life of a query?

19:52So I'm interacting with this over Slack. I put in a query, then what happens sort of behind the scenes? Yeah, so behind the scenes, when you're querying, basically, we will invoke API, which pretty much underneath tries to figure out what the user is trying to do. And there is also, as part of the query, we also have a very customized Slack workflow functionality that we have built a plugin inside, which can take additional information from the user on what they're trying to do, what is the action they're trying to perform, which particular product they're trying to interact with, what is the way to reproduce their problem.

20:30So pretty much think of all additional context that on-call and a bot needs to be able to even figure out what the user is trying to do. with this whole additional information that we send as part of the question. Then we pretty much generate embeddings on the fly for the question. We do a vector DB lookup. We make sure, you know, we have all the right context. And as part of the ingestion that we have done for the source data, we make sure the ingestion follows the schema. That way there is, you know, source URLs. There is metadata around what the page was about. And all of this pretty much is part of available as part of the ingested data in VectorDB.

21:09So when we are sending all the information to LLM, we want to make sure that there is information around citation. There's much more metadata that we can surface for different use cases, pretty much. So that's all of the metadata is fetched along with the source URL and everything. And we send that to LLM with different prompts and we allow different users to configure prompts. There is flexibility in what they want to solve. and as part of this then the LLM pretty much decides what the answer should be based on the prompt and all and then that's what is surfaced to the user today. In terms of the LLM what model are you using?

21:46Yeah we have experimented with different models that OpenAI came up with so we started with you know GPT-4 then we moved to Turbo there's GPT-4-0 now and then we're trying to look at the reasoning models also to see you know how we can have certain questions answer in a much more crisper and cleaner way with the detailed reasoning still. You mentioned at the beginning of that query to response pipeline that you're trying to figure out what is the user actually want so that you can sort of attach that to creating the correct context. What's involved with figuring out what the user actually wants, what the intention behind the query is?

22:28Yeah, I think part of this, what we are trying to also currently experiment with is user intent detection where we can figure out like is the user trying to debug a problem is the user's question about like a product those kind of things we are trying to experiment and see where intent detection can help us figure out like more of the user's thought process because not all type of questions we have understood also the bot can do a great job at so we want to also as part of like our accuracy enhancement is be more mindful of where the bot can excel and where the bot cannot excel and that's part of like where we are trying to do experimentations and detect some user intent detection right now yeah what about metrics around evaluating sort of the effectiveness of this like do you have things that you're tracking even in sort of the development process like using like an eval framework some of these newer frameworks exist for building generated ai applications in order to figure out if you make a change to how you're generating your embeddings or how you're figuring out the intent, that's actually a performance improvement versus a degradation of some sort.

23:37I think the main matrix is the customer feedback. That's, I guess, our end goal. Yeah. So if you make a change, essentially, you're waiting for sort of live feedback to see if your accuracy is improved based on the feedback from the users. So I think that's part one of it, obviously and then there's the golden data sets that people generally hand curate so that we make sure there is more like quality built in before also deploying a change so if somebody changes a prompt or something we generally ask the users to do more golden data set against testing so that they have thought about what what kind of implication that have and edwards can maybe chime in on the post-production rollout here and we also try with the different evaluation and i I think like more classical NLP and then also LM as a judge.

24:24And apparently most of the cases, actually, LM is just more simple, but it's actually typically works better. Was there any challenges or thought around the risk of like sensitive information being shared with Jeannie? Yeah, that was something we really brainstormed and thought a lot. And I think me also coming from Amazon where I was like a security certifier. So security was always like top of our minds when we started this. And, you know, we want to be very, very mindful of what data gets exposed to the outside the company. So in the beginning, we were very, very thoughtful about hand curating which data sources are secure.

25:02And we have different levels of gradations, like many other companies of what data is private versus public or, you know, what is very sensitive that cannot be leaked outside. So we work with our security teams. We hand curated certain data sources, which were reasonably, you could say, public inside the company. And we obviously went through a lot of different processes inside the company before we were okay to even create embeddings for those kind of data sources. And that was our due diligence to make sure that as we develop a new productivity enhancement, we don't leak out data that will mess up our company's reputation.

25:38One thing to add, I think there is a very cool solution which is built in Uber. It's a Gen AI gateway. It's pretty much, imagine that you have an OpenAI API, but then it doesn't go directly to OpenAI. It goes through a gateway. And the gateway, they actually filter PI data. So there's not high risk of leaking anything. Yeah, so if I put in my social security number for some reason, it's going to get filtered out by the gateway. Yeah. We really wanted, as Edwards mentioned, like the redaction PII data should be redacted before it gets sent out. So I think that's built into our other ecosystem that our sister teams have built to make sure that we have security built in into, we don't have to worry.

26:25But still as application owners, we have still done our due diligence to even make sure PII doesn't even come in our ecosystem. Yeah, basically shift that problem left before it enters the model. Exactly. You know, in terms of building the system, like what were some of the biggest sort of technical hurdles that you had to work through? Yeah, so I think there are many, many different angles to where we struggled in the beginning. One was obviously hallucination was a start where we were like, you know, just spitting out things that were pretty much wrong sometimes and not right. So there was a lot of this prompt-based evaluations that we had to do.

27:01There was also the UI experience that we really, really thought very deeply about because there were other solutions that Glean provided, for example. and those solutions were very individual access driven needed approvals from users even to see the answers in channels and we didn't want that kind of experience because we wanted a frictionless experience so definitely the experience was part of it then obviously when we started developing there was no industry standard on how to evaluate genia apps so building like the feedback looping system for example we had to come up with methodologies on how we can even compute and say we are saving time for the company and users so there was some methodology we had to develop inside to figure out how to even say there is some productivity gains here and then obviously like edwards can speak more about the eval part which he's driving the whole evaluation of how to showcase what is problem with your documentation that was like a unique thing that we had to brainstorm and the ui the product we built around to support this kind of monitoring that was a very new thing is all there's no industry precedence as such on this is how other people have done it so there's a lot of these new things we had to maneuver and also we were working in a very small team pretty much a two to three people team so that was like we were very short on people to try something like this and also other challenge was how to platformize this kind of stuff not only prove that this works well how to platformize in a way that we can benefit a lot of you know other parts of the company and make sure that people can leverage this fast enough and show gains so the speed of execution the the accuracy the ui experience the monitoring and working in a very small team.

28:54I think all of these were like different challenges we had to really maneuver all throughout to deliver something here. Yeah. And I think just to jump in for a second, like I think like one of the challenges that probably anybody building, you know, like sort of an AI or gen AI application like this today is facing is that even if you have, you know, deep expertise in ML, you know, very few people have 10 ,000 hours of experience, like building these types of applications, right? So there is a lot of sort of net new ground to figure out. And you can't necessarily draw on your 10, 20 years of engineering experiences that you've seen this problem a hundred times before.

29:32I think one of the challenges was the, actually not maybe UI, but the UX, like how to make sure that everyone can, to make it scalable so that everyone can create their own genie and be like specifically tuned for them. I think that was a kind of, quite a lot of design thinking how to do it. And also, I think one of the, not also a technical problem, but it's expectation management. I think it's like ChatGPT works well, and then everyone has this miracle experience, right? But then you see, you go to other help desk channel, and you see that it performs very well. Even though you feel that it performs well, because you don't know much of a context, you think that it works very well.

30:15But when you run it on your own documentation, like, oh, no, it doesn't work as well as you expected. And you look, okay, why is it? And typically it's, yeah, maybe just because the documentation is not up to date. So it was challenging to like explain that, yeah. Let's say in machine learning, we say like garbage in, garbage out, yeah. Yeah, I mean, it goes back to the data quality problem. And, you know, if your data is bad to begin with or some portion of it is bad to begin with, like what can you expect in terms of like The model can only do so much, right? It's not going to fix your data problem for you.

30:51Yeah. Yeah, and also circle back to the question you were talking about, like the lack of experience in building this kind of thing. I think there are some parallels that I kind of sense still, like, yeah, while nobody had experience in this technology and whatnot, I think we, inside the small team, we were all part of, we were trying to be scrappy at the same time, speedy in execution, and we had to balance obviously security i think those three angles we tried to do and i felt like pretty much most new projects have that kind of thing where you're wanting to be scrappy you wanting to be showing something but also being mindful because we are in a bigger company we're not in a smaller company where you know you can afford to make mistakes and this is a public company so i think drawing from our previous experiences we try to have these principles in mine.