Build AI products at on-AI companies with Emily Glassberg Sands from Stripe

0:05Today, Sarah and I are joined by Emily Glassberg Sands, who's the Head of Information at Stripe, which includes data science, growth, machine learning infra, business applications, and corporate technology. Emily was previously the VP of data science at Coursera, where she led development of AI-powered products to have personalized learning, scalable teaching, skill measurement, and more. We're excited to talk with Emily today about Stripe, AI, fintech, and education. Emily, welcome to New Priors. Thanks so much for having me. Oh yeah, thanks so much for joining. So you now look at the information org at Stripe.

0:38Can you organization does, how it's evolved under your tenure, and what are some of the span of responsibilities that you're focused on? Yeah, so I joined Stripe back in 2021, originally actually to lead data science. And David Singleton, Stripe's CTO, reached out. I didn't know a ton about Stripe, but I knew millions of businesses were using it to collect payments, which had to mean really interesting data on those businesses and on a large swath the economy. Stripe's clearly helping companies run more effectively and also in a position to learn from its data what kind of interventions significantly improve companies' long-term success, and in some cases, to actually action those.

1:18Today, I wear two hats. So the first is I support a bunch of different teams that are together tasked with enabling the effective use of data across Stripe. And this includes, you know, from decision-making internally to building data-powered products, we've been investing a bunch in foundations, which includes building out our ML infrastructure and better organizing our data, you know, the really sexy stuff, but also in applications like seeding a bunch of new Gen AI bets and getting them out to our users. So that's kind of hat one. And then second, I'm accountable for our self-serve business.

1:50So a huge number of SMBs and startups come to Stripe directly to get started. They self-serve through the website. And we're really focused on understanding who those users are, getting them the right shape of integration efficiently, building product experiences that meet their needs, including as they grow and growing the portfolio of products they use. So for many of our users, it's not just payments, but invoicing or subscriptions or billing or tax or rev rec, depending on what their business model demands. Yeah. And I guess Stripe for a long time has been doing different things in ML in terms of traditional ML.

2:25You know, I think fraud detection and the fraud detection API that you all have is one example of that. But you were actually quite early in terms of adopting LLMs and sort of early generative AI models. Could you tell us a little bit more about how that came about, how the interest was sparked and how adoption really took off? I mean, I think it's fair to say that Stripe isn't first and foremost an AI company. As you note, fintechs, including Stripe, have long used traditional ML in many contexts, including sort of fraud and risk. But first and foremost, we're building financial infrastructure for the internet.

2:57So Stripe got started by enabling first really digitally native startups to accept online payments. And then over time, millions of companies started relying on Stripe's financial infrastructure for a bunch of different needs, whether that's reducing fraud or managing money flows or unifying online or offline commerce, all the way to launching embedded financial offerings. And so as not a kind of first and foremost AI company, we probably like a lot of people listening to this podcast had kind of our like, hey, what the heck are we going to do moment a year or so ago when LLMs really broke through the zeitgeist.

3:33And we were looking at the technical breakthroughs and the product launches all over the ecosystem with awe, but also honestly a little bit of overwhelm. The sense of, well, there's very clearly a real opportunity here to better serve our users, but what is it exactly? And how do we get it off the ground quickly and safely? So it starts with a story of three engineers who hacked together in three weeks an internal beta for an LLM Explorer. And the basic idea of LLM Explorer was, hey, let's get a chat GPT-like interface in the hands of the 7 ,000 talented Stripe employees and really let them figure out how to apply it to their work.

4:10Our leaders all the way up to John and Patrick have intentionally crafted this strong culture of kind of bottoms up experimentation. And we think a lot about sustaining it internally as we grow. And with LLMs, it was no different, right? And so where we started was let's quickly unlock internal experimentation. Let's get LLMs safely in the hands of all employees at Stripe. The enthusiasm was palpable, you know, At Stripe, as it was across industry, we knew the experimentation was going to happen. And so we really wanted to make sure that we enabled it to happen well and safely. It feels to me like when people start adopting LLMs, they tend to do it in sort of three areas as an enterprise.

4:48There's what are you doing in terms of external products and how do you incorporate it? There's how do you use it for internal tools or use cases? And then there's what are your vendors doing? You know, if you're using Intercom or Zendesk, are they adding it? And if so, how do you think about that as a company? The third one seems to be each team kind of deals with it as their vendors bring it up. I found in general, people have tended to follow the pattern that you mentioned, which is they start off kind of thinking, hey, what should we do externally? And then they immediately collapse into doing something internally, just so that people get their hands on it, they try it out, they kind of see what it does and how it works, and they get some internal efficiencies.

5:22And then they start thinking about the external product side of it. You know, was there anything you did specifically to start to promote that internal usage? Did you do a big internal hackathon? Did you try other ways beyond sort of the, the, some of the things that you mentioned in terms of adoption internally, just so that you start spreading the thinking and knowledge about it? I think you're spot on that a lot of companies, you know, the first place they go in their mind is how can this manifest in our product? How can we help our users? And then they realize, hey, any one person can actually answer that question.

5:53We need to be putting this in the hands of folks with a range of different backgrounds and expertise, thinking about different parts of our product and business to really apply And that's exactly what kind of this built-in three weeks beta did. Within days, a third of Stripes were using it. And you can think of LLM Explorer basically as a front end that supports multiple models in the back end. So we started just with GPT 3.5 and GPT 4, but today we serve over a half dozen models through the tool. We knew it needed to have certain security features, stripping PII and rehydrating, etc., straight from the start.

6:27And we spun up, yes, a Slack channel and a hackathon and more to help kind of build momentum. We didn't actually need to do much to build momentum. Within days, a third of Stripes were using it. And so from there, we started to look at, okay, what are they using it for? What do we see in the logs? And the answer was Stripes were using it for all sorts of things, honestly. But there was this opportunity to create more community and sharing in the tool directly so that they could build on each other's work and weren't sort of doing that in inside Slack. So a simple example, but shortly after we launched the original tool, we set up this little functionality called presets and basically just lets you save and share your prompt engineering.

7:08Maybe this exists. If not, some startup should go build it for everybody. And then everyone else at Stripe can like search and upvote and you see what bubbles to the top. and basically overnight we had like 300 of these reusable llm interaction patterns and they ran the gamut but you know um just an example like thousands of stripes still today use the stripe style guide which basically you know i don't care if you're a product marketer writing copy for the website or a sales development rep writing a cold email or like you're an exec who's preparing for a meeting you run your copy or talk track or whatever through this style guide and it returns back to you the same content in stripe tone the enthusiasm was palpable and we had to figure out ways to harness it and build more of a community around it.

7:48The weekly active user count of this LM Explorer is still at almost 3 ,000, which is just shy of half the company using it every single week. And yeah, for sure, engineers, but also a ton of salespeople and marketers and all those folks. So I think there's a lot that the technology can actually do to create the community. And then the next step is, okay, how do you get it from this like internal prototyping to actually like enabling also more production grade solutions. How did you begin to look at that data or go from explore to exploit here, right? So one of the stripes I talked to said that I should ask you about applied ML accelerator teams.

8:29I don't know if that's like here or further along in the funnel of like, you know, your plan to get these new AI capabilities distributed across the company in real ways. We can talk about it here and we can talk about it later. So the idea of accelerators is basically ring fencing one to two pizza teams and multiple of them to get new AI bets seeded. And one of the accelerators was actually what produced this LLM Explorer. So it's very hard to just pull three engineers off of, you know, their work building radar to build an LLM Explorer. So we have this sort of experimental bets funding internally.

9:13It's run out of, you know, by David Singleton, so out of our CTO's office. And it'll basically be like, hey, we'd like to build a one pizza team and we want to fund it for six months. So relatively durable. And here's roughly the charter and here's roughly the milestones. But we're going to learn and iterate as we go. And so actually this infrastructure is an example of an output from the accelerator. We have other accelerators that are working on the applied side. So, hey, you know, we know that given the advances in LLMs in particular, there's way more we can do for our support experience, both user facing and also internally for our ops agents.

9:53And to your comment earlier, Alad, for sure, there's third party solutions we can buy. But is there some, you know, homegrown solution that can actually be used across a variety of internal applications? And can we just go build that? So that's another example of the kind of thing that our applied accelerators build. And I think, you know, the applied accelerators aren't you staff a one, you know, you fund a one pizza team and you go hire these people. They're actually opportunities for growth and development for internal talent. So the vast, vast majority of folks who join the accelerator have been at Strait for many years.

10:26They're doing a rotation onto the accelerator. It's likely to become their permanent home, but that's up to them. You mentioned you have some favorite applications that have already like these sort of assistant capabilities. Can you talk about some of them? The primary ways we're finding LLMs useful today at Stripe in user facing applications is first automating the writing of code and then second accelerating information retrieval. And both are proving really powerful for our users. So on automating code, Radar Assistant and Sigma Assistant are two new products that are in beta and rolling out to all users soon.

11:02Radar Assistant is really about generating custom fraud rules from natural language. So most folks listening probably have heard of Stripe Radar. It was one of our first non-payments products. It's an ML-powered product. It helps identify and block fraudulent transactions. But then in addition to the core Radar product, which works generally under the hood without any user-provided direction, we have Radar for Fraud Teams, which is about letting users write custom rules. So maybe you know you don't have any customers in a given country and you want to block any transactions from IP addresses in that geo.

11:36To generate these rules, employees that are users used to have to code up the rules themselves. But Radar Assistant lets them use natural language to write those rules. So it's a little thing, but speed matters a bunch in fighting fraud. You have to work faster than the fraudsters. And with Radar Assistant, a whole range of people in an organization from fraud analysts all the way to less technical folks can implement rules quickly and directly without having to work through a developer. I actually think that's, and I do want to hear about Sigma Assistant. I think that's a really interesting pattern that applies beyond perhaps the fraud world because there are so many, let's say, like just decision engines today that are some combination of heuristics and then machine learning together.

12:22and I think that will continue and the ability to take natural language, explicitly describe policy and have that work really well with less engineering assistance, I think is going to be useful in like other domains, like, you know, could be underwriting, could be fraud, could be other choices. Totally. And, you know, I think for some of our customers, and this is opening the aperture in terms of which employees can use solutions like custom radar rules, but for a lot of our customers, it's allowing them to use these solutions for the first time, right? So think about the non-technical small businesses on Stripe, a bunch of them.

12:58You don't have to be technical to get started on Stripe. You can use our no-code integrations. You need payment links. You can use hosted invoices. These are companies who like wouldn't dream of coding up custom fraud rules. And so not having to have, they just don't have the developer skills on hand. And so just not needing to, being able to use these tools with just plain English, I think is really powerful. And more broadly, I really love that democratizing power of generative AI. And it's very much aligned with our founding ethos. You're also going to talk about Sigma Assistant. Sigma Assistant is similar in that it generates code from natural language, but it's in a pretty different context.

13:34It's actually applied to generating business insights. So Sigma is our SQL-based reporting product. It lets businesses analyze and get insights directly from their Stripe data. And Stripe data is, as we've talked about, pretty interesting. For most of our users, it's all of their revenue data. So which customers where are buying what for how much, who's retaining, who's churning. Pretty central to a bunch of different decisions the firm has to make. And Sigma Assistant is all about making sure our customers' employees don't have to speak SQL to get access to those business insights. They can just use natural language to ask questions of the Stripe data.

14:11Some of the folks in the beta are asking, you know, really interesting questions and getting them answered, you know, from the very basic, how much revenue did we generate in December to, you know, what types of customers tend to be most delayed with their payments. So we're excited to be rolling that out broadly later this year. Where do you hope all this technology to be in one or two years? Like, how do you think generative AI will impact your business, your customers, the way you do things? When I step back and ask where should we be in kind of three years, five years, I think the vision, the opportunity is much bigger than what we could do in a year.

14:49With a fintech lens in general, I think the current sort of gen AI advances beg the question of what does it actually mean to apply generative AI to the economy at large? You could start with payments optimization. I think folks know that we do a bunch of back-end and front-end optimizations for payments. Is there some actually new foundation model built on financial data that would blow the existing conversion and auth and fraud and cost optimization models out of the water? You know, we can do incremental model improvements today, and quarter over quarter, they drive meaningful bips of uplift.

15:25But it doesn't feel crazy to think that a good foundation model could outperform more traditional approaches by, I don't know, 100 bps, 200 bps. So I think just in payments optimization alone, we can ask the question of what might foundation model look like in that context? And then I think where it gets really interesting with generative AI on all this payments data is, can we actually become more of the economic operating system for our users? And you can imagine all sorts of ways this could be productized everything from a dashboard of insights and recommendations to like an API you hit to get customer level predictions to like, you know, turning important business model and personalization decisions.

16:03So pricing, recommendations, discounting kind of on autopilot with Stripe. We know we can abstract away a bunch of the need for our users to worry about payments and refunds and disputes, but you could imagine also starting to tackle those sort of higher order tasks, understanding the value of users and setting the right price and determining the geo strategy. And then this This is more on a macro level, but businesses rely on all sorts of economic signals, CPI for tracking inflation or small business index for tracking the health of the sector. And those are very useful for steering business decisions based on macro trends, but they tend to be quite lagging.

16:39And so this question of can real time data speed the time to insight and thus response, I think is interesting. So, you know, those are all more sort of future looking, but I'm very bullish on a world where we're able to really holistically help users grow their businesses well beyond payments, but built on payments data. Rewinding all the way back to now, you are a year into the exploration. How do you decide to invest beyond a one pizza team? Like, does that happen organically in all of the product engineering teams you have? Does it happen where you like, at some cadence, look at the usage data and say like, oh, these like, you know, top down, bottoms up are things do you care about?

17:17Like, do you need to restructure the org to make that happen? Yeah, it's a great question. And I don't think we 100 % have the answer of what's the right operating model, but we've been very conscious and iterative as we're learning. And so, so far, the answer is both. Like, you know, it's not one piece of team or two piece of teams. It's four of them today? And should it be six or should it be eight or should it be 10? And then in parallel, where can we really support the vertical teams or the core product organization in adopting LLMs or generative AI more broadly directly? There are a couple of examples, but at Stripe, we're very focused on leveraging AI so that non-technical folks that are users can do things that they couldn't do before.

18:07And then also so that technical folks can move an order of magnitude faster. And there are some pretty obvious industry standard ways that we're finding LLMs can automate the writing of code and accelerate information retrieval. And we're building those both out of the existing vertical teams and out of the accelerators. That makes sense. You mentioned the UAPS, you know, I think it was at least six different models you're using internally. How do you think about what models to use for what? And do you focus on RAG, fine tuning, open source, closed source, time to first token inference. I'm sort of curious like what that matrix of decisions is relative to specific use cases and how you ended up with this sort of proliferation of models because I feel like the more sophisticated people get, the more they tend to have this proliferation happen internally.

18:53So we do have a proliferation of models, but we are not centrally, for example, like within our ML infrastructure group, super prescriptive about what model individual applications need to use. So I talked about LLM Explorer and the presets and sort of that was back in March. And we very quickly turned that into building an internal API for more programmatic use of LLMs, right? We wanted it to be equally easy and safer stripes to build production grade systems and services. There are 60 applications built on that now, a bunch internal, but also several external. And I'm happy to talk about a couple of them.

19:33That's what planted the seeds for a lot of the product initiatives we're now investing in more heavily. We have default models based on the use cases, but we also give individual teams agency to choose based on cost considerations, latency considerations. There's obviously like, depending on the application, different performance requirements. And then, you know, there also is this very real question of cost. So, you know, we're running this infrastructure centrally, but we found that for the most expensive applications, you know, we do bill them to the local teams. And so we work with them very closely to understand what makes sense given the economics of the product, the importance of quality at this stage, how they're thinking about scaling, what the latency requirements are, and so on.

20:19And we have heard that previously, it was a little bit overwhelming for individual teams to figure out what model to use, but also to go through the enterprise agreement and get the infrastructure up and running. And so centralizing a lot of that, I do think has sort of economies of scale. But again, we're not prescriptive and we do leave agency to the individual applications to make those trade-offs. What other infrastructure do you decide to build centrally? Right. So another strip told me that I should ask you about your internal experimentation and sort of testing infrastructure. And so love to hear about anything new you've built in order to like enable teams from your org or a central org?

21:03Yeah, so, you know, I think it's always a combination of buy and build. And we recognize that there are a lot of great companies building a lot of great ML infrastructure and experimentation solutions. And some of them are very plinked and some of them are very general. And, you know, we stitch together where there's a clear external solution and we build internally where we feel our need is more unique or somehow very important and not currently satisfied by the market. Our experimentation platform is one that we've built internally. We run a lot of charge-level experiments and latency and reliability requirements for charge-level experiments are very, very high.

21:47And so building and running that internally has been worthwhile. But there are lots of cases, flight, weights and biases. There's lots of third-party solutions that we lean on as well. When you think forward on the directions that the overall financial services industry is going, and let's put Stripe aside for a second, because I think Stripe is obviously a core company to sort of the internet economy, and it touches so many different pieces of fintech and things like that. But where do you think outside of Stripe the biggest white space for fintechs employing AI is? Like from a startup perspective or even an incumbent perspective, like where do you think this sort of technology will have the biggest impact?

22:29It's a great question. I don't know exactly what others will do. I think having a really robust understanding of identity, who businesses are, what they're selling has always been important. And, you know, I think often in industry, we think it's important for marketing or sales or sort of go-to-market motions, but it's also super important in fintech. Yeah, it's important for credit lending decisions, but it's also important for supportability decisions and understanding where, you know, the business does or does not meet the requirements of a given card network or a given bin sponsor. And so I think that that identity piece, like who is this merchant?

23:20Are they who they say they are? But also, what are they, what's their business? What are they selling? And how does that map to this pretty complicated regulatory environment is a really interesting and hard problem that lots of folks are solving in their own ways, but is likely an opportunity. I think there's almost certainly an opportunity to, you know, whether Stripe does it or somebody else does it, to make sort of financial integrations way more seamless. Stripe has a whole suite of no-code products, so you can use, you know, payment links or no-code invoicing. But how does one actually build a really robust, specific to the user integration without needing, you know, a substantial number of payments engineers or any complicated developer work?

24:21LLMs are proving that they can be very good at writing code. We have a couple of cases actually where we're already seeing it work. But as the decisions get more and more complicated, I think there's still a lot of work to do to build the right integration and to build it well in an automated way. And then I think, as I mentioned before, some of this layer on top of the payments data of like, okay, you could build solutions that make payments work better. but payments actually allows you to really deeply understand and improve the business is pretty fascinating. And you'd have to think about like, is it a startup that does that or is it an incumbent that does that?

25:05And what's the business model? What's the business model there? But, you know, if I think about the case of Stripe, you know, Stripe has the opportunity to be beneficent, right? Incentives are super aligned. The more Stripe can help its users, businesses grow, the more Stripe grows and the more the economy grows. And so whether it's Stripe or someone else using financial data to help businesses be more successful, to grow the pie, to grow the GDP, I think is really powerful. It's a really unique data set. Is there something in that data? The obvious example to me that comes up is radar, but otherwise like leveraging that data and giving it back to merchants in some useful way already?

25:51Yeah. So Radar is a great example. I think you also see it throughout our payments products. So maybe the most salient to a consumer, like an end user, not our customer, but our customer's customer would be something like the optimized checkout suite. So it's this bundle of front end payments optimizations. And it's a lot of little things, honestly, like dynamically presenting payment methods in the order that are most relevant for the customer that really add up in terms of driving efficient checkout experiences for end users and in turn driving up revenue for our customers and growing the internet economy.