How this PM streamlines 60k-page FDA submissions and saves millions with Claude, Streamlit, and clever AI workflows | Prerna Kaul

0:00Prerna Kaul:So you are working in the life sciences on really health impacting vaccines and treatments. And with all that amazing investment in scientific work also comes a lot of paperwork. We had to develop a nearly 60 ,000 page document. Would have taken about four to six months of effort and nearly 20 specialists, not to mention the millions of dollars spent. Where did you start? I gave Claude the problem statement, the pitch and the demo. The thinking I had in mind is that Claude is a software engineer and I'm talking to them and trying to tell them why it matters like any good PM would and trying to tell them what end product we want to produce as a result.

0:42And not only did it fully understand what I was asking it to develop, it created a little setup instructions markdown file for me. And it gives me all of the set of instructions and all of the tasks that it does and its capabilities, which is super handy. Were you really able to shift the cost there? Did you see an impact here? We did. We did. And any kind of cost savings you can generate have a direct impact to the bottom line. And in addition, if you're saving on time, you're bringing life-saving vaccines in the hands of people who actually need it.

1:18Prerna Kaul:Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive here on a mission to help you build better with these new tools. Today we have Prerna Kahl, who's taken years of machine learning experience and helped develop some of the products you know and rely on every day at companies like Amazon Alexa, Moderna, and Panasonic. She's going to show us how she uses AI to accelerate drug discovery and vaccine approval, something that takes tens of thousands of pages of documents, dozens of people, and months and months and months of work. She'll also show us how you can use Jane Austen, Dale Carnegie, and some AI to manage your stakeholders just a little bit easier.

1:58Prerna Kaul:Let's get to it. This episode is brought to you by CodeRabbit, the AI code review platform, transforming how engineering teams ship faster with AI without sacrificing code quality. Quality code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant code review comments and potential impacts of every pull request, beyond just flagging issues. CodeRabbit provides one-click-fix suggestions and lets you define custom code quality rules using AST grut patterns, catching subtle issues that traditional static analysis tools might miss. CodeRabbit brings AI-powered code reviews directly into VS Code, Cursor, and Windsor.

2:41Prerna Kaul:CodeRabbit has so far reviewed more than 10 million PRs, been installed on 1 million repositories, and has been used by 70 ,000 open source projects. Get CodeRabbit free for an entire year at coderabbit.ai and use the code HOWIAI. Prerna, it's so nice to have you. I'm really excited about your workflows. Thank you so much for having me. Very excited to be here. So you are working in the life sciences on really health impacting vaccines and treatments. And with all that amazing investment and scientific work also comes a lot of paperwork, if I am correct. And so in addition to doing all this amazing research, you also have to document and submit a lot of stuff to the FDA and other regulating industries.

3:37Prerna Kaul:And I am sure that is a total pain. So tell me a little bit about the problem that led you to building some of these AI workflows. Well, yeah. Firstly, thank you so much for having me. and you're totally right. With life sciences, I think the challenge across all parts of the organization is a lot of paperwork, a lot of regulation, and going back and forth with these regulated industries could take months of time, not to mention all of the people resources involved and delays in getting, in some cases, life-saving drugs and vaccines to market that people need. For example, when I was working at Moderna, we were developing the INT cancer vaccine, which is necessary for folks with skin cancer.

4:22It was sort of a new niche concept that had come out. And we had to develop a nearly 60 ,000 page document, which is known as the, it's a technical jargon, but biological license agreement, which took around four to six, it would have taken about four to six months of effort and nearly 20 specialists, not to mention the millions of dollars spent across the organization and getting this document together and providing it to regulated authorities across the world. Very labor intense, very time consuming. And that's where I kind of came in. I was a Gen AI product manager there and really came in to think through how can we reimagine this using generative AI?

5:05This was back in 2023, where I came up with the solution. So a couple of years back and I kind of evolved it over time to kind of specialize in certain areas as well. I'd love to show what we came up with.

5:19Prerna Kaul:So it sounds like you created a really high impact tool and 60 ,000 pages is pretty intense. What I love about this workflow is you're gonna show us actually how you use Gen.AI to build this Gen.AI solution. So like all good product people, you started with requirements. Can you just show us your process from you've identified this problem, which is we've got to create this massive document. And it's going to take millions and millions of dollars and tons of people to actually having this production thing that your team and colleagues could use. So where did you start? Like every good PM started with the requirements.

5:54And I actually didn't, hadn't fully formed all of the ideas initially, because I think that with, with the complexity in the domain knowledge required, it took a few iterations too. But I'd love to showcase sort of where we started with very simple pitch and then go through how we actually evolved the idea as well. So I gave Claude the problem statement, the pitch and a demo. The thinking I had in mind is that Claude is a software engineer and I'm talking to them and trying to tell them why it matters like any good PM would and trying to tell them what end sort of product we want to produce as a result.

6:35So I've included statements such as, you know, this is Stefan Barcells, who is Badanas, he was passion project. He's very excited about it. here is the cost savings you can get, here's why it matters and the impact. And then also spoken about the demo that I want to showcase and how we could sort of iterate and build on it. And Claude came back to me with not only did it fully understand what I was asking it to develop, it created like a little setup instructions markdown file for me. And it gives me all of the setup instructions and all of the tasks that it does and its capabilities, which is super handy.

7:14And not only did it do that, it also gave me all of the code and all of the different aspects to the code as well. So I kind of started here and started thinking through what do I really need to do to make this a production-ready system. And there were two things that were extremely important in the process. A first is because the BLA is a structured document, it is XML based, it has certain formats, I needed Claude to successfully, in a very strict and structured way, take data and structure it into a certain format. So that was a very strict requirement and that's not super easy to get with some of these models as well.

7:54The second was that because it is clinical data, it's highly regulated and being able to detect PHI, redacted information was extremely important so that it could correctly redact that and then not have that included in the actual BLA that we would be sending to any authority because it's patient data essentially. need. So the reason why we choose Anthropic CLAWD is because of the reputation around safety and alignment with human values as well. And we formed this partnership with CLAWD and really focused on some of the PHI and the safety aspects to make sure that not just from a compliance standpoint, but also as we think about alignment, we were taking care of those use cases.

8:41So this is what it came up with. It did a really good job. And I was able to get some code running as well.

8:48Prerna Kaul:And I'd love to show you what it came up with. If we can go back to the requirements really quickly, there's just a couple of things I want to call out for folks that I think is really interesting. If you go to the step-by-step instructions, what I love about it is if you go to the top, it not only gives you sort of how to technically write the code, but something I wanted to call out at the bottom is it actually gives you a pitch narrative to how to run your demo. So what I think is so interesting about this specific prompt is you gave it kind of pretty loose requirements. It's only a couple paragraphs up top and it gave you a whole set of things.

9:23Prerna Kaul:I'm sure many of which you weren't anticipating. I wouldn't have anticipated getting a demo script from Claude when I'm asking it to build Python, but it gave you that. I thought that was really interesting. And then if you scroll down in the left side, something else I'd like to call out for folks that I think is really interesting is it gave you these kind of like component names of different technical pieces of the product. So like coming up with mock data, doing PHI reduction, all that kind of stuff. You know, you as a software engineer and a product person, how long do you think it would have taken you to come up with this full set of just this first one-shot prompt requirements before?

10:06Prerna Kaul:At least a couple of days, at least about a week to really think through everything. And then in addition to knowing you wanted to work with the anthropic models because of safety and alignment, were there other libraries or other models that you knew you wanted to use? Did it pick the right ones? Like how close to the technical implementation did it get? In particular, with the redaction piece, it would have taken a couple of data scientists to figure out the right solution and work with us. So data scientists as well as software engineers to work with us and figure out what's the right solve for this?

10:41How do we implement it? How do we test it to get a certain synthetic data? So I think where it really excelled is in picking the right models. And I have some amount of machine learning knowledge, but it would have taken me a couple of days of study to figure out, okay, this is the right model for medical named entity recognition, and this is how you go about it. But it did such a great job with very little information up front on the domain itself. It was able to absolutely pick the right models and give me the code and then implement it and show me the confidence score across that as well.

11:14Prerna Kaul:Yeah. And if we look at the code, I just have to laugh because, you know, many, many years ago, I think I have lost hours and hours and days and days of my life to writing regex. just I used to spend all my time writing regex terrible that didn't work it was just awful and so one of the kind of miracles of AI I see is it reduces this toil in technical tools you know you need to use that have documented patterns but they're just so hard to research and get perfectly right out of out of the gate so this was my favorite part of seeing seeing your code up is seeing all that code just pre-populated for you.

11:56Yeah, that's such a common and shared pain point. Regex has been a, yeah, it's a bane of any software engineer's existence.

12:03Prerna Kaul:Yeah. So, okay, so it created this Python script for you, and you said you scrolled through it, and it looked, you know, pretty correct. And then you decided to deploy this on Streamlit. Can you tell us a little bit about that choice, why you picked that? Yeah, so I love using AI to solve for some and automating my own work and solve for my own sort of day to day problems, either be it related to work or something I'm trying to work on in my side projects or personally. But I think the real unlock, at least I found, is that if I can create something of value for other people, and then especially as they are non-technical folks, making a UI and making something generally available and hosting it locally has proven to be a really, really big unlock.

12:52Both career-wise and also, you know, working with different colleagues, they find that to be really valuable because they might have something in mind and I have all these skills and I can sort of offer that. So that's actually something I've been doing in my current organization as well, where I recently created a Google add-on to convert PRDs into Jira format. And that was a lot of fun. And, you know, the product managers especially were very excited about it because they don't necessarily like having to write requirements twice, one in a PRD and then also in Jira. And we found a good way to kind of circumvent that and do it in a matter of minutes.

13:31So I did that. And then the Streamlit idea, I also did in a couple of my previous roles and my current role as well, to give non-technical users access to all of these tools.

13:41Prerna Kaul:Yeah. And just for folks that don't know, Streamlit is a really easy way to put a pretty simple UI on front of like a Python kind of file instead of APIs. Exactly. So Claude created your PRD. It gave you code. You're pretty happy with the code. How do you get from us looking at this code right now to a product that your team could actually use to create these documents? I'm going to run for you the actual Streamlit app. So I'll take you to Terminal. I essentially run it from the command line and it opens up a web app. It takes me here. It has preloaded an API key from Cloud, which I've provided in the code.

14:21So I've hidden some of those properties. And I'm going to generate synthetic data for us. And I'll talk through and I'll go through the step as well because it takes a bit of time and I'll talk through what it's done. So, as I mentioned, there's this format called a common technical document as part of the BLE. There are several models, about five or six different modules, and it has a structure around it. So there is there is a study report that has a tabular listing. It's supposed to have very specific points across the data from the clinical trials and then different literature references as well.

14:57Prerna Kaul:Did Claude know about this format kind of from just general knowledge or did you have to provide the format to it? So it searched up the common technical document format for me and then it figured out the right module to apply to my use case. Great. That's awesome. Yeah, it's doing pretty well. So I'm running here. I'm going through the different, we have different participants here. And there are some clinical notes per participant. This is all clinical trial data, very, very sort of common that you would see in any trial that runs. And by the way, the trials itself, they could run in the order of billions.

15:40So redoing a clinical trial is very, very expensive. And so as much as possible, we want to give very accurate information to the FDA so they don't ask us to redo anything. And what I've done here is I've de-identified some of this patient data to remove any sort of names, date of birth, any information that could be associated with the person.

16:03Prerna Kaul:And one of the challenges I'd love to call out for folks that maybe aren't seeing this is you generated some fake data so you can validate it works. And it's not only this tabular data that has patient ID, ethnicity, whether they were in the control or otherwise, but you also have these freeform clinical notes where there may be embedded PHI written in by a clinician or a study administrator. So it's not as simple as saying just find all the data birth columns and redact them. You actually have to go through this unstructured data and find potential sources of PHI. Is that part of the challenge?

16:39Prerna Kaul:That's exactly it. It's a complex problem. Usually, you know, machine learning doesn't always do a perfect job there, but Claude immediately identified the right model to do this. Great. So, okay, so you have this clinical daily preview, and then you're going to show us how you de-identify it. So you're not sharing that with folks that don't have access to that health data. Exactly. Okay. So with this synthetic data, I clicked on this button to detect and redact PHI information. And it scanned through all of the rows and it found for me all of the PHIs. So here's what it came up with. So seeing the clinical notes, it found that there was some dates, names associated.

17:24There was some birth date information as well. and the remaining it was able to keep sort of in a de-identified way so that I can include it in my reports.

17:34Prerna Kaul:Great. And then how would this have been done in the past? I think you said there are software engineerings or ML engineers that would have built these models to redact the data. Was there any human in the loop at any point in this as well? How was this done before? For the past couple of years, most pharma companies have been using machine learning for this. But these projects, because it is PHI and there's a very high bar, they end up taking months of effort to get right. And especially when clinical trial data might not look the same for every drug. as a company like Moderna was trying to scale, we were trying to launch nearly 50 drugs, bring 50 drugs to market in clinical trials and then 15 to market in the next three to five years.

18:26That problem was exacerbated. So thinking about coming up with new machine learning models to solve different problems across each of these vaccines. It's a matter of scale and time, really. And we really needed to find a better way to do this.

18:43Prerna Kaul:Great. Okay. So we've got redacted health data. You have all your great billions of patient data points. What's next in this flow? So the next thing is to generate this common technical document. And I'll show you a small preview just for the purpose of this demo. So what it did is it took all of that and it gave me a summary of the different data. It was able to give me a synopsis, which is what the FDA looks for, the study participants, the counts, the age, all of the statistical data points, as well as some of the specific medical terminology that they're looking for. So it kind of summarized everything.

19:23It gave me this. And for context for folks, this is typically done by a medical writer. And their whole, I mean, I've worked very closely with them. It's so time consuming because they not only they can do a pretty good job of generating like a first draft summary, but they have to go back and forth with their stakeholders for weeks sometimes on a single module to come up with the right set of criteria and getting everyone to align all of their stakeholders to align is very, very time consuming. 20 specialists, as I mentioned, are working on it at any given point in time. And as part of what some of these, the chat GPTs of the world and Anthropic can provide, there are also common projects and common environments where all stakeholders can kind of collaborate and work together on these summaries, which is so much easier and so much better than working, let's say, in a sort of a shared document and just providing comments.

20:24Prerna Kaul:Yep. Great. Okay. So it generated from that data, this very, I mean, it looks official to me. I'm not, I'm not part of the FDA, but you know, it's a summary of statistics, a summary of the methodologies, a summary of the impact. It's pretty long and detailed. It looks like you can download it both to a TXT form, like a doc form, as well as this, as you mentioned, proprietary XML format that's required. And I'm presuming the Claude code just gave you two buttons to make that happen. That's exactly it. And I know it's a little bit proprietary, but I cannot tell you how excited I was to see this button.

21:04Prerna Kaul:This episode is brought to you by Lovable. If you've ever had an idea for an app but didn't know where to start, Lovable is for you. Lovable lets you build working apps and websites by simply chatting with AI. Then you can customize it, add automations, and deploy it to a live domain. It's perfect for marketers spinning up tools, product managers prototyping new ideas, or founders launching their next business. Unlike no-code tools, Lovable isn't about static pages. It builds full apps with real functionality. And it's fast. What used to take weeks, months, or even years, you can now do over the weekend.

21:41Prerna Kaul:So if you've been sitting on an idea, now's the time to bring it to life. Get started for free at lovable.dev. That's lovable.dev Okay, so if we could just take a halfway point of the podcast, the two things you were most excited about and I was most excited about. I was excited about the regex. You were excited about the XML. This is a very nerdy and very specific How I AI podcast. Okay, so you've generated the doc and then I love this last part. So let's talk about what's this last piece of the product. What's super interesting is I've worked in a couple of different organizations where as we are trying to scale different AI use cases, cost comes to be a factor because not just token costs, but also acquiring licenses for these models and then running it as the number of consumers or the users of the actual use case scale, the dollar cost also scales proportionally.

22:40So one of the ways in which we could potentially continue to launch and continue to build products around this is by actually being transparent about the cost associated with the operations. And, you know, generally, one of the stakeholder arguments that I've previously received pushback is, this is going to be high cost. Can we find a simpler way that's not AI-based? And how I've tried to solve this is actually being transparent about the cost associated with it, how long it would take. And as you can see, this trace and cost analysis is super useful because it says cost per operation. It gives a duration against it.

23:21And then it actually breaks down by operation what was the token cost and the total cost. So extremely useful. You can continue to track and monitor this in production as well.

23:32Prerna Kaul:Yeah. So then you can say, for example, you know, PHI redaction per patient costs 15 cents. And you can decide as an organization, is that more or less expensive than alternative models? I mean, I actually haven't heard anybody give this suggestion. So I want everybody to hear it, which is if you're getting internal resistance to cost. In fact, I feel this even as a solo founder who's vibe coding things. And I see Claude Code give me like, this cost you$5.13. And I send that to people. They're like, that's so much money. I'm like, it's$5. And so I love this idea of bringing true transparency. And I'm sure these numbers are much smaller than your production numbers on your million person studies.