#317 Steven Brown: Why Modern Medicine Needs AI-Assisted Decision Making

0:00What I'd been doing in AI was a concept of using a mixture of agents and models to try to create different pathways and pathways and pathways. the knowledge that's compressed into these large language models. But when all of this happened to me, I repurposed what I was doing, took some of the softs I'd been working on related to working with multiple agents and multiple points of view and multiple models. And I started having them all look at my medical record because I had been feeling like there was something going on and feeling like I was ill for some time.

0:37I'm Steve Brown. I've been an entrepreneur and much of that time in medical technology, working on things like chronic care management and patient monitoring and kind of left health care for a while to become a documentary filmmaker. but after the pandemic, I got back into tech and especially with AI because it was so such an exciting time to be in tech. I didn't think I was going to go back into healthcare, but then I actually had a health issue myself personally. About a year ago, I was diagnosed with a rare blood cancer and that dragged me back into healthcare. So I started taking what I was doing in AI and applying it to my own medical record, to my own situation.

1:29And what I'd been doing in AI was a concept of using a mixture of agents and models to try to create different pathways and through the knowledge that's compressed into these large language models. And I was doing a lot of that in the kind of educational field, a lot of things with Peter Diamandis and Abundance 360 and for that community. And I thought I was going to be not doing healthcare, but when all of this happened to me, I repurposed what I was doing, took some of the software I had been working on or the concepts I'd been working on related to working with multiple agents and multiple points of view and multiple models.

2:13And I started having them all look at my medical record because I had been feeling like there was something going on and feeling like I was ill for some time. And but I wasn't really being diagnosed with anything. My doctors were thought it was a fishing expedition. But then things got a little worse after our this is kind of a crazy story. I know, but it's like our house burned down in the big fires in L.A. That displaced me from the health care system where I was and the doctors where I was. And I ended up in near Palm Springs staying with friends. And that's where I ended up having a severe abdominal pain and all kinds of things that's led me to the emergency room, thinking that I was obstructed from steak dinner.

3:07So I went into the emergency room and I said, I need a CT scan to rule out obstruction. After I got the CT scan, they found a bunch of things on there, lymph nodes and things that I guess they hadn't noticed when I had my CT scan just two weeks before. And including, you know, I had a colonoscopy, an endoscopy, a whole bunch of tests just in the weeks prior, but hadn't really been diagnosed with anything serious. So it was kind of a surprise that to get a really serious life-threatening diagnosis where the prognosis, if you catch it too late, is not very good. But luckily, because of the fire, I actually caught it just in time.

3:55I say just in time, it's not too much damage had been done. And I got on kind of the standard of care therapy for it. It's a it's it's I basically have a version of multiple myeloma, which is a cancer in your plasma cells and your bone marrow. But the version I have, they create a misfolded toxic protein that can accumulate on your heart and your kidneys and in your gut and cause all those organs to fail if you if you don't catch it soon enough. So I knew it was very urgent and I needed to get this under control very fast because I needed to prevent getting my heart damage and kidney damage. Yeah.

4:36Let me ask just to establish your bona fides for listeners. I mean, give a little bit of your tech background. You were working with Peter Diamandis when this happened. Is that right? Yes. I was his chief AI officer. And, you know, what did that mean? That means he's got a really interesting organization that has an annual conference and some other things. And I was building apps for the membership of Abundance 360 for the conference. I built a kind of full conference application. And then I was building in a lot of AI features into all of these applications. And then I was doing workshops and I was doing a bunch of things.

5:18Like I spoke on stage the year before. I'd give a 35-minute main stage presentation talking about what I was doing in education, which was I had brought back to life, or back to AI life, all these great thinkers from history, you know, Socrates and Plato and Aristotle and, you know, 100 great thinkers from history. And I created them in a way where they could talk to each other. So when I could have them debate things and talk about topics and all kinds of things. I even made this thing where you could ask Socrates to write an app for you and then he would take over your computer, write an app, and they'd just show it on screen.

6:01So I was kind of demonstrating all of this sort of mixture of things, doing a lot with AI, which is just a fantastic playground, learning a lot. But I was very hands-on. I was actually, you know, I'd been a CEO of tech startups a couple times before, and I had always had a chief technology officer and an engineering team, and I was dealing more with the business side. But my background was studying physics and computer science at Stanford. And, you know, I started off my career as, you know, writing code. But with the pandemic, I went back to, you know, hands-on writing code. Kind of inspired a little bit, I heard And Sergey Brin from Google talking about going back to the office and actually writing code and how this is such an amazing time to be working hands on in technology.

6:47And I'm like, I'm going to do that too. I started writing code and the leverage today with AI, I mean, I could do things in a weekend that used to take like 10 people a year to do. So what does that do? If you have like ideas, it means you take on more ambitious projects. So I was taking on more and more ambitious projects and doing a lot of different things. I just wasn't doing health care because it's like I'm done with health care. Health care is too slow, you know, too painful. Life's too short to do another health care deal. But then, you know, so I had a lot of technology that I'd been working on.

7:26And I mentioned that I became a filmmaker for a while, but I was maybe a little bit sort of tired of, you know, the solution to every problem as, you know, some version of a software. where I kind of went a little bit different direction for a while, but my startups were all tech startups building pretty serious technology platforms in healthcare. We were the pioneers of remote patient monitoring and chronic care, the technology side of that. And we were the ones that got the national contract with the VA, the Department of Veterans Affairs, which was the largest health system. And then based on that data, we got our model and our ideas into an act of Congress.

8:09And that led to getting into Medicare. And we were very early in kind of paving the way with new technologies in health care. But everything just took forever. Yeah. What was that company that you were doing the health care work with? Yeah, it was called Health Hero Network. And it was really, I mean, we were focused on the patient in developing applications where the patients could be monitored from home in a way that kept them out of the hospital because you would identify problems early. But really, kind of the heroes in the picture were the nurses and case managers who are kind of the front line of chronic care.

8:52And we had a system where, on one hand, we're monitoring for patients. and a lot of kind of education and feedback and behavioral stuff for patients. Like, you know, every day asking people how they're feeling and collecting information about symptoms. And on the other side is a management system related to chronic disease and identifying problems early. Yeah. And then so when you went when you became ill, you went back to that software. And instead of pointing it at the patient for the doctors, you pointed it at the patient yourself in this case for your own management of care. Is that right?

9:38Well, I didn't go back to that software because that software was old news by the time I got sick. But I had I had been since then, since the pandemic and over the last three years, I had been developing all kinds of new applications, AI applications, applications that were using the APIs of the foundation models and doing pretty extensive ideas in a lot of different fields in education and entertainment and, you know, things related to movies. and I was doing a lot of different things with AI. I just wasn't doing healthcare. But the, so I had a lot of experience. I'd accumulated a lot of experience of developing agentic AI applications at the point that I got sick.

10:34So I, you know, when I'm in the hospital, I'm, when I started feeling, you know, like, you know, I'm on pain medicines, I'm on Oxy, but I'm like writing code. So I started writing code to take and organize my medical record and create kind of a context and kind of perspective of different medical perspectives. So, you know, I had I had a oncologist and a hematologist and a gastroenterologist and a cardiologist and an emergency room doctor. I had all these doctors. So what I did is I made agents kind of modeled on all of my doctors and I had them all like analyze my medical record and then debate each other about my medical record.

11:20And the interesting thing is, like everyone had a different opinion about what it might be. But they all agreed that in order to settle this argument, you should go get this test and that test. And that turned out to be exactly what I needed. Now, I was doing this after I was already diagnosed. I basically went back in time with my medical record and asked the question, why didn't they catch this sooner? So I was looking at my prior data and saying like my doctors at the time were saying it's just stress. It's just gas. You know, it's like it's something that's it's no big deal. So, you know, I was I was, you know, might have been a fishing expedition, but we were fishing in the wrong pond.

12:02But I went back and I said, well, why didn't they catch this earlier? So I took all of my old labs and imaging and everything that I had, and that's what I had the agents looking at. And everybody agreed, everybody meaning all of these agents, they all agreed that I needed certain tests. I needed a free light chains test, and maybe I would need a bone marrow biopsy. So that was very enlightening to me because if I had known about that test, I would have asked my doctors, I would say, hey, what about this test? and they would have given me that test if I would have asked for it because it would have made sense.

12:38They just didn't think of it at the time, but they would have given me that test. I probably would have been diagnosed, you know, maybe even a year earlier if I had known what to ask. So, you know, like in health care, you know, you if you just walk in and you don't advocate for yourself or you don't ask any questions, you just say, do what you're going to do. You're going to get kind of whatever's in the checklist most of the time. I mean, you know, like the doctors, my doctors are great and they care, but, you know, they're also like stressed and they've got a lot of patients and they don't have a lot of time.

13:10So the default for most people is you're going to get whatever's in the checklist of what you're supposed to do, given what we know. And, you know, there's not going to be a huge amount of digging deeper until things get, you know, much worse. But if you, you know, like, you know, your doctor has like a lot of patients and there's maybe 10 minutes or five minutes to deal with you, but you only have you. So if you have something going on and you want to make sure that you're getting the right care, you need to become a bit of an expert in what's going on so you can ask the right questions. So I saw that for myself and I realized, look, I need to become an expert in my disease if I'm going to get the best possible care yeah and the agents this initial pilot or whatever that you developed that was looking at your medical records from before the diagnosis you said you trained one on to act as each of the specialist, I imagine.

14:15How do you do that?

14:19Are you fine-tuning a base LLM? Is it an open-source LLM and you're fine-tuning it on hematology or on oncology? How do you develop these agents, just not in the product that you have now, but when you were working this out, what were those Asian agents based on? So if you hear, you know, you have an audience of people who know a little bit about AI, I assume, but that you hear these, you know, this new model came out and it has a million token context window. What the heck does that mean? That means every time you interact with the model there's a potentially like you know like 750 000 words you know like like hundreds of pages of documents that can go in with that prompt you might be prompting something let's says hey what's going on you know hey what's going on four words but if in the background of that prompt is you know 100 pages of your medical record you know it's setting a context so you're not actually training the model on your medical record.

15:34What you're doing is you're organizing the context for your question. And that is part of what is prompting the model. Now, there is such a thing as fine-tuning the model, adding your layer of intelligence on top of the model. We're not doing that yet. But the first test was what kind of training already is there in the foundational models. And can we get to it in a way, can we get to the medical knowledge in a way that we can trust? Because one of your challenges, you go to ChatGPT and you can cut and paste stuff from your medical record and put it in ChatGPT and say, what's going on? There's nothing that stops you from doing that, except when you've got cancer, you might have a really long, complicated medical record.

16:19So it's really not feasible to do that. And their terms of service say, don't do it. because there's a lot to organize and there's a lot of kind of quality assurance that's needed. So my methodology of getting to a more kind of reliable result was to rather than saying, hey, every time I ask a question, I get a little bit different answer rather than a deterministic response. Well, let me intentionally try to get a diversity of answers, you know, like I would with a board of directors or a tumor board or a medical advisory board. Let me get a bunch of different opinions and have them kind of cross-validate each other.

17:00Let's have them sort it out. And it turns out that when you do this with a mixture of agents, with a mixture of point of views, and you get a diversity of ideas, when you find convergence around ideas, that's pretty important. Now, it's one thing. It's like, you know, like if you've got like a, you know, whatever chance of getting some sort of hallucination. I mean, this is a problem that the foundation model companies have been working on like crazy over the last couple of years. And so it's gotten really, really good. But let's say there was, you know, like, you know, some finite chance that, you know, something is a hallucination.

17:39Well, the chance of getting two agents or three agents or four agents or five agents to have the same hallucination is almost zero. So, you know, this is a methodology of getting to a more reliable result. And I think some of the models are doing some of this kind of mixture of agents in the background. I mean, they're doing that. You don't even know they're doing that. But we're doing it kind of intentionally and kind of transparently because in the real world, you have a bunch of doctors. In the real world, they have a difference of opinion. Like, who are you going to listen to? I kind of want to listen to them all, get all the ideas, learn what you can about what all of this means for you.

18:21And, you know, like if you have cancer, there's a whole bunch of things where they're saying like, hey, we could do this or we could do that. Steve, you decide. Right. You know, like how do you make a decision like that? You need to get educated. So what we're building now, it's really in this category of like, let's help educate you. The doctors are diagnosing and treating you, but it really helps to get educated. Yeah. Now, I have a couple of questions. So you're architecting this mixture of agents. The agents, you're feeding them context in the context window, not using RAG or necessarily fine tuning.

19:04there's there reg means you know what am i taking something out of a database somewhere you know whether it's a vector database you know or you know there's sort of a semantic search with all these ai tools or it's just like pulling data out of a database rag means retrieval augmented generation what and what how am i going to augment what goes in there so when i take your medical record and i kind of format the most recent relevant things you know i don't need data if you don't need your metabolic panel from 10 years ago i need your most recent stuff but i need your you know like when you kind of take that um out of this sort of messy large medical record and and compress it down into a really organized way that is a form of rag that is going in okay so that plus a pretty extensive uh you know rag puts things in the context window that's right right right but with search i mean you're not taking all of your your curated medical documents and dropping them physically into the prompt you're you're having a rag search for the relevant i mean no well it's a i mean so so rag you know there's there's a lot of different versions of rag i mean if you just when you know whenever the whenever you go to um you know chat chiptina says hey searching the web you know that's that's rag it's searching for some stuff it's going to add that into the context it's not in stuff that's not in the foundation model um anything from your medical record, obviously that's not known to the model.

20:33That's new data that came out of a database that is data that's specific to you. That's a form of RAG. We retrieved it from your medical record. We actually did a lot of organizing it to make it processable by the AI because you can just dump it in. You need to organize it. You can call that RAG, but everything that we're doing, whether it's coming out of your medical record or the guardrails around how the agents are supposed to behave or the perspective of the agents themselves, that's all organized into the context. Yeah, I understand. And then agents, in what way are these agents? simply because they can look things up or, yeah, I mean, as opposed to just an LLM that you're having another LLM talk to.

21:37Right. So, I mean, agent is a term. Here's how to think about agents. Agents, that comes from agency. It depends on how much are you trusting them to do things where it wasn't exactly what you told them to do. The minute I take two of these agents and have them talk to each other, you know, that that's, you know, like they're prompting each other, you know, that's an agent. Now, you know, it's a lot of agents means it means there's some set of actions that they have some level of autonomy that they can go have agency make decisions and do things. But certainly when when they're talking to each other, that's an agent.

22:13Now, you know, but there's a spectrum from, you know, just a, you know, chat bot to, you know, something that's going and autonomous and going and doing a bunch of things. We're somewhere in the middle right now. And the agency part is talking to each other, cross-validating each other. And that's where agent comes from. But as over time, this evolves and there are more tasks that we allow the AI to do to help you do, you know, it becomes more and more agentic. So agentic is a spectrum of agents. And some of those some of those features, I mean, you're like, you know, there's one aspect of like, I need to get educated.

22:55What does this mean? I had this genomic test. What does this mean for me? Like, what is, what, what do these terms even mean? It looks like Greek. You know, a lot of it is Greek, Greek symbols in these three words. Like, like, how do you even know what that means? Like, there's a lot of it is just, you know, like it, that's chat. But there are a lot of other management tools. You know, if you get a serious disease like cancer, and this is true of, you know, pretty much anything serious, it's not just, you know, you have a lot to manage over time. And, you know, there's a lot of lifestyle factors and, you know, like, you know, if you're putting off diet and exercise and sleep and stress and all these other things, well, you know, if you're in chemotherapy, you got to worry about those things.

23:33I mean, they interact with your treatment. So, you know, and if clinical trials, it's not just, Hey, is there a clinical trial that I match for, but, you know, can I, can I monitor those clinical trials and can I get notified whenever there's something new that I, that I match for And there's AI that's determining the match and whether or not you qualify or might qualify. And so there's a lot of management and work to be done. So there's a lot more pieces of that where the kind of the grunt work of it can be offloaded to AI. So those are all part of the features and roadmap of what we're doing.

24:07But the on-ramp is I've got questions and I need to figure out what's going on. Right. And the medical data that you organize, when you say organize, are you curating that data? And are you, you know, putting it into like a graph, knowledge graph or into a vector database? What do you mean by organizing? And is that something that a layperson will have to do once your product is in general availability? Think of it this way. When you go to a doctor and if a doctor refers you to another doctor, they do a write-up or work-up on you and say, hey, this is Steve. Here's what's going on with Steve. And then here's a few pages of stuff about Steve.

25:05you know and it's you know they're not going to put in you know the fact that I had a flu shot 20 years ago they're not going to put in stuff that's not relevant to the the issue at hand they're going to try to be complete you know we're basically reproducing that well if you just pull the electronic medical record I mean there's there's a bunch of stuff in there that doesn't have anything to do with cancer there might be things that do have something to do with cancer but, you know, we only care about the more recent values, not the 10-year-old values. So, you know, there's a, there's a, and then we have a chief medical officer and we've got a lot of medical input on this.

25:44It's like, the real question is, what is the, what is, what is the kind of out of your medical record, what is the kind of essential information for asking, you know, and what's going on. I mean, it's not everything. It's a lot. And we don't want to miss anything that's relevant. But if you look at the, you go to my chart and you go like, you know, look at it. There's a lot of data and it's scattered in different places. There's a lab section and there's the visits and then there's a click on something through the note from the visit. And, you know, there's a lot of stuff and it's kind of all over the place.

26:31and how do you know? We need to categorize all that stuff. We need to put all the labs together. We need to just care about the recent stuff that's relevant. There's another category of genomics and molecular profiling in cancer. That's a special category. So it's more like categorizing and organizing your personal health record so that it makes a little more sense. It's not just a data dump. When you have a medical record, you pull your medical record, you get this like massive JSON file. It's kind of a data dump. That's not particularly useful. We need to organize it. Yeah. And are you automating that organization so that you can put in the URL and

27:18credentials for your MyChart and it goes and pulls everything? It's not putting it in MyChart credentials, but after the Cures Act, you're legally entitled to get your medical record. It's your record. And these things are, there are health information exchanges and these things exist. And a lot of this work, I remember what you mean years ago when I was doing healthcare information technology and remote patient monitoring and chronic care, all the conferences, all everyone ever talked about was interoperability and standards and all this stuff. A lot of that stuff has kind of been, you know, I think a lot of people in the industry will say, oh, it's still a mess.

27:59But a lot of that stuff has been worked out. And there's pretty good electronic medical records. The data is there. You still need to organize it. And the fact is, if you don't have it and all you get is a, you know, PDF that was a fax from a lab to whatever, but then we can OCR that and analyze those documents as well. Yeah. And then how many agents do you have? Does that expand and contract depending on the complexity of the case? And what foundation models are you using? Is it like you've got one based on Claude, one based on chat gpt one based on grok one base you know like that or or uh do you yeah i mean how does how does that work the model is independent from the actual context management so you can the models are interchangeable uh in that sense and and when i was doing this for myself and you know the first version of this which is in a private beta with just people that were are able to work with closely because it's still in development.

29:18All the models are in there. I mean, all the foundation models are in there. Too many models are in there. But as we go now to market with it, we're narrowing that field to the major foundation models that we can do in a HIPAA compliant way. So, you know, that's Anthropic and OpenAI and Gemini are the main ones. But all the other ones work, whether or not they add, it's kind of like if you go to perplexity, for example, it's like you can go to ChatGPT and you're just dealing with open AI. If you go to perplexity, it's kind of the same thing, but you can decide, hey, do I want to use open AI, Anthropic?

29:57You have a choice of models. So right now, there is a choice of models, but we're not expecting the average person to have any idea, well, which one should I choose? So we're doing some work there to kind of, to sort that out, to make it easier, to kind of have a default mode where you can, you know, there's an easy way to just get going. And we, you know, we made like 36 agents, but, you know, we based on your, kind of the summary of your condition, we recommend, you know, here's five that would be good for you to use. Like, you know, like, you know, if you've got something going on with your liver, you know, then you should add a hepatologist in there.