A New Operating System for Physicians with OpenEvidence Founder Daniel Nadler

0:05Daniel, thanks for doing this. Happy to be here. So give us a sense of this incredibly viral sensation that has been open evidence in terms of what type of coverage it has of American doctors today. As much as we would like to think that it's going especially well for us, I would sort of say as a qualifying point that in all of the sub-industries of AI, you're seeing an acceleration and compression, right? So the adoption cycles, even outside of open evidence before we get to open evidence, in other fields of knowledge work and coding and so on are hyper compressed. It used to take half a decade or a decade for something to become standard.

0:47And now it seems to happen in two years or a year. So the same things happened with open evidence. In about 18 months, it's become the operating system for clinical knowledge in the United States. It is used something like 20 times more than the next most used platform of any kind in our specific segment, which is high stakes clinical decision support for doctors. So high stakes clinical decision support for doctors is a specific category of medicine. It's distinct from, say, paperwork or it's distinct from scribing. Those things are part of the workflow of being a doctor, but the stakes and the consequences are different.

1:24If you get it wrong, you can go back and do it again. that's not the case with a patient. You have to get it right. You have one shot to get it right. And so clinical decision-making, of which clinical decision support is in service of, is unquestionably the highest stakes area of medicine. We're probably the only company working at the tip of that sphere. Most people have self-selected themselves out of the problem of high stakes. Clinical decision-making, certainly through an AI lens, because they view it as ambitious. And could you explain it, Orda, or Lianz Piers? I think fundamentally it's about picking information and then translating them to specific either recommendations or diagnosis for a patient.

2:03Can you tell us more about how that works? Yeah. One way to sort of simplify it down is at its foundation, it's a search problem, but it's a very semantic search problem. So most search traditionally works with keywords, right? So like, you know, flights to Barcelona or hotels in Barcelona, most of the, you know, most of the keywords there can be captured in a couple of words and certainly in a sentence. And that's sort of traditional Google search. Even if you were to think about clinical decision support as a search problem, simply describing your search query, if you want to think about it that way, usually takes many sentences.

2:35So an example I like to give is you have a 44-year-old female patient. She has moderate to severe psoriasis. That's the red stuff on your skin. You know, you're a dermatologist. That's so far so simple. You would just prescribe one of the many creams you see commercials for on television, except But she has MS. So now it gets interesting because you want to treat her psoriasis, but you don't want to make the MS worse. And you are not a neurologist, you're a dermatologist, so neurology is not your specialty. But you don't want to go refer to a neurologist because you want to treat her psoriasis.

3:10And if you just keep referring people in circles, medicine never happens. From the ether, you might have heard as a dermatologist that the new classes of psoriasis treatments, which are biologics, they're IL-17 inhibitors and IL-23 inhibitors, might have some interactivity with the neurological dimension of a patient's condition. That's about all you know. You didn't learn this in medical school because IL-23s were FDA approved in 2019. And so one of the great themes of open evidence is that the golden age of biotechnology is sort of the dark ages of physician burnout because it's just impossible to keep up with all the new drugs and all the new mechanisms of action and so on.

3:46So it was approved in 2019. You might have graduated medical school in 2005. So you didn't cover the medical school. And that's it. That's what you know. So your question then is, for a 44-year-old female patient with moderate severe psoriasis, is an IL-17 inhibitor, an IL-23 inhibitor, more appropriate and more safely tolerated with respect to not aggravating the MS? Now, that's not an academic question. That's a very consequential question. IL-17 inhibitors will actually make the MS worse. IL-23 inhibitors are safe and well tolerated in case of MS. That's an example of where medicine can go wrong.

4:23Because even five or 10 years ago, either you're referring that person to a neurologist, in which case you're just getting referrals in circles and medicine is not happening. Or unfortunately, what would more likely happen is they would just 50-50 and that MS might be aggravated. And it's well known and it's been often repeated that medical error is a third leading cause of death in the United States after heart disease and cancer. But even that statistic kind of understates it because that's just looking at death. In my example, this patient is not going to die as a result of taking an IL-17 inhibitor.

5:02She's going to have a relapse of MS. And so it's not just that medical error historically was a leading cause of death. It's that as many people died from medical error, probably a factor of 10 to 100 as many people had a comorbidity or condition that became aggravated and got worse and so on. So coming back to your question, that whole string is the search query. And so you can't just do search in a traditional way where you sort of say, you know, aisle 17, because that's not really what the question's about. about, nor does the physician have the time to go read book chapters on this stuff. What you need is a semantic understanding of the query in the way that another human physician would semantically understand that query.

5:48And then it's actually quite deterministic and simple after that. Once you semantically understand the query, you can, from the world of published biomedical literature, you could find the exact snippets in a phase three RCT, a randomized controlled trial in the New England Journal of Medicine, that tested each of these things and found that one aggravated MS and the other didn't, right? So once you have a semantic understanding of the query, the rest is fairly deterministic, and it's almost a search problem. But all of the juice is in connecting the very complex semantic meaning of a medical scenario to the answer, where the answer might be in a phase III RCT in the New England Journal of Medicine, in a snippet, not even in the abstract, but in the methodology section or in the population.

6:35I didn't deal with that ambiguity actually because I feel like in the context of that information, there's things that are in pre-bay clinical guidelines, you know. Yeah. Certain types of conditions, we're going to do XYZ and that's where the recommended path. There's stuff that's kind of recently published, there's evidence in a certain direction or maybe it's by the label or something else. And there's a bunch of stuff that's a bit more TBD in terms of those kind of trials that they kind of predict each other a little bit or maybe other information that may be a bit more sporadic. How do you deal with that third bucket of ambiguity and how do anything they can tell about capturing that broader knowledge process over time?

7:05So the first way to deal with that third bucket of ambiguity is ensure that your users are physicians and not patients. And we've made that strategic decision. And we keep thinking we're going to change that decision. And we've been talking about changing that decision since the inception of the company and so far have not changed that decision. For all the reasons implicit in your question, there's an enormous luxury that we have as builders in having doctors as users because the MD is attached to their name, right? So they need to protect that MD. And they're going to use us as a tool in the same way as a Wall Street trader might use a Bloomberg terminal.

7:38If a Bloomberg terminal, for example, produced, you know, an inaccurate quote on a bond, that was very obviously inaccurate, you know, it was off by an order of magnitude. And the trader, you know, in a hedge fund, just sort of, well, I mean, that's odd. Do you indicate in the user in our case that pay there's some ambiguity around this or there's conflicting evidence and here's the... Absolutely. So there are areas of medicine where there's a lot of conflicting evidence and that's indicated. And it's not presenting answers. You know, we're used by 40 % of doctors in the United States daily on average.

8:06It's about 20 times as much usage as the next thing that could be described as a clinical decision support platform. It's become the default operating system of clinical knowledge. And a lot of the value proposition early on is that we made references and citations of first-class citizen before that was in ChatGPT. So we were actually providing references and citations six months or nine months before ChatTPT started doing that. That was a big reason we had adoption, because people could interrogate and audit the source, right? So right there, there's a difference, because then it's not an answer engine.

8:37It was never presented as an answer engine. It was always presented as a search engine. The way we did frame it was as part of the long continuum of search and Google. We're a Google portfolio company, and I've always framed this as part of the very long continuum of search engines, as opposed to something net new. because I do view technology as a progression and continuum. And that created a certain social contract with the users, who in addition to being physicians and have that MD that they need to defend, on top of it, viewed this as a router to the phase three RCT in the England Journal of Medicine, and maybe the conflicting phase three RCT in JAMA, right?

9:15And we'd route them to both. Very useful for attention. So users do look at source material some of the time? All the time. I would say that's almost the default behavior of a user to start with some complex query that you could not put into Google for the reasons I mentioned, because it's a paragraph long. And then have it produce from a search space or a surface area of 35 million biomedical publications, the exact three to five canonical landmark phase-free RCTs or guidelines or other sources of information that are responsive, not answers, but that are responsive to their question. And then I would say almost the default behaviors, then they go out.

9:59You know, I think we're one of the largest sources of referral traffic to the English Journal of Medicine after Google. The rankings, I don't know if they're number two or three or four, but we're one of the largest sources of referral traffic to our partner in the English Journal of Medicine. That's a testament to the way people use it. Historically, it was very hard to do two It was hard to describe a complex patient scenario or case into a search engine and have it come out with anything useful. And it was hard to find from the tens of billions of tokens, if you want to think of it as an engineer, that constitute the world of peer reviewed public medical literature, it's very difficult to find the seven snippets that are directly responsive to a question and into the semantic meaning of the question as opposed to a few keywords.

10:45So we just did those two things, just did those two things extremely, extremely well. We framed the right social contract. We picked our audience extremely well. And all of those things start to stack into something that looks more like a Bloomberg terminal for doctors, where it's just a pro tool. They're using this because it has the right data that goes in, because AI is gold in, gold out, garbage in, garbage out. So they know this is not training on tweets. They know this is trained on the England Journal of Medicine and JAMA and the rest. They know that we have these partnerships, these strategic partnerships with the gold standards of medical knowledge.

11:25They know that they're not going to get an answer from open evidence. They're going to get a routing to a source that answers the question. And so I think all these things sort of stack into something that feels just like a pro tool. I want to remind for a minute. You were already a successful entrepreneur before you started Open Evidence. You wanted to build an impact-driven company. You wanted to work in health. What was the moment of decision to serve physicians versus consumers? Because you also think a lot like a consumer entrepreneur in terms of growth. Well, I served both. So this was a hack.

11:59I wanted to build a consumer internet company for knowledge workers. And I don't think that had ever been done before. So I didn't want to build a health care company at all. I love Sequoia's quote that Open Evidence is a consumer internet company masquerading as a health care company. I had zero interest in building a health care company. Open Evidence is not a health care company. I wanted to build a consumer in a company, but I wanted to do something that no one had ever done before, which is treat knowledge workers like consumers. So my whole career had been, prior to this, dealing with knowledge workers.

12:31Right. And people have a reductive view of consumers. They think of, you know, they think of like 14 year olds on TikTok. And that tends to be like their archetype of what a consumer is. And it's one type of consumer. But, you know, traders on Wall Street are consumers and people. Lawyers are consumers and people. And doctors are consumers and people. And what I realized is no one had ever treated doctors that way before. Doctors were just kind of treated as these appendages of health systems. I was like, it's an interesting way to organize the medical system and the health system. And you start to investigate and pull the thread a little bit, and you start to understand why there are very few things that people can agree about in America.

13:16They can agree Congress is dysfunctional, and they agree that American health care is dysfunctional. It's like bipartisan universal consensus. But you start to really investigate, and you come across two or three things, and you're like, maybe that begins to explain the dysfunctionality, right? And to me in particular, the idea that doctors who were the fighter pilots, who were the knowledge workers, who were the people who have that MD on the line and have to make that high-stakes decision, weren't even their own gatekeepers as far as the technology they use. That was a pretty profound realization.

13:48And so we did something that had never been done before ever, which is we treated them as consumers and as people that could go on to the App Store and download a free app and start using it. And it sounds so stupidly simple, but it was really profound and it was really effective because no one had ever done that before. It's kind of almost analogous to in relationships, whether friendships or romantic relationships, people can get caught in these sort of cul-de-sacs where there's a rigidity to their dynamic and to their relationship. And then there's a breakthrough where one person says something that they've just never said it before, or they've just never said it in that way before.

14:29And then there's like a breakthrough. It hits different, right? And in psychiatry or psychology and therapy, a lot of that field is encouraging this behavior in others, is to just sort of break free of cul-de-sacs, of dialectics, of relationship dynamics, and just say something in a way that's never been said before or do something that hits different. And long story short, we did that with doctors. And it wasn't the complexity of the idea. It was just no one who had ever addressed them as consumers before. And we had this realization, which is pretty obvious, that while this wouldn't have been possible 20 years ago, today, virtually every doctor in America is walking around with a computer in their pocket that they own, called an iPhone or an Android phone usually.

15:19And they own that computer, right? That's really cool. Yeah. I mean, the velocity of it and usefulness and value is reflective in that velocity. The scale and the speed of it. And more common cases are cases in which the leadership of the hospital system are very avid users. So the entire, you know, this whole senior leadership of UCSF, of MGH, of Mayo Clinic, of Cleveland Clinic, of New York Presbyterian, Mount Sinai, Cedar Sinai, you know, right up to the chief medical officers, the chief physicians, and the CEOs in many cases are personally avid users. The reality too is that people are basically using Google for some of these use cases, or they're using a new tool as just word carrier.

15:58I have a sort of slightly separate question, which is maybe back to the consumer versus medical or physician side of this. Because, you know, I started a digital health company maybe a decade or 15 years ago. And one of the things, and we were basically initially providing really key genetic information. We had a physician of the loop at all times. But one of the things we ran into was what I kept trying as almost a journalistic viewpoint in the medical community towards what information their patients should and should not get. And I think part of that was real concern about what the patients could do in terms of insacting on information.

16:28But I think a lot of it was just wanting to be a gatekeeper or part of it was just not going to deal with the questions of the patient. How do you think about that philosophically in terms of what type of information should patients have access to versus not? How much should patients be able to advocate for themselves? So I've experienced both sides of this. So I've been on the patient side, and I'm very sympathetic to that because the reality is medicine is not perfect. If it were, everyone would be living to 80 or 90 years old. So clearly, medicine is not perfect. And in a world where it's not perfect, patients should definitely have some role and agency in that.

17:03What we have done is encourage physicians to use open evidence to generate patient handouts. And that's actually a very widely used secondary. It's mainly clinical report, but we have all these secondary use cases like prior authorization letters and insurance appeal letters. And one of the most common of those sort of secondary use cases is generating these patient handouts. The other side of this that I can appreciate is it took me personally taking my first graduate level statistics course at Harvard to really understand these clinical trials, right? And so I'm sympathetic to the idea that a patient simply finding some clinical trial published in the New England Journal of Medicine because it was mentioned on CNN or Fox News, and then going and trying to read it, especially through the lens of fear or hope, is not necessarily going to result in the most sort of constructive decision-making process.

18:02I mean, there's no good answer. The reality is it's very tough, right? You want to enable patients with all the answers that are clear in consensus. And certainly, you want to give them the tools to make sure that their physician is not missing anything. At the same time, you can imagine all the failed cases where that could go wrong, where they're coming and saying, well, why aren't you putting my mother on this drug with their own handouts? And the answer might be a very technical answer, right? The answer might be that because your mother also has this other comorbidity. And if you look at the p-value, the p-value of the efficacy of this drug is not statistically robust in the presence of this other comorbidity.

18:51And the patient is like, what's the p-value? But they're not going to just stop at what's the p-value. They're going to get really upset. It says in this case that this other treatment is effective. And then you're just in this endless circle where the physician, who has by definition taken at least one graduate level statistics course, is trying to explain to a civilian what a p-value is. And I think that's probably not a constructive outcome. So it's a balance. We encourage physicians to use open evidence to brief patient handouts, especially where guideline-based medicine is concerned. So I think you mentioned something really interesting earlier, which is the velocity at which your product got adopted was incredibly fast.

19:34And I think part of that was just it's incredibly valuable. And I was just a lot of these new and I've driven tools. And I think that's one of the almost underappreciated aspects of this wave of AI is not only is there a fundamental technology shift that's enabling all sorts of new products, but also there's this massive shift in terms of openness of adoption of people and organizations to new technologies. And that's in terms of what you've been doing with evidence. It's to your point of the medical scribing thing. its company like Abridge or Commure or others. If you think ahead 10 or 20 years, and this may be impossible to extrapolate, how do you think the change of medicine or the state of medicine changes in general?

20:08Are you still going to the doctor's office for visits? Are you interacting with some online tool when it's backstopped by a doctor? Are you trying to develop differently? I'm just sort of wondering at a high level how you think about the whole industry evolving or changing, given that some of the markets are open in ways that they weren't before, but also there's new technology ways that are going to impinge on markets. It's getting difficult. The definition of a singular event horizon is you can't even project into the near future, let alone the far future. And I think we're probably in the midst of something like that.

20:37With respect to doctors in the loop, planes have been able to land themselves for a very long time. It's a peek into, in a way, a future by analogy, because that's a domain or an industry where there's no debate, really, as to whether the technology is there. And yet you don't see this sort of mass movement of airline passengers to get the pilots out of cockpits. There just isn't. I'm not aware of one mass movement to get pilots out of cockpits. Then the question was why. And of course, that is a attribute of human psychology, that we are anthropologically tribal and we don't abstract trust well.

21:22We personify trust and we trust things that we personify and anthropomorphize. And there's a whole history. So already doing a lot with the chatbots, right? In other words, there are people who effectively view themselves as being in relationships with. Yeah, they don't have bodies yet. I mean, you can start to reason by analogy. Would there be any more of a mass public movement to have computers land planes if you still had a cockpit, if you just remove the two seats, no one wants that, okay? What if you keep the two seats, but they're empty? I still think no one wants that. What if you keep the two seats and there are mannequins, essentially, that act as visual surrogates for the computer system and what it's doing?

22:04I think if you were to poll people, that'd be the first time you see this little uptick in willingness. I think it would still be the minority. Can I ask a question if we're talking about the near future? You've mentioned before, like, we are in an era of, you know, in an amazingly optimistic way, like, explosion of biomedical knowledge. And it should accelerate. You've mentioned before that the half-life of the knowledge you learn in med school as a physician is decreasing rapidly. Do you think that's going to change, like, how you are educated as a doctor? I think medical education is going to radically change.

22:41I think doctors are going to be in the loop for a very long time. They have been a loop since the ancient Greeks, if not the ancient Egyptians. I think they're going to be in the loop for a very, very, very long time and for the rest of our lifetimes, if not longer. Medical education is going to change radically because it's just the statistic I cite, and all of this is in peer-reviewed publicly available medical literature. the rate of doubling of medical knowledge as measured by citations in 1950 was every 50 years. So every 50 years, the number of total citations of peer-reviewed medical literature doubled.

23:18Today, it's every 73 days by an estimate in the British Medical Journal and one in Nature. I think that methodology was a little bit aggressive because they were looking at the totality of all publications. Not all publications are equal. But we came up internally with a more conservative one because we didn't want to drink the Kool-Aid. So we said, OK, let's just look at the top quartile of peer-reviewed medical literature. And let's pretend that physicians never need to read the bottom three-quarters of medical literature, which is not really true. But let's do this with one hand tied behind our back.

23:55And if you do it that way, it's every five years. So if you use the more conservative methodology, it's not every 73 days, but every five years, the total sum of the top quartile of peer-reviewed medical literature by citations doubles. Now, you could say, well, look, luckily for humans, medicine has become specialized. So your dermatologist doesn't need to read everything in neurology. That was my initial example. And now they have open evidence so they can bridge some of this stuff. So why don't we go even more conservative still and say, if a physician just needed to read the top 10 % of peer-reviewed medical literature in their own specialty, so now this is very conservative.

24:40There's no cross-functional interdisciplinary medicine at all. Everybody's hyper-specialized. It's not a great outcome, but let's just pretend that's the case. What would that mean? Well, now you're in the realm of doable. Obviously, every 73 days and every five years is not doable. But now you're in the realm of doable, but that physician would need to spend on average nine hours a day just reading the top 10 % of peer-reviewed medical literature just in their own discipline. Of course, the university patients, they spend time with their family and so on. Now, you can sort of keep going more and more conservative with these methodologies.

25:13And realistically, not everything even within pediatric cardiology is relevant to every pediatric cardiologist. And so maybe it's not nine hours, maybe it's four hours. Maybe it's three hours a day. But there's some point at which it's going to be like you'd want them to know all this stuff, even narrowed down all the way. And it still is kind of impractical. At minimum, I think that this framework of medical school being a very defined period in time, and then having continuing medical education, which has kind of historically been in this sort of like, uh-huh, OK, sort of wink, wink kind of thing.