In the inaugural episode of Health Policy Forum: In Conversation, host Paul Costello talks with Michelle Mello, JD, PhD, a professor of health policy and of the law, about where AI in health care actually stands—and why it looks so different from the AI dominating headlines.

Mello explains that adoption so far is concentrated in two seemingly unglamorous, but hugely helpful categories: ambient scribes that transcribe clinical visits, and administrative tools that handle billing and insurance approvals. The tools that generate the most excitement—such as clinical decision support and risk prediction—remain rare in practice. 

Meanwhile the hard policy problem isn't the technology but governance: whether every hospital, not just the best-resourced ones, can evaluate and monitor the flood of tools being marketed to them. Mello also describes how listening to patients changed her mind about disclosure, why clinicians are right to worry about liability, and what she'd tell Congress to do first.

This is white space

TRANSCRIPT

Host Paul Costello

Welcome.  This is Health Policy Forum: In Conversation, a Stanford Health Policy podcast.  I'm your host, Paul Costello. We'll talk with an array of experts in health policy about what's in the news and emerging issues in health care that will impact all of us. We hope you'll follow us and check out our webpage that you can find in the show notes.

In this inaugural launch of Health Policy Forum: In Conversation, I'm pleased to welcome Stanford Professor of Health Policy and of the Law, Michelle Mello.  Dr. Mello is one of the nation's leading experts on health policy.  She has her eye on what's happening now and also what's around the corner. Thank you for being with me, Michelle. Let's get right into it. Thanks for having me. Michelle, if you were to paint a picture.

Where are we today in the world of the implementation of AI and healthcare?  What is proceeding aggressively and what's on the back burner right now?

Michelle Mello

I think we're very much in the early days. What we are seeing is wide, pretty widespread adoption among higher resource organizations of two kinds of AI and not as much adoption of other kinds. The two kinds that we see a lot of are AI scribes; these are ambient tools that listen to a conversation like in a clinic visit, create a full transcript, and then summarize that transcript for the doctor. And administrative tools that do things like handle insurance approvals, prepare and submit bills, and automate other kind of back office operations within hospitals. The kinds of AI tools that people get really excited about that might really improve quality of care like risk predictors or tools that help doctors make clinical decisions are not as widely implemented at this time.

Costello

How are doctors and patients feeling about the scribe potential? If a physician walks into a room or a nurse walks into the room, nurse practitioner, they say, can we record this? How are patients liking that? And what about physicians using it?

Mello

Well, of course, it's hard to know how all patients think about it without survey evidence, but I think in general, it feels a little weird to a lot of patients. More and more of us are becoming accustomed to seeing that in the examination room, but it's still, from a patient perspective, new and kind of invasive technology to encounter in the doctor's office, something that's listening to everything. And the patients that I've interviewed about these kinds of technologies have questions about where the information from that conversation goes, who keeps it, and how it's used at the same time that they recognize it has a lot of value for doctors who are trying to save time.

Costello

Does it change that sense of intimacy between the physician and the patient?

Mello

You know, I think there are mixed reports about that. In one study, about half of the patients who had had a scribe used in their care reported that they got more face-to-eye contact with the doctor. So that's a good thing. On the other hand, why is it only half? This is something that's taking notes instead of the doctor having to type into their computer and yet half of the patients didn't think they got a more intimate connection at all.  And as you I think you were suggesting it could have the opposite effect as well that patients feel like there's a robot listener in the room, they might have greater hesitation about what they say. 

Costello

What do you consider the most significant public policy issues about the use of AI in health care?

Mello

I think one major risk is cybersecurity.  that, you know, the good news is that healthcare organizations are really on top of that and they're getting a lot of help from private sector firms. The other issue that I worry about that is much less readily solved is how do we govern the huge amount of AI that's being pushed out to healthcare organizations in the market and make sure that all healthcare organizations, not just the best resourced ones, are in a position to be able to intelligently evaluate all these marketing pitches that they're getting, decide the tools that are best for them, and then monitor them to make sure that they're used safely. We're learning that that is expensive and tricky, and absent regulatory requirement, it's hard to imagine that all healthcare organizations will do it. 

Costello

And when you consider the cuts in Medicaid for rural hospitals, you would think that this is an added burden on them. How will they work around this, the financial aspects of this?

Mello

Yeah, I think that the way that a lot of AI developers would answer that question is to say, “Oh, we have so many tools that will save you money. And so yes, it might cost you money to license our product and to monitor it, but you're going to save hugely also.” And that is probably true for some kinds of tools, particularly the tools that interface with the process of creating and billing insurance claims. It's not true for a lot of other tools. And so, for those hospitals, really the best solution that we've hit upon so far is sharing information out. So, there's a lot of really good work going on at Stanford Health Care and other leading health systems that are large and academic oriented about what these tools do in healthcare, what do you need to be on the lookout for, what questions to ask. And the good news is that those organizations are very actively sharing out that information so that not everybody has to start at zero when they're trying to evaluate a new AI tool or figure out how to monitor it.

Costello

You mentioned regulation and I'm wondering, the federal government right now seems more focused today on innovation and less on regulation. And in that vacuum, who proposes the guidelines and the guardrails for AI in health care? 

Mello

Well, there are government agencies that are doing so. They're doing it on a voluntary basis. For example, NIST, which is a standard setting arm of the federal government, has developed a voluntary certification program that AI developers can participate in by meeting certain standards. They can get a gold star, so to speak, and be able to advertise in their marketing materials that they're compliant with these standards. And then there's an organization called the Joint Commission. They actually have an important role with the government. They inspect and certify hospitals for participation in the Medicare program. They have also developed a voluntary auxiliary certification that you can get for responsible AI governance. Again, don't have to, not a condition of Medicare participation, but you get the gold star. And that has been in partnership with a private organization called the Coalition for Health AI, which is a big umbrella organization for a couple thousand health systems, tech companies, insurance plans, and others that have spent a couple years now developing best practices for vetting and monitoring of AI tools.

So, there are these private initiatives, they're kind of interlacing a bit with the government, but right now most of what is happening is voluntary and anything that is voluntary and costly will have uneven uptake. 

Costello

You mentioned a few moments ago the work being done at Stanford and I know that you've called it landmark work. What is being done here and what is Stanford trying to do to make sure that AI is safe use responsibly and is fair to our clinical workforce. Can you explain that?

Mello

Yeah, well, when we talk about Stanford here, we mean our hospital system, Stanford Health Care. That's a separate organization from Stanford University, but there's a lot of cross pollination in faculty who have appointments in both. And what has happened is that hospital leaders, at a pretty early stage, recognize that number one, Stanford being Stanford, it's going to be a hub of innovation and experimentation with frontier technologies like AI. So that was going to mean tons of things being proposed and tried here.

And number two, it had to have a process for wrangling all of that activity into a manageable, governable space. And so pretty early on, in terms of what's going on nationally and even globally, it developed this quite robust governance process. It had the foresight to say, we’ve got a great lever here for making people submit their things to governance, which is that we will not let our digital services unit turn it on until you can certify that you've gone through this process. So, anybody who's got kind of a patient-facing or otherwise potentially risky tool has to go through this governance process. The tools get a review for their clinical utility, for their financial sustainability, and through my team's work, their ethical risks.

And the goal here is to try to proactively spot problems before the organization falls in love with the technology and puts it into use, and then to figure out in advance what is important to measure about the impact of this AI and how we're going to do that?

Costello

One of the things that I would think health-care providers around the country seem to have very different practices around what they should tell patients and when they should tell patients. What should patients be told about the use of AI?  And do you think that this should be uniform and who would make it uniform? What's the regulatory process?

Mello

Yeah, those are great questions. And as an ethicist, my priors coming into working in this area were, you know, that patients should be told about everything all the time so that they can be active participants in their care and make good choices. I've come to have a different view over the last couple of years, mostly as a result of talking with a lot of patients.

As part of the ethics governance work that we do here at Stanford, we have a patient partner panel of folks who have been trained in AI and who speak with us in a group format every month about AI tools that are being considered for adoption here. And their attitudes about being told or occasionally asked for permission to use tools in their care really depend on the tool. But in a much larger share of use cases than I would have guessed, they're not really focus on notification. They are much more focused on the question, what is the organization going to do to keep me safe? And, I’ve heard them say things like, you know, when I'm in the hospital, I got a lot going on, I'm sick, I have piles of paperwork pushed at me. The last thing that is helpful in this situation is another piece of paper asking me to sign on the dotted line. It feels like a waiver of responsibility, like they're just trying to cover their bases. More meaningful to me would be to know that there is a healthcare provider in the loop of this AI that's going to be reviewing it and that they have been kind of set up for success in that task by the organization, that they've created the time and information that the person needs to do that job. And that the organization itself has a good process for vetting these tools and is choosing and monitoring carefully.

But there are certainly exceptions to that general feeling like you know, if you've got a choice of surgical modalities, for example, one uses an AI guided robot and one doesn't, that's a clear case for informed consent. In California, an AI scribe is something you have to get permission for, not informed consent in the medical sense, but permission, and that's a wiretapping law, it doesn't have anything to do with healthcare, but there are exceptions. But the general rule I think should be, particularly because in a hospital like Stanford, you might have like a couple dozen different tools that touch your care in various ways. I think the presumption should be rather than using a notification to shunt responsibility onto the patient, we ought to be preserving that process for the most important, highest risk tools where they have a choice, you know, there's something they can do to opt out of the tool or take action in response to its use and otherwise really focus on our own safety and responsibility.

Costello

As you well know, there is a growing fear about AI outside of health care. Can it be contained? A recent column in The New York Times stated, “If there's ever a reason to doubt the dangers of unrestrained artificial intelligence, those reasons have gone out the window. What's next is a vital window of opportunity to slow down AI progress so we can better understand these models and avoid the calamity that may be on the horizon.”

And I'm wondering, how do you think that this writ large concern about AI is leeching into healthcare and the concerns that patients and physicians will have?

Mello

Well, the spaces that I hear about in healthcare have to do with cybersecurity. So that AI agents, whether it's something that is in use in healthcare, that's gone rogue or more likely some external agent, would be hacking into patient data and causing problems. That's one, and I think that's a problem in every sector in all settings for all of us all the time. And the other is that as we move increasingly from AI products that are human mediated, meaning there's a human being who reviews an AI suggestion and acts on it, accepts it or rejects it or modifies it, to systems that are agentic, meaning autonomous, they don't have a human in the loop, that those systems might get things wrong in ways that actually can physically hurt people very quickly and in a way that's very hard to intercept.

And that makes healthcare a little bit distinct from other contexts where we might worry about autonomous AI, like, I don't know, human resources or investments.  We're talking about physical injury happening very quickly here. And so, there is, I think, a heightened concern about being able to meet the challenge of vetting and monitoring those agentic tools as they come online. 

Costello

You testified before Congress quite a bit. If you were to advise a member of Congress on legislation to regulate AI, where would you start?

Mello

I think the most important thing that we could do at a federal level in the healthcare space would be to require healthcare organizations to have a governance process for AI as a condition of participating in federal health programs. We require hospitals and other facilities to have all kinds of other structures and processes in place to ensure that patients are safe and receive high quality care. And I just think we've reached the point where we cannot justify an ecosystem in which hospitals can operate AI tools with very little vetting and no oversight and have no regulatory consequences.

More broadly, I think the thing that is rightly occupying Congress is that much harder question of what you do about frontier models, general, you know, all-purpose models like chat GPT and the other modern LLMs that, as you've sort of alluded to already, have such far reaching potential impacts and harms from the kinds of cybersecurity risks that we've talked about already to, you know, more granular, but already present harms like mental health harms to kids. And I think that landscape is extremely difficult to figure out what the regulatory strategy should be because they do so many different things. 

Costello

Let me close by asking you, what is the one question that clinicians ask you about AI and what is the one question that patients ask?

Mello

By far the most common question I get from clinicians is, am I liable for mistakes that hurt patients if AI was involved? And that answer is not straight forward, but usually the answer is probably, yeah, you are, because developers of AI typically include provisions in their licensing agreements that immunize them from liability and disclaim ordinary consumer warranties that people usually make about their products. And so, liability would really rest on the physician, the nurse, and perhaps also the hospital and that makes physicians understandably leery about using AI tools, especially when they're not given a lot of information about how to be good overseers of the AI. 

I think patients just tend to be curious about what's going on. Like, what don't I know about how AI is being used in my care? They just have a curiosity about how these technologies that they hear so much about are actually being used on the ground. What are the benefits? And again, how is it that we're going about trying to keep them safe? But I think there's increasingly, at least around here in Silicon Valley, where we're all kind of AI natives, more of a sort of hopeful curiosity about AI and healthcare than there used to be, or that might be the case in other parts of the country.

Costello

You mentioned here in Silicon Valley, just the posterboards along 101 of AI are overwhelming.

Mello

Yeah, absolutely. You can't drive for more than 30 seconds on Highway 101 without seeing a billboard promising that agentic AI is going to change the world and make our lives better. 

Costello

Thanks for joining me today, Michelle. I really appreciate it. I want to again thank my guest, Dr. Michelle Mello. I hope you enjoyed the show.  If you liked it, tell a friend or a colleague.  This is Health Policy Forum: In Conversation. I'm Paul Costello. See you next time.