FEMALE SPEAKER: Welcome to Mayo Clinic COVID-19 Expert Insights and Strategies. The following activity is supported in part by an independent medical education grant from Pfizer Inc., and is in accordance with ACCME guidelines.

BRIAN PICKERING: Good afternoon, and welcome to our certain Metric 2020 lecture series. And thank you to my colleagues who've joined by phone. We're all, as you can see, in relative isolation from each other. And having six out of seven folks on one telephone call is not so bad. We're hopefully going to have Sonal join here in a few minutes. On my left here, Joe Levitt is Assistant Professor of Medicine in Stanford University Medical Center.

And a real interest in ARDS and pulmonary disease, and has been heavily involved in informatics, and setting up some of the informatics fellowships up in Stanford. So welcome. Vitaly Herasevich is my colleague in Mayo. He's Professor of Medicine and Anesthesiology who has a particular interest in applied clinical informatics, and expertise in delivering that into our health systems.

Erin Barreto is a phenomenal clinical pharmacist who works within the critical care space in Mayo, and has a particular interest of individualizing pharmacotherapy for acute [INAUDIBLE] patients. Mike Joyner, who's run away over there and has his red shirt on is a very, very well-known integrative physiologist, performance medicine research [INAUDIBLE] Professor of Anesthesiology at Mayo Clinic.

And Sonal, who has just joined us again, is on duty. Sonal, we do appreciate you joining us. You are in Ohio State University Medical Center in Columbus. And from the looks of it, you are in the ICU at the moment.

SONAL PANNU: I am. Thank you. Are you able to hear me?

BRIAN PICKERING: We can hear you very well.

SONAL PANNU: I am glad this is working. Thank you for having me. I'm very excited about this.

BRIAN PICKERING: Well, we can't escape anybody's notice that we are in the middle of a COVID-19 pandemic. I know I've been working harder than I've ever been in trying to set up some systems at Mayo to help with this. And I'm sure you're all in the same position. What I really wanted to talk about was how are our health information technologies that we've put in place over the past 10 years, how are they helping us respond to this pandemic?

And if it's OK, I'd like to restart with our EMR. Can I ask you, Joe, what are some of the things that you're seeing in your hospital that your EHR is enabling that maybe wouldn't have been enabled 20 or 30 years ago if we were faced with this situation?

JOSEPH LEVITT: So I can't really claim that we've used it to much advanced use. We certainly do have a screening list that are-- You can easily log in, we use the Epic system. So log into Epic and find a list of all of the COVID-19 confirmed, or patients under investigation. So that's an easy list for us to follow. We have had our fellows in the ICU kind of monitoring that list and making sure that they're closely monitoring those patients who are outside the ICU.

So that if there is evidence of deterioration, that we're figuring that out hopefully not when there's a rapid response, or a code blue call. So that we can move people to the ICU a little earlier because all of the issues that are surrounding intubation of these patients is challenging. So we want to make sure everyone has proper personal protective equipment, and we don't want any surprises.

So beyond that, I'm not sure we're using too much smart technology, and I'm sure others are probably using more.

BRIAN PICKERING: Sonal, you're in the ICU at the moment. Your EHR, is it helping or hindering your ability to look after these patients?

SONAL PANNU: I think at this point it's helping more than hindering. We have enough data to identify whom we need to be suspicious about. I think the speed of [INAUDIBLE] and the EHR interface is helping us identify high-risk patients. BPAs are, I think, helping at this point most for all multidisciplinary staff to identify what kind of procedures are needed to be done, what kind of protective gear that they need to wear.

How should care be bundled, we're using that to see that as well. How should interface with the patient be minimized as well to help protect.

BRIAN PICKERING: And Mike, you work in the operating room, and you also have a research kind of interest. Are you finding that having an electronic health record available to you is helping [INAUDIBLE]?

MICHAEL JOYNER: I got the res-- I got some underemployed residents working with the IV people in some sort of Mayo institutional list. We're already generating a spreadsheet of symptoms, and we just inserted some type of [INAUDIBLE] master order sheet about research-related blood draws after we give our convalescent plasma. And there's no reason that this couldn't be more widely shared with all the other institutions that are going to do this sort of thing.

BRIAN PICKERING: So what I'm hearing from the clinicians who are in the room here, and I'm not [INAUDIBLE].

MICHAEL JOYNER: [INAUDIBLE] I'm a hobby clinician.

BRIAN PICKERING: I'm not discounting. I want to get to the pharmacy issues in just a moment. But the electronic health records are really-- At the moment, we're very used to them. And we're using this institute to compile lists of patients that maybe are COVID-positive or negative, and using them in a day-to-day's work as we normally would expect, maybe use a paper chart or something else.

Erin, you're trying to match drugs to patients. And you're part of the multidisciplinary team. As we go through this COVID-19 pandemic, do you see any opportunities that we're not exploiting in the electronic health records that we should be looking at?

ERIN BARRETO: I think it's a really interesting time for pharmacists. I have a lot of colleagues who work in the informatics group, and they suggest that this is tantamount to a go-live with a new EHR like Epic with the amount of builds and things that they're encountering. And the things that we're finding out from the centers who are doing this well, and have higher volumes than we have in Rochester, is that it's coming up in lots of different places that we might not otherwise expect.

So for example, medication restriction or justification in some of these resource-limited drugs is one that we're seeing. For example, how are we justifying the use of inpatient hydroxychloroquine or azithromycin? And how is that being documented and monitored? Another example is how can we deploy clinical trial, medications into the EHR so that it can be identified that a patient is on-study, what drug that they're using, and how can we manage the safety checks?

And these are all things that are having to be operationalized extremely fast as the landscape around us is changing. And so I think certainly the EHR is helping us with those things. But there are new items every day. An example of one that we're finding out is to minimize the amount of individuals going into the room and utilizing PPE. We need to develop larger medication bags simply. We need to increase the total amount of drug that we're using so we can decrease the number of times that somebody has to interface and exchange the bag out.

And that means reprogramming the pump library, it means building new things in the EHR. So there's a lot of different pieces of the infrastructure that we're learning the more patients that we have.

BRIAN PICKERING: So it's that connection between the kind of electronic environment and the real world, and the people and processes that we're kind of struggling with and we're building around right at the moment. Vitaly, you've been intimately involved in building applications and tools for practice, particularly the acute care practice, which is the group that's getting most maybe requests for expansion of service.

Can you comment on some of the strengths and weaknesses of our EHRs right now to support expansion outside of the ICU to other parts of the hospital if we're asked to look after patients out of the ICU?

VITALY HERASEVICH: Yes. Yeah. As you can see, over the time, all EMR they're designed to manage single patient. They design around services and kind of the convoluted [INAUDIBLE] ability to treat population. I mean populations across the hospital. They don't have to build into EMR, which could allow see all this population together in a single place, and see movement around hospital, or so.

It's type of the technology. They exist in other industries, but never ever implemented inside current EMRs. And they're very focused on the single patient view, on the particular service and transfer of data between services very challenging and very undeveloped. And to add to that, a lot of this technology, they're not designed to direct patient outside hospital at all in community.

They're not connected to emergency services, or they're not connected to some public health systems or so. Very convoluted, very single patient-centered system.

BRIAN PICKERING: So I'm being challenged here to think about larger populations, and I'm thinking about moving teams of ICU physicians outside even beyond rapid response team calls. And I wonder if we could talk just for a moment about how we're identifying those patients that maybe might need intensive care review. As we anticipate larger more acutely ill patients sitting on the floor and deteriorating rapidly with this illness, is there anything in any of your hospitals or any of your experiences that are becoming useful tools that you're repurposing to help you identify those patients who are not doing so well a little earlier?

Maybe Joe, we could.

JOSEPH LEVITT: Yeah. No, I can see we're not, specifically are not doing that. I mean, certainly within the electronic ICUs where people, the sort of doc in the box where developed where one critical care physician can sit and monitor several different ICUs remotely, those systems usually have some automation built into them where they have vital signs and laboratory results organized by organ system.

And they can turn from green to yellow to red if there's some suggestion of problems happening to kind of help a person who's sort of monitoring multiple screens at one time pick up impending problems. I think that would be an ideal situation. I would guess with a little automation, it would be pretty easy to predict who's going to go onto respiratory failure in this group of COVID patients just by oxygen saturation and respiratory rate and heart rate.

And just seeing kind of slope and rise of those vital signs over time, we would probably have pretty good fidelity if we kind of had those systems kind of plugged in and ready to go. I don't know if any of your other centers are-- Or if everyone else has personal experience. I have not as far as being in the electronic ICU. But it certainly seems like it'd be useful even-- Especially in these cases where we're trying to stay out of patients rooms, and it would be easier to kind of monitor these patients a little bit remotely from an infection standpoint.

BRIAN PICKERING: So kind of the telly ICU concept, brought outside of the ICU onto the floor.

JOSEPH LEVITT: Yeah.

BRIAN PICKERING: Why haven't we done that more extensively before this? Because this always seems like it would be a useful monitoring tool. I mean, patients come into the hospital, they're sick. Why don't we know when they're getting sicker or better on a kind of system-wide perspective? Mike, you've worked in-- You work with a lot of kind of high performance athletes. Can you just tell us a little bit about how they're using technology?

I mean, we've got cell phones, we've got wearables. We've got all sorts of stuff out there to gather study. How are they using it in their world to improve performance? And I think we've got you on mute, Mike, sorry. Mike, I think you're still on mute.

MICHAEL JOYNER: I should-- I'm not now. It's highly, highly variable. It depends on the sport and the athlete. And some sports, people are using a lot of technology all the time. The cyclists, for example, are very interested. The triathletes are interested. The Kenyans take the heart rate monitors and take the straps that fit across your chest, hang them between two trees, and dry their clothes on them. So it really, really is variable.

The level of validation is weak. And a few key things, like something called perceived exertion has been shown to be pretty effective. So I think that people are using them. The extent to which they make any difference at all is very limited. And for example, things like the weight loss trials have shown that they actually make things worse because people get to their 10,000 steps and feel like that's a ceiling, and then they don't feel the need to be active anymore.

So I would say it's a real mixed bag. I think that if you talk to the sports analytics people, they will tell you that you can get all sorts of very interesting, very detailed data. But the most important data is a few simple things. I've told you this before, does anybody here know why Tom Brady has played well into his 40s? He's been hit 50% less than anybody in the last 15 years. So he's just got less mileage. Yeah.

So that's not-- And why does he have less mile-- We can get into reasons why. So I think the other thing, too, is I sent you that thing, Brian. You may have shared it with the group about the dog chasing the Frisbee.

BRIAN PICKERING: Yes, yes.

MICHAEL JOYNER: [INAUDIBLE] APACHE Ranson's score scores, which are helpful in the ICU is I think they're finding with the sports metrics is that the three, or four, or five things that explain 80% or 90% of the variants are the things that frequently you can actually do something about. And train people about, or improve your decision-making about. And I think there's more thinking now about signal to noise versus evermore granular data.

BRIAN PICKERING: So that's a great point that you bring up. I mean, Erin, you're a pharmacist. You've got a lot of signal and a lot of noise. How do you guys filter all of the information that's coming through the EHR now?

ERIN BARRETO: I think a lot of it is focusing on the high-value targets. I think especially in light of the COVID situation and other types of situations, finding low-hanging fruit and doing that well. I mean, there's a lot of ways to get cute and creative with new technology and new metrics for various patients as Dr. Joyner pointed out. But ultimately, we can't honestly do the things that we should be doing now correctly.

And so I think we need to focus on the low-hanging fruit, the easy stuff, the protocolized stuff that is not actually making it to the bedside. And leveraging the technology to accomplish those before we try and get too creative with our approaches. I'm not sure that I have a great answer for that. I think to build on one of the discussion points is how do you identify patients earlier on the floor? I think the same principle applies to how do you get patients out of the hospital safely, and monitor them in the community setting at their home so as to avoid the limited resources and costs associated with remaining inpatient.

And I know connected care and remote patient-monitoring are a couple of strategies where we're trying to use technology to accomplish that. But that's another way I think not only in the escalation of care phase, but also in a de-escalation of care phase that some of these tools might be beneficial.

BRIAN PICKERING: Has anyone experienced the phenomenon of information overload in the ICU where you've got lots of data?

MICHAEL JOYNER: The last time I was in there, there was no information so we couldn't get [AUDIO OUT]

BRIAN PICKERING: Sorry, Mike. You're blocked off there. But you practiced in the ICU, what, [INAUDIBLE] years ago?

MICHAEL JOYNER: A long, long time ago. [INAUDIBLE] 5 years ago so there was no information so you couldn't get [INAUDIBLE]. There was just those stupid Siemens ventilators, which made no sense.

BRIAN PICKERING: So no information versus all the information we have now. I mean, I've worked--

[INTERPOSING VOICES]

JOSEPH LEVITT: If you don't mind, usurping the monitor role here, and turning this around, and asking you a question. Maybe for you and Vitaly, I mean, I don't know how long ago it was. But you guys published that paper out of Mayo where you identified people with acute lung injury, identified people who were on too big a tidal volume. And what I thought was sort of the missing link of all this sort of risk stratification, and risk scores, and identification of patients, you actually showed that you were able to reduce tidal volumes and time exposed to potential injurious mechanical ventilation.

That was not a-- I thought that was a breakthrough paper. I tried to tout it as much as I could. It didn't seem like it rose to the level that it should have, and we're still not routinely doing that. And I guess maybe you guys aren't even routinely doing that at Mayo. Can you maybe talk about why and what the hurdles were?

BRIAN PICKERING: Well, Joe, thank you very much. And I know-- And you're very welcome to ask me questions. I'm the nominal panel leader. But I'm hoping as we go through here, there'll be lots of discussions. But no, you're absolutely right. I mean, a lot of credit to Vitaly and [INAUDIBLE] who kind of pioneered that work where they really looked at the problem. So number one, what problem where we solving?

And the problem to solve was we know that high tidal volume is injurious. Therefore, let's do a low tidal volume. So when you look at all the steps you need to go through to actually deliver low tidal volume to a patient, central to them is the calculation. And the calculation has-- You need to know the height, which we weren't routinely gathering. So you need to gather height correctly. And then you need to put it through a complex calculator before you could identify the tidal volume, predicted tidal volume, and then set it.

That's a lot of steps. And the complexity of it-- When it doesn't sound complex on a patient-to-patient basis, when you get overwhelmed like we might be now with hundreds of patients, whatever you can do to simplify is phenomenal. And I think that's what Vitaly and [INAUDIBLE] did really well with that study, they simplify it. The steps between seeing the patient and delivering the right tidal volume.

And in fact, in the first iteration, Vitaly, you used a card which just kind of an estimate of height and tidal volume requirements based on height. And that worked phenomenally. Maybe you could talk a little bit about how you took that and then made it an electronic signature, and how you managed with that over [INAUDIBLE] time, Vitaly. How did that change behavior?

VITALY HERASEVICH: Yeah. That was interesting time in regards even this paper. Journals, they did not want to accept it. They thought, oh, that only can be done at Mayo Clinic. No other places who can do this. But in reality, it was really simple because this simple calculator, we apply rules pooling data from different electronic medical record sources at that time. And we just did this simple calculation, no fancy artificial intelligence, no prediction models, nothing.

I mean, there are, like, rules, which makes sense. And that worked really well. Implementation was one of the important steps. Great tool without appropriate implementation do nothing. And we paid attention for both parts, how this performed and how it was implemented. And that worked well.

BRIAN PICKERING: But that complexity to simplicity, I think, was really key. And Joe, one of the things that I was really impressed with about the study as well is that once the behavior was changed, we switched off the alert. So we were paging the respiratory therapists, and that led to a change in behavior. So they were anticipating a call. We had the tools there at the bedside that allowed them to fix the problem themselves.

And after a while, you could see this figure went from a few calls to fewer calls, so no calls. So what did Vitaly and [INAUDIBLE] do? Well, they switched off the alert and nobody knew it switched off. So they still behaved the same way. So you eliminated the noise of false alerts. Because even with the really highly-specified algorithm they have, there was still some false alerts there. But they switched it off.

They weren't counting every single time there was a problem like we do now, it was targeted really specifically towards changing behavior. And once that behavior was changed, they switched it off, let it run in the background, and more of searching for behavior going bad again, and it really never did.

ERIN BARRETO: Can I build on that?

BRIAN PICKERING: Yeah.

ERIN BARRETO: One of the things I think you've been talking a lot about is signal to noise. And I know at least from pharmacy, one of the things that is challenging for me is that our drug information systems that we use are not tailored to the critical care environment. So we get tons of-- Perhaps this medication is contraindicated. Say, for example, an opiate analgesic in the critically ill patient because the patient is geriatric.

Well, obviously that's not necessarily context-sensitive. And so there is this tremendous amount of information that is inundating the clinicians and it's a lot of noise rather than value. And I think that's one of the things that I miss about AWARE, as an example, is that the laboratory information and some of the signals were tailored to the environment. So a given lab being deemed abnormal was not necessarily abnormal in the ICU if it was just above the laboratory limits.

And I think that that's a pretty important point, at least in the critical care environment, where there's so much data coming in that it needs to be contextualized in the ICU setting.

BRIAN PICKERING: Sonal, your [INAUDIBLE] so you had a comment.

SONAL PANNU: Absolutely. I wanted to make some comments. My comments are building up now. But going onto your point, kind of building on almost everything that everybody's been saying, I think the answer to information overload which is a true situation is what you had done, which Erin brought up very well, was customization. To see what you need to see and what is important to you. So there's just this whole world of data out there, what is it exactly that you need and how can you best find it? Correct?

So that's kind of the one thing that I feel is helping-- Is the difference between the EMRs which are helping us versus the one which are hindering us. In terms of identifying patients who are on the floor and getting worse, we've had some early prediction scores as a part of the daily EMR, which get populated in, but only the things that you would want to see. So for example, if you were in the ICU, we actually put the PF score on--

The P to F ratio on there rather than seeing what's different. We don't actually put the tidal volume on there. We only say what tidal volume per ideal body weight is there. So you see what you exactly want to see. And then, for example, on-floor patients say-- I don't know what we're exactly using now. But if it's a [INAUDIBLE] or if it's something else, then that will change graphs as needed. And that's what's seen by the primary team or by the ERT team.

So if you have an on-floor patient, [INAUDIBLE] ERT team can pick it up even faster. So those were my couple of points about customization, about bringing about simplicity as you all were talking about tidal volumes and implementation of that. And I had just the same example I had come up when we did electronic alerts for oxygen management, which was done at Mayo and now it's getting implemented here at Ohio State as well.

That when it was too frequent and too much, it was affecting how efficacious it was. And now that we've gone to a more efficacious but less alerts, but they're more effective, it's actually useful in changing behavior and people feel that now it's more helpful to them.

BRIAN PICKERING: And kind of as a segue into the next topic. This is great because what I'm wondering is we-- we're building more and more complex scores where we take all of these pieces of information, amalgamate them, and then say, let's talk about this idea of predicting deterioration. Have you had experience of some of these prediction alerts? And how have they worked in your hospitals?

And again, I'm thinking about the situation where we've got a triage situation, or we've got a very full hospital, and we're trying to identify patients who maybe we should be getting an intensivist too with the hope that we wouldn't cause an event. Can I ask you what your experience in your hospitals are of these early detection systems, early warning scores? If you're using them, how well they've worked for you? Joe, maybe I'll start with you.

JOSEPH LEVITT: Yeah, we're pretty rudimentary. We're using a sepsis alert is probably our biggest one, and that's the timings just to know that if we're-- Someone's meeting criteria for sepsis that they're getting an early blood draw and they're getting early antibiotics. That's about as far as it extends as how we're using it. We did have-- Just an example of that signal to noise, the first time we tried to do it automatedly, there was about 20 overhead alerts going off every hour for sepsis alert.

So we had to put changes so that instead of an overhead page, it was going to a nurse. And then a super user nurse was kind of evaluating every case because obviously, multiple people can have it go off multiple times in the same patient. It's not-- everyone knows that their heart rate's up and the respiratory rate's up, and it doesn't help us to have an alarm telling us that. I have colleagues at Keiser who have built more developed systems of predicting transfer to the ICU, and using that as a screening.

They use it on-- screen it daily, and evaluate patients that way. And can that-- sometimes even as a way to just evaluate who may not be appropriate for going to the ICU, and having a goal as a care discussion in real time before the patient's in extremis. Or having a softer landing in the ICU instead of when an RRT is being called. But I'm interested to hear how others are doing it. I mentioned first that they have BPAs specifically for COVID.

And I'm just wondering how you were able to get that built in fast enough in real-time? And whenever we've done this, it takes-- If we're asking Epic to build it for us, it takes a long time to get these things done.

BRIAN PICKERING: [INAUDIBLE] I think you mentioned BPA is associated with COVID. Maybe that's where that's come from. Could you comment on the BPAs with COVID, and how you've been able to get them up and running so quickly?

SONAL PANNU: So I meant BPA as in the COVID data, but we don't really have a BPA for COVID. We only have [INAUDIBLE].

[INTERPOSING VOICES].

BRIAN PICKERING: It was you who mentioned that, OK. Well, tell me, have you got any early warning systems in your system, Sonal?

SONAL PANNU: So we do have early warning systems on the floor-- for medical [INAUDIBLE]. I'm trying to remember which ones they are, which I said were more directed toward the ERT team to have a look at them, which is basically-- and then for the primary teams. So those were the ones which helps protect-- For our patients to see where we need to get to the ICU. We have sepsis alerts in the ER for clinical and for research purposes to help with early enrollment and screening.

And then we have the Austin alerts in the ICU. That's in terms of the BPAs that are being used for right now.

BRIAN PICKERING: If I could comment. I mean, it seems to me when I look around the country, that we have these fancy EHRs installed. We're gathering lots of data. And their ability to kind of use that data in a kind of compound fashion, or a kind of complex fashion is very, very limited. Would the panel agree that we really have very limited tools in our EHRs to kind of help us predict maybe where patients are going or--

I'm not hearing any great examples. Would you agree?

[INTERPOSING VOICES]

VITALY HERASEVICH: Yeah. I even add to that. I usually use this example of the IBM Watson. Nine years ago, they came and totally changed world with IBM Watson, and [INAUDIBLE] predictions. And nine years later, now there are no prediction tool which used in the practice or show any very good outcome, or so on. There are multiple problems with data which coming through these systems and algorithms.

It's not a one-side answer. But reality is what all the system, they still over the years show no big impact on the prediction. People build prediction models all the time, but there are no single system which works well.

BRIAN PICKERING: And we've got a lot of experience of building alerts on retrospective data sets, validating them on those data sets. And then as we bring them into practice, we realize that-- We built a sepsis prediction tool, for example. But a lot of the data that you need to predict sepsis, it turns out, is generated by people who think the patient might have sepsis. So somebody take a blood culture and you say, aha, that patient might be septic.

Let's pop up an alert there. And when you look at it, it's the clinician who's looked after the patient who is kind of aware of the situation in a way that's more nuanced than is represented in the EHR, and they're making the decision. So what we see in the EHR, in my view, is a fairly poor representation-- It's OK, but it's not a very nuanced representation of what's happening in the moment on the patient.

And I wonder-- We spend a lot of time entering data, and our nurses spend a lot of time entering data in the EHR. Is there anything about the data that they captured that you would like to see change? Are there any gaps there that you'd really like to see addressed?

ERIN BARRETO: I think it would be pretty interesting in the context of AI, if data processing using radiomics or natural language processing could be something that's a little bit more real-world applicable. I think certainly these machine-learning algorithms that pick up patients and are heavily affected, like you say, by the clinical decision-making that underlies data acquisition. And so it's really hard to parse those pieces apart.

But the text of somebody's gestalt about a patient or their physical exam is something that I think we are not in the routine habit of trying to use in our algorithms, but probably provides a little bit richer data that isn't currently able to be or being processed naturally.

BRIAN PICKERING: On the topic of AI, before COVID, AI was all we could read about in medicine. At least in my kind of area. Can I ask you what are you reading about AI that kind of excites you? Is there anything that kind of grabs your attention, and you're kind of excited about? And Sonal, maybe I'll start with you on this.

SONAL PANNU: Let me see. I think identification of disease patterns more accurately than humans. Identification of more [INAUDIBLE] in all the [INAUDIBLE] that we are providing, and if machines could do that better than us, I think.

BRIAN PICKERING: So you're referring a little bit to some of this imaging studies that come out where you've got an X-ray and there's a diagnosis? And because you don't get fatigued as a computer, you can kind of be a little bit more accurate maybe in the diagnosis. Is that the type of thing you're talking about, Sonal?

SONAL PANNU: Yes.

BRIAN PICKERING: And Joe, you're in the heartland-- You would hope in the heartland of innovation. What's going on in this space from your point of view that you're interested in?

JOSEPH LEVITT: Yeah. It is remarkable the disconnect between the Googles and the Yahoos that have come out of Stanford and [INAUDIBLE] into the health care system. I think, in part, it's got a big-- Health care, in general, is conservative and risk-averse. And probably the good analogy is the self-driving car. That even if it's much safer-- If a self-driving car makes a decision and kills somebody because of it, even if it's saved nine other lives along the way, it's the one person that dies that you hear about.

So we, like I said, aren't doing a lot here. The things that-- Sort of low-hanging fruit is simple things. Like we have stacks, pages and pages of protocols that have been built to apply to patients. Whether from insulin management to sepsis management, antibiotics, fluid management. And so we know how to manage patients the best. The problem is we just can't do it right all the time. And it seems like a pretty low-hanging fruit would be to have system protocols that are much more opt-out instead of opt-in so that the health care system would--

The EHR would be identifying that, hey, this patient meets criteria for this protocol. And it would pop up every time you open that patient's chart. And then you would say yes or no whether you think they actually-- A human could then decide that yes or no, this protocol actually applies to them. I thought maybe what Sonal was mentioning is that it also does allow us to do the sort of clustering analysis, identifying--

I think ARDS is a great example of a bit of a fool's errand where we were grouping patients. And really, we've made this heterogeneous group of patients that makes it pretty hard to study and intervene on. And that now recognizing the sort of subgroups within that, it seems to be more important. Chris Seymour has an interesting study where they looked at patients come into the ER with sepsis.

And clustered the patients not by any clinical feature, just by how many orders that they had in the EHR. And broke out, I think, five different groups that predicted the hospital course. And it kind of speaks to what you were saying about retroactive where the doctors may know this patient's sicker because they're ordering 10 times as many tests on that patient, or orders on that patient. But going back and then looking at those clusters and seeing who makes up those clusters may actually identify important subgroups of patients that's not intuitive to us, that we otherwise recognize in our routine practice.

BRIAN PICKERING: So recognizing patterns and big data sets that maybe in the past, our epidemiology tools didn't allow us to do. I think that's a very relevant use for AI. Vitaly, you've been heavily involved in this space. We're trying to build AI off of electronic health record data at the moment. Could you talk to us a little bit-- And we've had lots of conversations about that, about what some of the risks of that are versus where the opportunities may be for doing a little bit more of what Sonal and Joe mentioned, which is that kind of gestalt of a clinician.

How can we give the algorithms, I suppose, the right data they need to manage and to predict something that a clinician won't do?

VITALY HERASEVICH: Yeah. Of course there are many limitations, which sometimes not appreciated, sometimes people [INAUDIBLE] appreciate it. Yeah, one of the probably data challenge for people who try to tackle some difficult problems is EMR data, pre-test probability.

The recent data is there. And we talk a little bit about this already, but data is there because somebody want to have this data there. A physician order troponin because they expect myocardial infarction, there's some suspicion of that. It's not a screening test of everyone. And [INAUDIBLE], this algorithm is built on the top of that and say, OK, there's some late [INAUDIBLE] because of myocardial infarction because troponin is high, that's obvious to bedside clinicians.

And this probably biggest limitation, number one. Second limitation would be data most of the data coming a little bit late. People do trap, and then they document it. When data set get to the hand of the data-mining guys, they apply these rules on the flat big data set. Everything is there and it looks perfect. But in normal life, this data is simply not available.

At the time when this algorithm could trigger it, it's a little bit late because everybody knows about the situation from course of the clinical work, and so on. And this is biggest limitation. And how they could be addressed it's hard to say, but probably some ambient data which coming to the system without any interaction with human. Sensors, video camera, video recognition, this type of information could augment this technology.

But simply building something out of the big data set would not work. And I would just point, again, to this example with IBM Watson. They have all the resources, all talents, all money, everything. But IBM Watson not is there for nine years in health care. They are brilliant in security work. They do a lot in financial work, but they did not get there. And why? And the reason, obvious answer, it's obvious.

The amount of data, it's not the data which could be used for building this AI prediction tools. We need more data, augmented data, and new dimension of the data to make sense of the data.

ERIN BARRETO: Can I make a comment? I'm curious maybe for you, Dr. Pickering, on this. I think there is hesitation among clinicians who are not in this field to accept or absorb some of these recommendations, alerts. There's I think potential pushback on a perception of big brother, or what is in the black box of AI. How do you find in your approach to deploying these tools that you really can bring the clinicians in, secure buy-in, so that when it comes time to implementing that things are going to be successful?

BRIAN PICKERING: Yeah. And that gets really to all of the points here. So first, I think you have to be solving a problem that you have buy-in for. I think if you're telling me that I need a BPA for something that I don't really think is-- We get lots of BPAs for prolonged QTC in our ICUs, for example. You don't get propofol and fentanyl together. Gosh, I do that every day. So number one, are we identifying a meaningful problem to solve that can't be solved by the human being?

Two, when you're building a solution, are you using the data that actually is really the predictive or reliable data element? Or are you building an imperfect tool and pushing it down? And I'll give you an example of this in just a minute that I'd like to discuss with everybody. And then number three. When you're bringing it back into the workflow, how do people want it delivered to them? When do they want it delivered to them?

There's a sense that if you're an engineer, you're going to want to hit the human being like a machine every time with an alert. And I think we've heard from several kind of examples now that actually no, you want to change behavior. And you'll tolerate not [INAUDIBLE] every patient because what you really want to do is to change the behavior. And there's really only two things you can do.

You can need fully automate the system where you can hit it every single time. There's an aerosol-closed system, and we don't have many examples of that. Maybe things like glucose management or Coumadin management maybe. Those types of things, or closed-loop ventilators. Potentially, that's something you could take the human error out of. But for most things, you require a human being interacting with the data, making a decision.

And I think the very best thing we can do in the short-term with these alerts is build them around meaningful problems, bring in data that is reliably representative of the mechanism of how the problem emerges. So not taking serum rhubarb and saying gosh, if you have that measure five times, it means that you need a breathing tube put in. Well, we know that has nothing to do with respiratory problems.

So we need to really think about how physicians and conditions think about disease, and the mechanism, and build those into the algorithms. When you got a data and you're a data scientist, you don't need to know medicine to draw associations between the data and the data set and then outcome that you measure. But as a physician, I need to know that the data driving the decision is actually relevant.

And that's to Vitaly's point, a lot of the data in the EHR, there's no mechanistic insight in the EHR. You couldn't read the chart as a person who knows nothing about medicine and learn medicine from it. It's probably the worst thing you could do, read the chart with all the billing data and all those other things. You're not going to learn medicine from it. So there's no mechanistic insight necessarily buried in there.

Maybe you may uncover mechanistic insight, but it's not the primary intent of the EHR to do that. So as you build algorithms from them. I think the problem we've had is that we've tried to build algorithms from that data. But what we need to do is we need to identify the data. We want to feed a model, extract it from the environment, and then the model will do really well and I think the clinicians will embrace them.

We also need to think about the culture piece where I want-- We've got this COVID piece. We're talking about triage and surge. And we're talking about how in the situations where we're overwhelmed, we make decisions about who does and doesn't get a ventilator. Now, for me, I go to the bedside, review the patient. And based on my very best knowledge, I'll try and make some of those triage decisions.

It's not very usual in a resource-rich environment like the United States that you need to make those. But I've worked in resource-poor environments where we've had to do that with human beings. The proposals that I'm hearing relate to very simple scores like sofa scores being used to differentiate between who should and shouldn't have a ventilator. Clinicians are going to balk against that because they realize how flawed that data can be from the EHR, and they're going to want to be involved in the decision.

So again, pushing back against the idea that we really don't want these advisories, that we don't want to decisions. I think that the environment in which we're building them, and the data we're using, is what leads to a lot of the bad behaviors that you have in rejection of these tools. At least that's my bias. I think when we've gone the other way and we say, let's start with the model. Let's start with the data we need.

Let's bring the clinicians in, make sure it represents their view of the world. And then we implement it, it usually works really well. I think what happens right now is that we've got people building these who maybe don't understand medicine, and who have the EHRs, their tool kit, and that's all they're using. Anyway, that's a bit of a long winded discussion. But we've just got a couple more minutes here.