Thomas Frost on Clinical RL with Natural Timings
TalkRL podcast is all Reinforcement Learning, all the time. Featuring brilliant guests, both researched and applied. Join the conversation on Twitter at TalkRL Podcast. I'm your host, Chauhan. Doctor Thomas Frost is an emergency physician based in London, UK.
Robin:He's also in the final stages of completing a PhD at the University College London, where he has been looking at offline reinforcement learning applied to health care settings. Welcome, doctor Thomas Frost.
Thomas:Hello, Robin. Thanks for having me.
Robin:So I've had the pleasure of getting to know you a little bit at the past two RL conferences, and I've been able to see your your PhD develop and and follow part of your work. I'm super excited to talk today. Also, it's an area I found fascinating for years, Clinical RL. So I can't wait to get into it. Can you can you give us kind of a high level view of of your thesis?
Thomas:Yeah. Absolutely. I mean, first, thanks for having me on. I've really enjoyed our conversations about kind of, like you say, health care, RL, where my work fits within that. I think if I had to summarize my research, I would say that I'm I'm interested in sort of reinforcement learning applied to sort of clinical decision making.
Thomas:So are there aspects of clinical care that could be done better through an oral framework rather than sort of current clinical practice? And within that, I'm sort of interested in offline RL. So learning from exclusively historical datasets, and the sort of topic of my thesis is how much processing we should or shouldn't do to that data before we try and train oral agents on it. So I'm really interested in sort of where timings of decisions fit within that. That was sort of the sort of the the overall theme throughout the thesis, I think.
Robin:So you are a medical doctor, and you worked in emergency rooms for, I I understand, a long time, and you left that to do this work. Can you can you talk about that decision? What led you to to that path?
Thomas:Yeah. So I've been a doctor for about eight, going on to nine years now. Initially qualified, practiced full time, worked through kind of a COVID. And coming through the other end of COVID, I think I was a little bit burnt out as a lot of doctors were. I'd already been quite interested in sort of machine learning and in particular, RL by that point.
Thomas:I the thing that kind of got me interested was the really the AlphaZero work. I was I remember being so excited by this idea of something that could learn to make better decisions in a way that was free of human bias. So if this kind of, like, self play, you only know the rules of of chess, figuring out a good way of of playing games through that without any of our sort of human based historical biases about how one should or should not play chess. And I remember thinking that would be such an exciting thing to apply to health care because so much of clinical practice is unknown, and so much is kind of hereditary knowledge that sort of passed down from one generation to the to the next. But, obviously, a lot of biases come along with that.
Thomas:Anyway, so then post COVID, I was sort of looking to do a a sidestep and have a change of page, do something a bit different. And I came across this PhD program at UCL in London, which is where I'm based. And in The UK, we have these things called CDTs, centers for doctoral training. And what's nice about CDTs is, essentially, you have guaranteed funding for sort of three to four years over a very broad research team. So for example, the theme of my CDT is AI in health care.
Thomas:And it means you're very free to lead the research in the direction that you think it should go as you are doing the research. So you're not really bound to a particular funding call or or anything like that. And that worked out really well for me because I sort of went into it, played around with couple of other themes, like temporal clustering of sort of COVID ventilated ventilated patients with COVID. And then about a year, and I sort of remembered that the thing that had got me interested in the first place was was RRel. And this belief that there are and saying this very much as a clinician, there are aspects of clinical decision making that are certainly not optimal.
Thomas:We're sort of the best available option, but we're not really practicing it's not really a solved problem how we practice. And so, yeah, I've just been really interested in exploring which aspects of that decision making maybe we could improve through sort of autonomous AI.
Robin:So for the listeners who might not join us for the whole episode, can you give us a preview of some of your main findings in this work, both on the RL side and the and the clinical side?
Thomas:Yeah. Absolutely. So from the clinical side, I mean, I think I've got some results that I'm quite excited about that, essentially, when you're if you look at sort of an intensive care setting, so really the sickest of the sick, one in six intensive care patients won't survive their admission. So these are really high mortality patients. A lot of those patients will get drug infusions.
Thomas:And a lot of those infusions, there's no one fixed dose that we give to all patients because everyone is a little bit different physiologically speaking. So instead, what happens is we often need to continuously titrate or adjust those drug doses depending on each person's response. So a really simple example of that would be insulin and blood sugar. And critically sick patients often have quite high blood sugar, and we know that if we do nothing about that, it can lead to worse outcomes for them in the long term. So we often give those patients insulin infusions that then get adjusted based on what their blood sugar is.
Thomas:But sort of the catch is that in terms of clinical practice, what we're doing is we're adjusting for short term biomarkers. So we're really trying to keep blood sugar in some prespecified optimal range, but with some air quotes around the word optimal. And the reason there are air quotes around the word optimal is that we don't really know what is optimal for different cohorts of patients. And patients themselves don't care about blood sugar as a as an outcome. You know, it's not a patient relevant outcome.
Thomas:Patients aren't thankful that you kept their blood sugar at eight. What they care about is not dying, getting discharged sooner rather than later, getting fewer infections, etcetera. So I think what's exciting about my work is this idea of being able to train a reinforcement learning algorithm to optimize drug doses, specifically insulin doses, for those long term outcomes directly. So being able to say, you know, here's Robin in our intensive care unit, and right now, giving him this dose of insulin will actually optimize his long term survival rather than simply trying to optimize a a short term number or measurement. And so my early results suggest that we have successfully done that, that we've identified cohorts of patients where certain changes to practice would improve survival.
Thomas:So from the clinical side, I think that's really exciting. And then from the RL side, I think I really spend a lot of work in my thesis talking about the state of the sort of current health care literature. There's a lot of challenges when you're doing health care based RL because the data is really it's very chaotic. It's sparse. It's irregularly sampled.
Thomas:And one of the sort of quick and dirty ways of dealing with that is to preprocess that data very aggressively into sort of a tidier, neater, simpler version, and then do offline RL on that. And I think probably one of the biggest findings from my thesis is showing that when you train RL models on data that's been processed like that, you're essentially training them on a fictional version of reality, which means that then when you deploy them back into the real world, they are liable to perform worse than you expected because the environment you've put them into is actually different from the environment they thought they were trying to optimize. So a really simple example of that would be, maybe I'm treating patients, you know, at random intervals, thirty minutes, ninety minutes. But when I preprocess my data, the model just sees fixed regular decisions every one hour. Well, from the model's point of view, that's the environment it expects to be deployed into is having decisions once an hour every hour.
Thomas:And so when you start exposing it to an environment that isn't like that, it may well do poorly. But I think the most interesting bit about that is that if you are trying to predict how your model's gonna do and you do that analysis on your preprocessed data, then your predictions are also wrong because you're predicting how it will do within this fictional preprocessed dataset, not how it will do in your real world messy, regular decision making. So I think those are kind of probably the most exciting bits of the of the thesis.
Robin:So your off policy evaluation is kind of using a different MDP than where the data came from. Is that one way
Thomas:of saying it? Yeah. I think that's that's exactly how I'd summarize it. You've just essentially completely changed the nature of the decision making that was generating the original data. Because, obviously, the data was generated using unpredictable irregular decision frequencies, but you're pretending to the model that it was generated using a a regular decision frequency.
Robin:And and we call those the semi MDPs. Right? When the when the time time step varies?
Thomas:Yeah. I think I think semi MDP is the right way to frame it. It's, in a way, it's a continuous time setting. But unlike certain other continuous time settings like robotics, where maybe the scale, the precision in which you want the frequency of decision making is very small. For health care, it doesn't really matter if you make a decision at one minute resolution versus three minute resolutions.
Thomas:You know? But patients the physiology doesn't really change that quickly. So you can kind of discretize it into, like, really sort of well defined discrete time steps, like, every five minutes. But if the decision making itself is happening over irregular numbers of steps, then that is best framed under a semi MDP framework for sure.
Robin:So I I kinda came across this line of work a number of years ago looking at the kind of a famous paper, AI Clinician, the Komorowski paper that was on the cover of Nature, and it claimed like a alpha zero type, you know, we're gonna optimize, sepsis treatment. And there was a very specific, quite quite a complex methodology used there, and I gather you're doing something quite different now. Can can can do you wanna talk about the kind of this line of work and and where it came from and how it ended up like this?
Thomas:Yeah. Yes. So absolutely very, very happy to talk about that. So let's say AlphaZero comes out in around about sort of 2017 or so. There's a lot of increasing hype around deep reinforcement learning, very much led by sort of deep mind research.
Thomas:And then in sort of 2018, you get this paper published in Nature that is really building on that hype by saying, well, we've taken reinforcement learning. We've applied it to health care data, and we think that we can treat sepsis, which is kind of a condition of a disordered response infection. They can have quite a high mortality rate. Basically, a paper saying that we have found a better way of managing sepsis through reinforcement learning for these these sick patients. Published in Nature, and I I think to the best of my knowledge is the most highly cited paper in health care based reinforcement learning, and has probably led to a lot of interest, you know, post that paper being published.
Thomas:And I think much in the same way as DeepMind doing our research on Atari is not reflective of real world challenges, but was still a very valid, like, important sort of initial step. I think I feel similarly about the AI commission paper in that I think for an early bit of work, it's it's it has a lot of sort of positive aspects. But I think that the field of health care RL, specifically offline RL, has sort of fallen into a little bit of a stagnant no. That's the wrong word. A little bit of, like, a trap.
Thomas:So, essentially, in the AI clinician paper, one of the things they do is that they they bin this data. So they take this irregularly sampled data, and they bin it into four hour fixed regular windows. And that's immediately problematic because sepsis is not managed like that. Sepsis is a rapidly evolving condition. You often need to do a lot of higher frequency decisions early on in the management.
Thomas:Once they've stabilized, then it might be sort of a longer amount of time between successive decisions. But early on, it's not the case that you take a single decision and then you wait four hours. And what has happened post that paper is that we're seeing a lot of other reinforcement learning papers in health care also use sort of binning or temporal resampling of their data. And I think the reason it's done is because it is a very convenient fix for the problem of irregularly sampled and sparse health care data. And you have this paper, this sort of seminal paper that's so highly cited.
Thomas:So I think it's very reasonable for someone to do a literature review, see this paper, see the works that follow, and decide to continue in that vein. But as I said, it's problematic because at some point, we're going to want to run clinical trials with these these models. I mean, I think that should be always the long term goal. It's not to publish papers. It's to actually get something to the patient bedside that's gonna change our management.
Thomas:And if that's our end goal, you know, clinical trials are very difficult to set up. They're costly. They're expensive. There are risks to patients. We wanna make sure that the models that we get to clinical trial have the best chance of success.
Thomas:So how do we do it with that? Well, we need to make sure that when we train them, we train them in a way that is reflective of the environment they're being deployed into. And so I suppose where my work differs is my work is essentially saying, look. You know, we have probably more than 90% of health care offline oral papers are are preprocessing their data in this way into fixed regular intervals. And I think that that is problematic because we train models on a fictional version of reality.
Thomas:And I also think that we're not going to accurately identify that this is problematic. Because when we do our evaluations, most of those evaluations are retrospective. They're done on the same bin data, so they're not accurately reflective of how the model's gonna perform. And to date, I think there's maybe been by my last count, there were I could find three prospective trials of of reinforcement learning models trained on historical data and then tested out on real patients. One of which was a behavioral intervention, so very low stakes.
Thomas:One of which was sort of managing deep brain stimulators in a very controlled setting. They didn't need to do any sort of temple bending because of the nature of their the stimulators. And then there's I could only find one inpatient prospective trial of an offline RL algorithm, which was done in in China. So the vast majority of the literature is relying on retrospective evaluations. And so what you're seeing is you're seeing papers like the AI clinician publishing very positive sounding evaluations, but you have to bear in mind those evaluations were done on heavily processed simplified data, the vast majority of the time.
Thomas:So the sort of the first chapter of my PhD looks at exactly that. I essentially get a there's an FDA approved clinical simulator for for diabetes management. And so what I do is I take that simulator. I modify it so that we have irregular intervals. I collect a bunch of data from that simulator, and then I do varying amounts of preprocessing to that data.
Thomas:So now we have some resample training datasets. We have some non resample training datasets. I do some offline RL on those various different datasets. What I'm then in a unique position to do is I can prospectively evaluate my models in the original unpredictable simulated environment. And so you end up seeing exactly what I sort of theorized would be there, which is that, one, when you train on preprocessed data that has a a fictitious temporal sequence, your model does poorly if you then deploy it back into an unpredictable environment.
Thomas:Whereas if you train it on data that hasn't been preprocessed, the model can actually improve and do a lot better. But more concerningly, when you do retrospective evaluation of all those models on their original dataset, retrospective evaluations don't predict that difference at all. So retrospective evaluations think that the model is going to do very well because, in fact, it has solved the optimal actions for your fictional dataset. So from retrospective evaluation point of view, your model was doing exactly what you wanted it to do, which was to find optimal actions for that dataset. But it just so happens that that's not the same as optimal actions in an unpredictably timed environment.
Robin:So, like, if you do off policy evaluation on the Kalmarowsky dataset, which I've done it, it'll it'll tell you, well, it'll tell you, well, this this policy would be better. It'll find a better policy for that MDP. You're saying that MDP isn't doesn't represent the MDP we really wanna solve.
Thomas:That's exact yeah. And there's That's exactly what I'm saying.
Robin:Yeah. And there's there's a few I wanna zoom in a little bit. There's a few different aspects of this. Right? Like, there's the the time binning forces the observations to be binned in time where they where they're they're generally not, and also the actions to also be lined up with the observations, which may be not the case in in in clinical work, and then also the this notion of was there a no op?
Robin:Was there actually did any act did any doctor actually come by at that moment and decide, no. I'm not gonna take an action versus there was just no decision made. And that is that is that correct? That we don't see the difference between those two things in the data?
Thomas:Yeah. I think that's I think what would we'll come to the no op aspect in a little in a in a moment. But I I totally agree with you that there are several overlapping things going on when we observe that models trained on bin data then perform poorly in unpredictable environments. The first is exactly as you say, we're sort of if you imagine a bunch of irregularly timed input data or irregularly timed decisions, if you're binning over a fixed interval, there are gonna be some intervals where you have multiple measurements. So you have to aggregate those together into a single measurement.
Thomas:So that could be averaging or summing or taking the last measurement, whatever you like. And there'll be some windows where you don't have any measurements at all. So you end up imputing a value using whatever method you wanna use for imputation. So that's already somewhat problematic because you're saying to the model so let's say let's say the patient received a drug dose of one, and then thirty minutes later, they received a drug dose of four. When you bin that data, you're saying to the model, the patient received a drug dose of six because you're just adding those together.
Thomas:But that's not really what happened. What happened was they received a drug dose of one, and then some time passed, and then they received a drug dose of five. It's it's physiologically quite different. So you have this sort of alteration of what the numbers themselves mean. So that's sort of one issue.
Thomas:The second issue is that you have this, what I like to call, a failure of temporal expectation. So what I mean by that is, let's say that your model was trained on data that had been binned to a very high frequency resolution. So maybe binned to every 10. So you're saying to your model, an independent decision was made every ten minutes. Then your model starts to learn this expectation that it will always be able to adjust its actions after ten minutes have And if you then deploy into an environment where it takes an action under that assumption and then an hour goes past before it can make its next decision, then it's at risk of making decisions that are actually unsafe that it probably wouldn't have made if it had known that it wouldn't get an opportunity to intervene again for another hour.
Thomas:So a really simple example would be, you know, maybe their blood sugar is a little bit high. The model might learn to give it a very, very big dose of insulin or a very high rate of insulin infusion, knowing that in ten minutes time, it can just check the blood sugar again and adjust it once it started to normalize. But, obviously, if an hour goes past and for every minute of that hour, the patient's just getting a high dose again and again and again, then you end up overtreating. So this kind of failure of temporal expectation, I think, is important. It's important for the model to know that there might be a longer than ideal delay between its decisions.
Thomas:So I think those are sort of the two main ones. The third one is a little bit trickier, and it has to do with bias. So there are a couple of different ways that you can aggregate your time windowed data for observations and for actions. Now you can aggregate them over the same time window. So you get all your observations from t zero to 60 and all your actions from t zero to 60, and you aggregate them.
Thomas:And then you say this was the observation, and this was the action. Now the problem with that is if multiple actions are happening and multiple observations are happening, then you actually have look ahead bias, where the observations that you're giving to your model have already been confounded by the very action it's trying to model. So, you know, you've you measure blood sugar. You give some insulin. You measure it again.
Thomas:You give some insulin. There'll be changes to the observations based on those two insulin doses you gave. But if you aggregate them together, so now you just have a single average blood sugar measure and a single average insulin dose. When you train that in offline RL, what you're saying is to the model, given this observation, this was the action given. And that's not really true because the observation has already been changed by the action itself because of aggregating it.
Robin:Yeah. The the MDP or the original SMDP has a certain causal structure, and then when you apply this binning, it completely changes the causal structure of the resulting
Thomas:Yeah. That's that's exactly right. And so that that's from a a piece of work by Cheng Prutang at Emory University. I'd encourage anyone to to have a look at that paper. So then you could say, okay.
Thomas:Well, that's a problem. How do we deal with that? Okay. Well, what you could do and what in that paper is recommended is you average your actions from t to 60 sorry, t equals zero to t equals 60, and then you average your observations over the previous window. So now all of your observations that you've aggregated are all before the action that you've aggregated.
Thomas:So you've removed look ahead bias, which is great. However, because of how these aggregation windows align, you can end up with very weird sort of causal inversion artifacts where you have these strange things where maybe in the original data, the patient had a big glucose spike, and the clinician responded by turning the insulin up. But if you then average with these kind of separate windows, so the observations are all averaged after the actions, it can look like the order of that has changed, that the insulin was maybe turned up and then the blood sugar went high, which is obviously also not ideal because that's not actually what happened in the environment. So, essentially, there's no good way that you can do these aggregations when you're having multiple observations in each window, which I think is sort of the third important problem with with binning as a practice.
Robin:I mean, it's it's it's like we've tried to turn the patient into an Atari learning environment game, and we're plugging in, like, the the classic DQN stuff is all is all very simplistic when it comes to regular time steps. So, I mean, it makes sense. We had all this RL machinery that was assuming that, but you're you're really you're really zooming into how that really doesn't fit here. But I wanna point out, it's not just I mean, you're solving this for clinical RL, but what you're saying really applies a lot more broadly to, I would guess, all sorts of areas in real world RL, industrial RL, where where where that has these properties. Right?
Robin:That has the properties that you're you're finding here where the irregularly spaced actions, observations, this decisions are not made on a on a on a clock tick. Way ways in which the real world is very different than the assumptions with under DQN.
Thomas:Yeah. I'd say that's that's completely accurate. I'd say the conclusions from the sort of the chapter we've been speaking about, they're really specific to a setting where you have irregularly sampled decisions in particular, the important one, but irregularly sampled data. And you're trying to do reinforcement learning on that data by first binning it and turning it into this neat, clean MDP when the underlying environment itself, at least in terms of how the data was collected, was not an MDP. And so that's obviously not limited to health care.
Thomas:Limited to any setting where you have datasets like that, where the intention is to sort of deploy it back into the environment from which you source the data. And I think I think that should be intuitively obvious. I mean, if even let's say I had a big dataset of just chess games, and I'm trying to teach you, Robin, how to play chess. It's so intuitively obvious that if I just show you every fourth position from the games, then you might think you've learned how to play chess, but you obviously haven't learned how to play chess. Because I've lied to you about how chess is actually played.
Thomas:It's not the case that you make this move, and then you see this new position that actually happened four moves later. And I think that the real world is is no different. So we really wanna be training these models in a way that is as accurately reflective of the source environment as possible.
Robin:So, Thomas, do you think I guess you think that RL in health care and clinical RL is is a worthy pursuit, and and it'll lead to real benefits?
Thomas:Yeah. So I think if for anyone who's kind of listening who does research in RL or is interested in doing research in RL, I'd really like to really encourage people to consider health care as one of those domains to be working in. I think it's it's very challenging, but also the potential benefits are are huge. So, you know, health care is health is very important. Anything that anybody cares about is only possible because of good health, and it's very easy to underappreciate health until you no longer have good health, and then you realize the sort of value of it.
Thomas:I think it's a really interesting reinforcement learning challenge because if we think about most reinforcement learning, most reinforcement learning is is online. You have this opportunity to interact with the environment, make mistakes, and get feedback from that. And part of that is this idea that RL is always changing. Sorry. RL is always affecting the the state it's seeing, the data it's seeing.
Thomas:So in supervised learning, if I deploy a model, the predictions of the model don't really necessarily affect the data the model is being exposed to. But in our realm, the decisions it makes affects future states that it gets exposed to. So that sort of distribution of data is always changing as the model evolves. And that's all well and good unless you are encountered with a domain where you can't actually train our own models in that way. So health care is often a canonical example for that sort of problem because you can't let algorithms experiment on people for kind of obvious ethical ethical reasons.
Thomas:So instead, we end up using kind of offline URL where we're looking at historical data. And there are pros and cons to historical data. Advantages are that we can really cut out a lot of unnecessary learning where we're sort of rediscovering the wheel. You know, we've got decades of clinical trial knowledge, clinical expertise, all of which is distilled into the behavior of of everyday clinicians. So all of that learning has already been done and is is present in the dataset.
Thomas:But then the limits are that you can't really learn past the behavior you're seeing in the dataset. So even though clinicians are noisy decision makers, you can only really perform at the towards the best level of known clinicians. You can't really extend much beyond that just because you can only learn from the data you're seeing. So I think that's a really interesting challenge because we have these huge datasets. We have so much data.
Thomas:But, you know, in much the same way that online RL, you have algorithms affecting the data you see. When you train in an offline setting, when you deploy your algorithm, it's going to change the data it's seeing based on the decisions it makes. So then that becomes sort of an interesting challenge in and of itself because you want your agent to be a bit better than how clinicians are behaving, but you don't want it to be too much better. So you're sort of aiming for, like, the top 20% of clinician behavior.
Robin:So why don't want it to be too much better? Can can you help me understand?
Thomas:Yeah. Yeah. Of course. Yeah. So if your if your model's behavior is too far away from the behavior of clinicians, your model ends up running out of distribution behaviors.
Robin:Oh, you just don't have coverage there?
Thomas:You just don't have the coverage. So it it could be that it's better, but you don't know. And if we're thinking about the sort of safety critical nature of health care, you want your model to learn behaviors that you have good reason to think are are better actions rather than just out of distribution actions. So that, you know, in and of itself is is interesting. And I think is part of the reason why, you know, models trained on bin data then perform poorly is that they're being exposed to unfamiliar data.
Thomas:They make decisions that are out of distribution because the data is unfamiliar. They then get exposed to even more unfamiliar data, and you get this sort of compounding distributional shift where the model sort of just collapses. Collapses. So, yeah, trying to have your model learn behaviors that are relatively within the distribution of the data is important, which earlier in the earlier in our discussion, you talked about sort of no op actions, and I think now is kinda a good time to talk about that. Because when you bin data, you establish every single decision point very clearly.
Thomas:You know, you have time steps one, two, three, four at regular intervals. And at every time step, there must, by definition, be a decision. But if you are only interested in a regular decision intervals, then you can fall into a trap where you may not know all of the decision points in your data. So to give you basic example, if we look at outpatient clinics, we know when every decision occurred. Because even if outpatient clinics are irregularly spaced apart, we know that the only opportunity the clinician had to change their management of the patient was at the outpatient clinic.
Thomas:And between clinics, there's no opportunity to interact with the patient. Whereas when you're in an inpatient setting, it's much more of a continuous time setting where, in theory, the clinician could go into the ward or the bed bedroom of of the patient at any point in time. And you know all the times when the clinician has changed something because that might be recorded or should be recorded in the dataset. But what you don't know is all the times when your clinician went into the room, thought about doing something, and decided, nope. I think the best move is to do nothing, and then they go away.
Thomas:And that's, you know, that's a missing label, and it's a label where imputing that becomes very important. So you could naively say, well, why don't we just treat every single instance where they don't make a decision as being a labeled no op action? And the problem with that is you then fall into this trap that we talked about earlier, that the model then just gets this false temporal expectation because it you're giving it labels every single minute of do nothing. And so it thinks it can intervene frequently.
Robin:And is that the classic way that the older papers have handled this?
Thomas:I think yeah. It implicitly, yes. By binning, you are effectively doing that because you're saying if there are no clinical decisions made in my in a particular time window, then we'll just impute the label of of do nothing. But that's obviously not really what really happened. And what's really important in terms of minimizing this risk of sort of distributional shift and maximizing the chance that your models learned and accurately reflective visualization of of the environment, what you want to do is you want to make sure that when you impute those no op labels, because they are important, it is important sometimes to do nothing, that you're imputing them at a a frequency that matches up with how often you think the model's gonna get queried when you deploy it.
Thomas:So the model is then learning, okay. Here are two NAWP labels that are about an hour apart. So I'm gonna learn to expect that there might be about an hour until my next decision. And in some settings, it can be easy to impute that. So if you take my clinical setting in my thesis, so insulin infusions, we have a very strong prior that we can rely on where we can say, in current clinical practice, clinicians only adjust insulin straight after a blood sugar check.
Thomas:That is a mostly true statement. No clinician is gonna go fiddling around with insulin when they don't know what the recent blood sugar was. So that's great because then we can just look at how often bedside nurses, for example, are checking blood sugar, and then we can just anchor our NOOP labels onto those blood sugar checks. So every time a blood sugar gets checked, we look for any nearby actions. And if there are no adjustments made, then we can impute a a do nothing label.
Thomas:And it is it's a proxy, but any sort of imputation is gonna be trying to approximate when we think these labels should occur. And what's nice is I can then you can then deploy a model, and you can say to the hospital, the ward you're deploying the model, listen. This model expects to be queried every time you check blood sugar. So check blood sugar as often or as infrequently as you would normally do so, but just know that when you check it, the model's expecting to be queried, which is nice because it is naturally irregular. It doesn't change how the bedside, like, clinical practitioners are currently acting.
Thomas:Basically, say, do what you would normally do, but when you check a blood sugar, query the model. So that's a sort of an easier example. But there will be other situations where you don't have those clinical priors, and there's some sort of research that looks into that problem, not by me, but by some other researchers, around this idea of can we use sort of kernel based method or or similar methods to identify certain states, certain moments in time where there's a very obvious spread of actions being taken by people. In other words, this is a point of contention. Some clinicians go for action a.
Thomas:Some clinicians go for action b. Whereas maybe there are other points in time where all the clinicians are very consistent with the action they take. Does that make sense?
Robin:Yeah. So, I mean, that's really interesting. Why would clinicians disagree? And, you know, I wondered if the are there different philosophies of care, different schools of thought, or possibly could there be an issue with state construction? Like, the something wasn't recorded, and so we don't notice the difference between these two states.
Robin:There may yeah. I guess these questions come up when I look at this type of data. What what what causes clinicians to disagree or or apparent disagreement in the data?
Thomas:Yeah. I think all of the things you've suggested are are very insightful and and definitely are part of it. I think at the end of the day, we are clinicians are human, which means we're subject to all the same restrictions as other humans. In other words, differences in clinical experience, differences in local guidelines, differences in cognitive biases. For example, if you recently missed some catastrophically low blood sugar or high blood sugar, then maybe you're more likely to be extra acutely aware of whenever a blood sugar was was checked.
Thomas:Whereas certain other clinicians maybe might be more prone to missing results, and then you see no op actions where you wouldn't expect to see no op actions. So there are loads of different reasons to do with both individual differences and also kind of regional differences between hospitals and areas and countries. And I suppose the additional factor is in some settings, you know, we just don't know what the right thing to do is. So even, you know, maybe there isn't a best standard. Maybe there's some ambiguity, and that's where personal preference starts to become more more obvious that some clinicians think that certain ways of behaving are are better than other ways.
Thomas:All of which is not good for patients, but is good for offline reinforcement learning because it means that we get to see a spread of different actions for the same set of kind of physiological patients, which means that we can use these big datasets almost like natural experiments where we can say, okay. Are there groups of patients that receive a spread of actions and the groups of patients who received action a do slightly better than the groups of patients who received action b? So maybe we should only take action a. So it's only possible.
Robin:Yeah. So we're not getting full Epsilon Green exploration, or we're getting a little bit of diversity.
Thomas:Yeah. Yeah. That's absolutely right. And if we don't have that diversity, then you can't do offline RL because you don't have the coverage. If every state has a deterministic action, it doesn't work.
Robin:I mean, so how you you've you've been on the other side. So how does the you know, I've seen doctors and for for certain things, have flowcharts or decision trees to work with. How does a doctor coming into to doing this, what what what level of guidance do they have, and what level of freedom do they have in in, like, executing their own policy?
Thomas:Yeah. So there's normally a for insulin infusions, there many hospitals will use a sort of a prespecified protocol. They're very simple. They essentially say, what is the blood sugar? And depending on what band the blood sugar is in, so it might be, like, four to seven, seven to 10, you should adjust the insulin by this amount.
Thomas:And maybe you might have different columns for whether they're, you know, already a known diabetic versus not a patient with diabetes, but their blood sugar is is high anyway because they're they're very sick. So these kind of protocols do exist. There's not a single right protocol. So different hospitals have different protocols, and different clinicians might be more prone to overriding those protocols. So they might say, actually, I think that we need to take a different action for whatever reason.
Thomas:Maybe we think that the action is too aggressive from the protocol, or maybe we think that the action recommended by the protocol undermanages these patients. And then, obviously, there are the cases of of missed blood tests, which happen semi frequently where you don't miss them forever, but it might be the case that a blood glucose was checked and someone only really registered that it was higher than normal or lower than it should be an hour later or ninety minutes later just because it's especially in intensive care, there can often be a lot going on, and it might just be that a certain one of multiple tests was done, and that particular blood test just happened to get overlooked. So so, yeah, they they they can be generally standard protocols for these things, but I don't think they're optimized protocols. I mean, the argument I make in my final chapter is that when we are optimizing something like insulin infusions using offline RL, what we should be doing is using the power of RL to optimize for long term outcomes directly. So for example, for me, I'm looking at survival, but it could be length of stay or it could be some other longer term outcomes.
Thomas:And we sort of disregard this idea of an optimal blood sugar range kind of completely. I have one of my chapters essentially talks about a preprocessing pipeline for putting together a dataset that is sort of naturally timed, realistically timed, both from an input data point of view and from a decision label point of view. And the pipeline is sort of agnostic to the domain you're working within. But along with the pipeline is sort of a published dataset that I call insulin for a row, which is encouraging researchers interested in health care RL to try their hand at often RL using health care data that is naturally timed within the context of insulin infusions in in critical care.
Robin:Yeah. Okay. So let's talk about insulin for RL. Now the MIMIC dataset that you drew this from covers many different types of conditions, and we we've seen historically a lot of work on sepsis, and you went a very different way here. Can you talk about how you decided to focus on insulin and and why that's different and and and how you built this dataset?
Thomas:Yeah. So I'm I'm really happy you asked that question. Because if you look at the health care, like, oral literature, often oral literature in particular, what you will see is that most papers look at sepsis. And I think part of that is because of the AI clinician paper, the Matthew Kamarovsky paper. And part of that is because sepsis is just it's such a good headline grabber in terms of the domain you're working in.
Thomas:You know, the mortality of sepsis is high, so there's a lot of potential gains to be had for helping patients. But I actually think that sepsis is probably the last problem that we should be working on because it is so fiendishly difficult. There's so many things going on. It's such a heterogeneous condition, and there are so many different aspects of management. And so what I think is that we should really be going about it, the sort of deep mind Atari route.
Thomas:I should be starting with a much simpler version of that kind of problem and solve that, and then iterate our way up to more complicated settings from there. So the reason I choose insulin is that insulin is the simplest example of what we call titratable infusions, infusions where you have to adjust the dose at sort of maybe not regular intervals, but certainly frequently. You just have this very simple feedback loop where the patient is getting insulin, and you're checking their blood sugar. If the insulin is high, the blood sugar will go down. If the blood sugar is high, you probably need even more insulin.
Thomas:So it's it's it's a very clean simple feedback loop. And the argument I make is that we should start by just really being happy that we've solved that problem. Because then when we move on to something slightly more challenging, like, say, blood pressure control, we have far fewer problems that we need to solve for for that. So that was the rationale behind choosing insulin. In terms of the insulin for our dataset, a lot of patients, as I say, often put on insulin infusions when they're critically unwell, regardless of whether they have a diagnosis of diabetes or not.
Thomas:So the incident for our data set within MIMIC, I think, identified approximately twelve thousand patients who were admitted to ICU and had a period of time where they were receiving an insulin infusion. Most of those patients just have a single episode of of receiving the infusion, but there are maybe sort of five percent or so who who had two or more episodes. So about twelve thousand patients. And if you include the sort of no op imputed decisions with every blood glucose check that didn't have an action, then in total, the data set has about 375,000 labeled decisions. So just over, I think, half a million cumulative hours of sort of patient exposure.
Thomas:And I think that should be more than enough samples for us to kinda make good headway on that kind of a problem. And as I say, optimizing insulin is not a solved problem. Yes. We may not get the same glamorous mortality improvements as something like sepsis, but we're much more likely to be able to definitively say, we've probably solved this problem based on the data that is available to us. So, yeah, that that's probably the sort of the headline things that I would say about about the dataset.
Thomas:Maybe one other thing I would add is that as far as I'm aware, it's the only reinforcement learning dataset for health care that uses naturally timed data that is specifically intended for reinforcement learning. And, hopefully, if people access the dataset, they'll find that they're set up for reinforcement learning researchers. So there's a data frame that you can use, but there's also tenses that are just states and actions and rewards, kind of the more familiar format. I mean, I think part of the reason so many reinforcement learning researchers rely on binning is that there are already well established binned health care datasets. And if you're not super familiar with health care data, it can be very intimidating to try and clean that from scratch.
Thomas:So people are probably prone to wanting to use preestablished cleaned health care data search, which I think is entirely reasonable. So the goal of the dataset is basically to say, look. We've we've cleaned the data for you so you can just focus on doing the offline RL with natural time data.
Robin:Can you spend a minute on state construction here? Like, Komorowski had this clustering thing with discrete states, and I always wondered about how that was over the logic behind the clusters. Do you yeah. How does the how does the state or or maybe this is it doesn't assume too much because you're just providing the dataset, and then but how do you do state construction when you when you build your policies?
Thomas:Yeah. Of course. So so the Kamarovsky paper, for example, uses as you say, it uses clusters. So you end up with a a discretized number of states. I think off the top of my head, it's around about 750 states.
Thomas:Now you could ask the question of whether or not all possible ICU phenotypes can be grouped together in 750 states. I think that's sort of a separate question. My approach is that I think we should give deep learning models the rawest possible data we can and let them form their own latent impression of what they think, so the patient status. So the way that I define states and implement states in in Sanfarraro is that we we have our decision point. So blood glucose has just been checked, and there will be some label that we could learn from.
Thomas:And what we do is we look at the last about seven days of historical events. And by events, I mean, lab test results, changes to infusion rates, drug boluses. Those are sort of the main ones. And what I sort of recommend in the dataset is I recommend using something called the medical event data standard. So you can take this history of medical events.
Thomas:And for each medical event, you represent it as a tuple of the particular feature. So it could be their sodium level was checked. The feature, the value, let's say a 140, the time, so let's say 60 ago. And between those three aspects, you sort of capture the most important things related to that that particular medical event. And you can order these as a sequence.
Thomas:So you have a sequence of these tuples, and all of a sudden, you can just run this through any sequential network that you like. So it could be an LSTM, could be a transformer, could be state space model, whatever you like, where you start with an embedding network to embed the tuples into kind of a single vector so that that medical event is represented as just sort of one vector. But you have this sequence of irregularly sampled events that you then just provide to the model. And from that, the model draws its own sort of latent impression of of what the state might be. And I think that's good because it means that you have done really as little as possible to influence it.
Thomas:There's no hyperparameters around state clustering. There's no binning of the input data. There's no imputation. There's no missingness because the model should develop this implicit understanding that the only observed data is the data being given to it. And so anything that has not been explicitly observed by definition is unobserved.
Thomas:So you don't need to explicitly state, you know, sodium wasn't measured at sixty one minutes, sixty two minutes, sixty three minutes. You know, that's sort of not necessary. It doesn't really add new information. So I think it's a more it's a more powerful way of representing the state. And it was inspired by a paper that came out, I think, five years ago called oh, it's called Strats.
Thomas:So it's a sort of self supervised transformers for sparse and irregularly sampled time series data, something like that. But they're really sort of the origin point. That and the medical event data standard between those two was what inspired me to set up the input state in that way.
Robin:So in chapter five of your thesis, you you focused on using temporal difference learning for mortality prediction. So you're using TD learning for something in a in a very different way here. Can you tell us more more about that and what you found?
Thomas:Yes. So the the inspiration for that that chapter, sort of trying to use temporal difference learning for this purely mortality predictive task, the inspiration came from the sort of the chapter that followed it, which was, you know, I know that I want to optimize insulin infusion doses for for survival, which is sort of a novel thing that people haven't really tried to do before. And I know that I want to use this kind of new naturally irregular health care data, but I suppose there's this question of how effective is temporal difference learning at learning to predict mortality when you have such an irregular structure. And, also, we know that health care data, you know, is is sparse. The states are partially observed.
Thomas:So maybe this is actually a really challenging thing to do. So I wanted to sort of empirically evaluate that a little bit So what I did was I took all the patients in Mimic. So not just those initial infusions, just every single patient in Mimic. And I took every single medical event in their history within my sort of feature set of lab tests and drugs. Drugs.
Thomas:And I set up a TD learning, temporal difference learning sort of framework or approach where well, I'll take one step back. In temporal difference learning, the idea is that you're bootstrapping from your predictions from the next state. But here in our sort of messy, irregularly sampled environment, it's not really clear what the next state should be. So you could take the super naive approach. You could just say, okay.
Thomas:Well, at a particular point in time where we've just measured something, we've just had a medical event, and we've got some history that we can give to the model. The next state is just gonna be whenever the next observation comes in. Now that could work, but in somewhere like intensive care, where you're having observations happen very frequently, that interval might be really small. So it could be, like, every two minutes or three minutes or five minutes. And we already know the temporal difference learning can really fail to converge when the intervals between states shrinks to really small intervals, and you're getting very small differences or imperceptible differences between your states from one moment to the next.
Thomas:So there is an advantage in you implementing some form of delay in time between your two states. And what I suggested was that we should basically implement, like, a mandatory delay, let's say, four hours. So we've got our current state, which is a bunch of recent history, sort of medical events up to now. And then our next state is gonna happen. We'll wait twenty four hours.
Thomas:And then whenever the next medical event comes in, we'll just pick that point, and we'll just say, okay. Well, whatever the history is leading up to that point, that will be our next state. So I trained a temporal difference learning model in that way, and the task was to predict end admission survival. So this is very sparse reward setting. You know, you're you've only got one binary reward coming in at the very end of the trajectory, and everything else is bootstrapping.
Thomas:So there's obviously a lot of risk that you could have, like, significant bootstrap related bias. Your model might not be very well calibrated. It may really struggle to converge. And pleasingly, what you see is that none of those problems arise that not only does it converge, but it converges very well. So it ends up with very well calibrated predictive probabilities for predicting mortality.
Thomas:But what I think is interesting about that chapter is that, you know, you kinda wanna have some sort of baseline to compare to. So what I did was I also trained a set of supervised learning models to predict mortality at various horizons. So predict one day mortality or seven day mortality. And I would compare the performance of those models to the performance of the TD trained model, again, at each of these different sort of labeled time points. So one day from whenever your state was constructed or three days from from that state.
Thomas:And what you see is as the prediction horizon gets pushed further and further into the future, TD starts to lead to much better performance than supervised learning models that have been trained on similarly distant mortality labels. And the the theory that I put forward is that this relates to the variance of the label. So if you imagine predicting twenty eight day mortality, a lot can happen in twenty eight days that can influence whether the person then died at the twenty eighth day. And a lot of those things might not necessarily be predictable from the information you have today. But the problem with deep learning models is that, you know, they're encouraged to overfit the data.
Thomas:So if you have this very high variance label, what will happen is the model will start to overfit to spurious correlations. Either correlations that don't mean anything at all or correlations that are really super, super specific to the mimic dataset, but they don't really generalize in a physiological way. And what I found is that if you evaluate all of those models on an external dataset from a different continent, you see exactly what I predicted, which is that the supervised learning models trained on distant mortality labels, they have big drops in their performance when externally evaluated because they've overfit to this kind of noisy distant label. Whereas temporal difference learning models, because they're trained against this bootstrap target that is much closer in time horizon to sort of now, so in my experiments of twenty four hours from now, the variance of that is much smaller. And we know that this is an advantage of temporal difference learning, that you have this bias variance trade off.
Thomas:But, yeah, what you see is models trained using TD learning. They have way less overfitting when you externally evaluate them. And you also see that their mortality predictions, even if they're not as well calibrated, they're less well calibrated in a way that is uniform. So what I mean by that is that it systematically over predicts mortality, which means that you can do post hoc correction to try and adjust that. Whereas all the supervised learning models don't have that.
Thomas:They have quite a messy disruption to their calibration. So to sort of summarize all of that, in essence, I shall show that you can use TD learning to predict long term horizon outcomes in in a setting where, you know, it resolves this issue of overfitting. The models generalize really well to unfamiliar data from different settings, much, much, much better than you would get otherwise from supervised learning. And I think that's kinda cool. I think that's interesting because as you say, it's applying TD learning outside of an RL approach to just a purely prognostication prediction setting.
Robin:I had to really stare at that and cross my eyes a bunch of times. I and squint, I I was surprised by your finding, and you mentioned that that in this setting, you're facing the deadly triad, which is the function approximation, bootstrapping, and off policy learning. You have all those factors in play, and and yet, yeah, I was I was surprised. I was actually really quite surprised. Were you surprised or you really expected this or I had to look at your diagram.
Robin:You had a good diagram kind of explaining it as well. I had to look at that for quite a while.
Thomas:Yeah. I think I was I I think I was pleasantly surprised at just how well calibrated it was. I think I expected it. I expected its ability to predict survival to be better than supervised learning for the reasons that we talked about. But I also expected the predictions to not be very well calibrated because of bootstrap bias.
Thomas:So I was very pleased to see that actually the calibrations were even better than than supervised learning. I think the reason I talk about the sort of deadly triads in sort of the early bits of that chapter are because as you've sort of already hinted at, there are lots of good reasons for this to not work, which means that before we even get to the task of trying to optimize insulin for survival, we need to know that TD learning can actually converge to accurate predictions in such a setting where you have this partially observable data, function approximation, boosted up bias, etcetera, etcetera. So for me, it was very important that we actually drilled into whether TD learning works in that specific setting. Obviously, TD learning works kind of when everything has been optimized, and it's a wonderful technique. But there are scenarios where it fails, like the deadly triad.
Thomas:So, yeah, I thought it was important to drill down into that. And I my expectation was that it was gonna outperform supervised learning, but I didn't expect that it was gonna be so consistently well calibrated.
Robin:Very cool finding. So should TDU learning be used across the board for clinical predictions of this of this type?
Thomas:Yeah. I mean, I I would love to see more interest in t learning from a predictive point of view. I suppose there's a caveat to that. There was a really interesting comment piece, I think, that came out of, I wanna say, King's College London. And it was sort of making the point that in a clinical setting, a lot of our prediction algorithms, like in AI, are really decision models in disguise.
Thomas:You know? So why do we want to predict that a patient is at a higher risk of deterioration? Because we want to intervene in those cases. Or why do we wanna predict that a patient is at higher risk of developing sepsis? Because then that means we probably need to give them antibiotics.
Thomas:And there are some problems with that that reinforcement learning will be very familiar to reinforcement learning practitioners practitioners that if you're going to be training a decision model, then you need to be wary of these things that we talked about earlier around having behaviors that are still in the distribution of your dataset, conservative learning, etcetera. So having a pure prediction model that is a decision model in disguise has its pitfalls. Nonetheless, I would like to see kind of a wider uptake of temporal difference learning as an alternative approach to supervised learning for these particularly longer term horizon predictions. And that's not specific to health care. I mean, that in any setting where you are trying to predict long term outcomes and you have this sort of time series data that is being re maybe not regularly measured, but you have a lot of time series data, exploring whether temporal difference setting might be an alternative way for you to accurately predict these long term outcomes, I think, is is worth a look really by anyone.
Robin:So you found some pretty interesting things regarding insulin policies. Do you wanna get into that now, Thomas?
Thomas:Yeah. Yeah. So let's let's talk a little bit about that. So for your re for your sort of listeners, in terms of the kind of standard best practice, we know that really high blood sugar is bad, and we know that really low blood sugar is bad. And then there's a sort of a middle gray area where we don't know whether patients being slightly high on their blood sugars is bad for them or whether it's a protective mechanism that their body is intentionally using.
Thomas:And sort of trials that have tried to figure the answers to that question. Trials that have explored that have really struggled to get a definitive answer. So I trained some models on insulin for a while to optimize survival. Bearing in mind, that doesn't know anything about optimal blood sugar ranges. It just knows that you have a patient state, and you can change their insulin infusion, and it's trying to reduce their expected mortality for certain actions.
Thomas:And what you see is you see that, first of all, from an off policy evaluation point of view, there's a predicted reduction in mortality of about zero point five percentage points. And what's interesting about that is that that sounds realistic. You know, insulin correctly managed will have some influence on survival, but not a massive it won't halve your mortality. But, certainly, it's a sort of a a realistic looking number. I had a I was very keen throughout my thesis to be using as robust statistical methods as I could come up with.
Thomas:So I did a lot of the sort of deep reinforcement learning at the edge of the statistical precipice. You know, I was very into training lots and lots of models, bootstrap confidence intervals, etcetera. And so I'm I'm quite confident that within the limits of what we can know from the data we have, that that probably is a realistic figure. But then you can kind of get into the qualitative side, which is what are these models doing differently that is leading off policy evaluation to think that survival will be better. And what you see is that most of the time, it doesn't do anything differently from clinicians.
Thomas:So, you know, 93% of the time, it is happy with what clinicians are doing, as in it doesn't think that there's a survival benefit that is being missed out on. But if you have a look at the sort of really small minority of states where it differs, then two broad patterns emerge. The first is that there are a subgroup of states where patients are they have very high blood sugar, and the model thinks that we are undertreating them. So we need to be more aggressive with our insulin. And if we look at the sort of physiologic physiological aspects of those patients, we can see that they're they're sick patients.
Thomas:You know, they're more likely to have biochemical markers of of being very sick. But they're also patients who are on high insulin rates, which suggests that these are probably patients where we're following these protocols that say, if their blood sugar's high, turn insulin up by by one unit. But because they're already on quite a high insulin rate, in terms of a relative adjustment, we're probably under managing them. You know? If you can imagine if someone's on fifteen units of of insulin and then you turn it up by half a unit, that's a very small relative change compared to if they're on one unit power and you turn it up by half a unit.
Thomas:And then the other set of patients are patients whose blood sugar is actually normal or at the lower end of the normal range. But if you have a look at the biochemical characteristics of those patients, these are, again, patients on a high insulin rate, so they're getting a lot of insulin, and their blood sugar is trending downwards. And we know that low blood sugar can be much worse for your survival. So these are patients who are on a high insulin rate. Their blood sugar is falling, but currently, it's normal, which means clinicians are probably missing out on those patients until they become, you know, hypoglycemic with low blood sugar, and then they intervene at that point.
Thomas:And that's the sort of interesting thing is that the the patients whose blood sugars are actually low, the model is happy with how we manage those patients. So once we know their blood sugar is low, we're managing them correctly. The problem is we're not always identifying the higher risk patients who are trending towards that state. And I think for me, that's really satisfying to see because it means that the model has converged to two quite realistic sounding policies of behavior in terms of where it wants to overall clinicians rather than something that would look very weird and spurious, and it doesn't really make sense. And, you know, maybe it's figured out some sort of high dimensional benefit, but we're not really sure why it thinks that's a better action.
Thomas:No. You have two sets of patients where it's pretty obvious why the model wants to overrule what clinicians are currently doing. And if you look at the existing literature, it fits with what we know about which patients are most likely to get survival benefits from certain approaches to to insulin and blood sugar control. So I think that's that's quite a cool finding to come out of that for for me anyway.
Robin:Absolutely. So you've you've opened the black box, and some of your findings I mean, maybe there's a chance you could even influence ER ER practice with with with this work. Is that right?
Thomas:Yeah. I mean, I I think if I said either of those things to a clip to a clinician, I don't think I've I don't think a clinician would find that controversial. If I said, look. If the patient's blood sugar is normal but trending down, maybe you should turn the insulin down. If the patient is sick and their blood sugar is really high, maybe you should push the insulin a little bit more aggressively.
Thomas:I think those are not too controversial, so definitely could influence practice. I think the sort of future steps would be you know, as I said, there's there's almost no prospective trials of RL. And I I think if we think about the long term future of reinforcement learning in health care, I strongly strongly believe that reinforcement learning is going to become a widespread presence in health care because there's so many examples of decision making that we as clinicians are just we're not that good at. And I think it will come, but I think it will come late, if that makes sense. So, yeah, health care famous is one of the last to get any technological innovation.
Thomas:I mean, there are still hospitals all throughout the world that use paper paper notes, paper records, and we're in, you know, 2026. So what I suspect will happen is that we'll see autonomous AI in other bits of daily life. So for example, in London, we're having rollouts of both Waymo and WAVE, so autonomous taxi service type vehicles. I think when we have many more self driving cars on the road, when we are seeing a lot more autonomous robot behavior in daily life, at that point, people will naturally wonder, well, shouldn't we be doing this in health care as well? But probably not before.
Thomas:And I suppose what I'd like to see is I'd like to see these successive rollouts of these types of policies, like the one that I trained in in my final chapter, where you sort of do, you know, maybe like a silent trial, just get the model to make some recommendations and get some clinicians to say whether they think it's safe or not safe, but you never affect a a patient's care. And then if that's looking encouraging, maybe we do a human in the loop trial where clinicians get the opportunity to decide whether to act out a recommended action or not. But, yeah, longer term, I think the trend needs to be not to so much to change clinical practice being done by clinicians, but rather to actually devolve or delegate certain aspects of practice completely to autonomous AI. That's why I think long term, the field is ultimately gonna go.
Robin:Okay. So if but but if you had a choice right now of deploying the policy that you had built for insulin, or rather distilling the insights that you found and updating what the clinicians are doing, like, would you would you pick the human interpretable route, or how do you think about that?
Thomas:Yeah. I mean, I think that's really interesting because what you can say about clinicians is that we, although there isn't anywhere near as much evidence in medicine as people might like to think, clinicians are very responsive to evidence. They really they care about trial evidence. They care about seeing the evidence for things. I suspect my research on its own would be analogous to sort of case study level research, as in probably not something that's gonna change most people's practice.
Thomas:The thing that would probably change people's practices is a formal trial. And if you're gonna do a trial anyway, then I suppose you're asking the question, do you want to do a trial where you've just adjusted these prespecified protocols using, you know, the distilled knowledge from the model, or do you wanna do a trial using the model itself? But I suspect at the moment I mean, for me, I I would have no problems with adjusting my practice from what the model's recommended, because I think what the model has recommended is very, very reasonable and not it's not something that would be controversial. But clinicians can be quite wary creatures in terms of formalizing whether something is best practice, and I suspect that the the expectation would be that the evidence that would change practice would come from a trial. So one way or another, a trial would have to be done.
Thomas:I guess it's just how you want to go about conducting that, either with an adjusted protocol or with sort of an RL model itself.
Robin:So your thesis represents years of hard work, lots of insight for for the community. What are you gonna do now, now that you're just about done, Thomas? What are your plans going forward?
Thomas:Yeah. I think that is the the the big question. Probably gonna have a little bit of a break from to just have a holiday. And then after that, I don't have fixed plans. I have ideas.
Thomas:One of the ideas is that, you know, I think maybe I would like to try and be the person who takes this sort of research forward into, like, a prospective deployment type setting in in The UK. But there are a lot of barriers to that. It was gonna if I did do that, it'd be a a long and hard road. So I would just need to spend a bit bit of time thinking about whether that's what I want to do. I know that some people, when they get to the end of their PhD, very much welcome the breath of fresh air that is a completely different, you know, problem setting.
Thomas:And I certainly sometimes think about that. Other than that, you know, I I think I'm I'm open to opportunities. I I I think I've loved clinical practice. I've absolutely loved it. But I'm also, you know, I'm also in love with the RL side of things, and I'm I suspect I'm probably happy to hang up the stethoscope and and pivot to just the AI stuff full time.
Thomas:And I probably will be AI in health because that's where my skill set is kind of best utilized. But the question is, you know, is that gonna be in an academic role, industry role, startup role? I haven't fully figured that one out yet. And sometimes it can be frustrating working in health care seeing that things don't move as quickly as you'd like them to just because of the nature of health data. But when you can get things to work, yeah, you can have such a big impact on on people, society, etcetera.
Thomas:And as I said before, it is just such a challenging area to work in. You know? It's got so many technically interesting, complex, difficult things to work at that I probably wouldn't mind continuing doing more health based RL. But, yeah, I think I don't have one fixed answer to that question other than that I'll probably continue focusing on on the RL and health stuff, but it's just what the right setting for that's gonna be.
Robin:Now we did get a chance to speak to Harvard professor Fanale Doshi Velez about RL in healthcare, and she was she was commenting on how different countries have different approaches to this in terms of their homogeneity of the data and their ability to move on this stuff. So I guess but how do you have any comments about that? Like how the you're in The UK, right? So how The UK system versus the American system, or she had mentioned Singapore. You had mentioned trials in China.
Robin:Yeah. Mean,
Thomas:I yeah. Yeah. No. I was gonna say, I mean, unsurprisingly, Finale is obviously completely correct in saying that. I sort of sometimes joke to people that The UK is, like, a great place to do health AI research, and then The US is a good place to try and implement health AI research.
Thomas:But then the they're both lacking what the other one has. So in The UK, obviously, we have this kind of national health care service, and we have growing interest from the public and the government to try and figure out how we can get research done on large amounts of data to improve the the care that we're giving to patients. On the flip side, because The UK is responsible for so much of the health data, it's probably prone to being overly conservative with it. By which I mean, you know, humans are hardwired to feel that if someone does nothing and something bad happens, then no one person is at fault. But if one person signs off on people getting access to data and then something bad happens, then, you know, that person is at fault, which means that people are generally incentivized to say no to a lot of sort of research applications, access to data, etcetera, etcetera.
Thomas:There's a very kind of conservative mindset from from that side of things. Whereas, you know, the perspective I get is that other countries are a bit more willing to to move quickly, which I think is good, actually. I think that if you're moving slowly, you're missing out on opportunities to really improve people's health. And sticking with the status quo for ten years potentially causes a lot of a lot of avoidable harm if you're sacrificing ten years of of of progress. But it does depend, as you say, on on which countries you're working in.
Thomas:I suspect that The UK is probably a better place to be working in from that side of things just because we're a bit better getting access to that data. But the question is whether the right people can be convinced to start implementing these types of models, and we'll see. You know? As I say, I think society in general will start to change its attitude when there's lots of RL everywhere else.
Robin:So I love to ask guests about lines of work that we may not have mentioned yet that you find interesting that the that you want the audience to be aware of. Anything like that?
Thomas:Yeah. I think the the two things that come to mind, neither of which really have anything to do with language models. But the the first is I really loved the the continual learning paper that came out of Rich Sutton's lab by one of his students. I just thought that was so cool, this idea of having models where you're you know that you're ending up with sort of dead neurons in the network. Having this sort of plasticity where you can continually identify neurons that are really consistently not contributing at all to any of your outputs and being able to reset them and reset the.
Thomas:I thought that was just such a cool bit of work, and I'd love to see more stuff like that. Other things that I've been interested in recently, I have had, like, a an an interest from a distance in more neuromorphic architecture type stuff, spiking your networks. I think there is this big question around energy usage of these massive neural networks and where the future of that is gonna go. And I suspect the neuromorphic architectures probably still haven't had their heyday yet, but it probably will come in due course. And then just in general, I'm really excited about the advent of startups that are gonna focus really on, like, RL as a way of trying to achieve AGI.
Thomas:So thing things like, obviously, David Silver, etcetera. I think that to me as a as a doctor, you know, as a biologist, that's so intuitively appealing. You know, the only examples we have of general intelligence are what we see in nature. And in that setting, it evolved as a way of basically doing movements better, making actions more effectively in order to reproduce, you know, pass on genes. So actions around finding food, making sure you don't become food, reproducing, etcetera.
Thomas:And so to me, the idea of trying to achieve AGI through, like, an action based perspective, it it just it makes sense because that's the only way we've really achieved not we, but the only way nature has achieved AGI. Whereas stuff that's like world modeling, but without sort of an action RL based element, to me, feels like it's putting the cart before the horse. So, you know, it's not specifically, like, any one research paper, but I'm I'm excited to see more and more of these startups that are that are RL based.
Robin:Doctor Thomas Frost, this has been fantastic. It's been a long time coming. I've been looking forward to this episode for, honestly, couple years at least. And it's it's been fascinating hearing about your work. Congratulations in advance on your PhD.
Robin:And when so when people are citing you, are they citing your papers, or are they citing the thesis?
Thomas:Yeah. Well, I guess the first of all, thank you. This has been this has been a lot of fun. I love talking about my work, and and I enjoy all of our conversations. I think if people are citing me, the mortality paper has been published, so they can take a look at that.
Thomas:The other two papers are under submission, but there are preprints available. So they're very welcome to cite those. The thesis will probably not come out for another yeah. Probably, like, two to three ish months or so. But, yeah, they can definitely the the main bits of work minus the insulin policy stuff, The main bits of work are are out there.
Thomas:I have a Google Scholar profile that people will be able to find fairly easily. So, yeah, I just encourage them to to to take a look at that. And, yeah, thank you again. This has been so much fun.
Robin:It's been great, Thomas. Thanks again, and we'll have links links to the key papers on the and and your profile on the on the episode notes. Thank you, Thomas Frost.
Creators and Guests
