From Average to Individual: Causal Machine Learning and the Validation Gap That Closes Our Precision Arc
The finale of our four-week precision critical care arc. How causal forests and individualized treatment rules aim to estimate how much this specific patient benefits, why the famous AI Clinician for sepsis drew warnings that it could cause harm if deployed, and the one discipline that ties all four weeks together.
In 2018, a paper in Nature Medicine announced something that sounded like the future of critical care. A reinforcement-learning system, trained on tens of thousands of ICU records, had learned what its authors called optimal treatment strategies for sepsis. It analyzed each patient’s state, considered the fluids and vasopressors that had been given, and recommended the doses that its model associated with the best chance of survival. The headline finding was striking. When clinicians’ actual decisions matched the algorithm’s recommendations, patients did better. The implication was that following the machine could save lives.
Then the critiques arrived, and they were serious. The system had been trained and evaluated entirely on retrospective data, because you cannot ethically let an algorithm learn to treat sepsis by trial and error on real patients. That constraint forces a method called off-policy evaluation, which tries to estimate how a new treatment policy would perform using data generated by different decisions. The problem is that off-policy evaluation cannot detect the biases, the confounding, and the missing information baked into the original data. One group of critics concluded that the claims could not be supported by the method used, and warned that deploying such an algorithm prospectively could cause serious harm to patients.
Sit with that tension, because it is the perfect close to this arc. Here was a sophisticated model, trained on real data, that appeared to outperform human clinicians, and that may have been dangerously wrong in ways its own evaluation could not reveal. The gap between a model that looks optimal on retrospective data and a treatment that is actually validated to help patients is not a technicality. It is where patients get hurt.
This is the finale of our four-week precision critical care arc. We began by establishing that the average treatment effect can mislead. We spent two weeks on subphenotypes, in ARDS and in sepsis, as one way to find the patients hiding inside the average. This week we reach the analytic engine that aims to go all the way to the individual, estimating not just which subgroup responds, but how much this specific patient stands to benefit from this specific therapy. And we reach the discipline that has run through every week of this arc, which is that a derived rule is a hypothesis until it is prospectively validated.
ICCN Update
The new ICCN website is live at iccn.io. Every article published in the past two weeks is now archived in one place, and our new Research section pulls recent published data from 26 major critical care and medical journals into a single curated feed for subscribers. Bookmark iccn.io.
Why This Matters
Every previous week of this arc pointed at the same destination. If the average treatment effect can hide opposite responses, and if subphenotypes are one way to find groups that respond differently, then the logical endpoint is to estimate the treatment effect for the individual patient directly. Not the average across everyone. Not the average within a subphenotype. The effect for this person, given everything we know about them.
This quantity has a name. It is the individual treatment effect, and its estimable cousin is the conditional average treatment effect, the average effect among patients who share a given profile. A rule that assigns each patient the treatment predicted to benefit them most is called an individualized treatment rule. These ideas are the analytic core of precision medicine, and a family of machine learning methods has been built to estimate them.
The promise is real, and it addresses a specific weakness we identified in Week 1. Traditional subgroup analysis examines one variable at a time and is prone to false positives, which is why the field distrusts it. Causal machine learning methods such as causal forests estimate how treatment effect varies across many variables at once, without requiring the analyst to pre-specify the subgroups in advance. This sidesteps the subgroup-analysis trap and can uncover patterns of response that no one thought to look for.
But the same methods introduce a new set of dangers, and they matter enormously in critical care, where the therapies are potent and the patients are fragile. A flexible model that estimates individual treatment effects can overfit, finding patterns in noise. Its estimates can be poorly calibrated, confidently wrong about how much a given patient will benefit. And most importantly, a model that predicts outcomes well is not the same as a model that predicts treatment benefit, and neither is the same as a treatment that has been validated to improve outcomes when actually used. Understanding these distinctions is now part of the basic literacy of anyone who will encounter algorithmic decision support in the ICU, which within a few years will be everyone.
Prediction, causation, and validated benefit are three different things. A model can predict who will die without knowing who will be helped by treatment. It can estimate who will be helped without proving that acting on the estimate improves outcomes. The distance between these three is where precision medicine succeeds or fails.
The Study / Evidence in Context
The machine learning approach to individual treatment effects has a clear methodological lineage and a growing critical care footprint.
The methodological foundation. In 2016, Athey and Imbens introduced causal trees, adapting the decision tree to estimate heterogeneous causal effects rather than to predict outcomes. In 2018, Wager and Athey extended this to causal forests, an ensemble method that estimates the conditional average treatment effect for each individual while providing valid statistical inference. In 2019, Athey, Tibshirani, and Wager generalized the approach into the generalized random forest, implemented in the widely used grf software package. These methods, along with related approaches such as double machine learning and the meta-learners developed by van der Schaar and colleagues, form the toolkit for estimating who benefits, and by how much, from the data.
The crucial conceptual finding, benefit is not risk. One of the most important results in this literature is easy to state and easy to miss. In a machine learning analysis of dexamethasone given as a prophylactic antiemetic, the estimated conditional average treatment effect did not correlate with the patient’s predicted risk of the outcome. In plain terms, the patients most likely to respond to the drug were not the patients at highest risk. This confirms, with data, the distinction between prognostic and predictive enrichment that we drew in Week 1. Selecting sick patients is not the same as selecting responders, and a model built to predict risk may point to the wrong patients entirely when the question is treatment benefit.
The critical care applications. Causal machine learning is now being applied across the ICU. Pirracchio and colleagues used machine learning to estimate the individual treatment effect of corticosteroids in septic shock, connecting directly to last week’s sepsis endotype story. Causal forest analyses have examined questions such as early beta-blocker use in critically ill patients with heart failure, where a 2026 study estimated heterogeneity in the individual treatment effect on mortality. A 2025 review in Current Opinion in Critical Care surveyed how clustering, predictive modeling, causal artificial intelligence, and reinforcement learning can reduce the heterogeneity that has caused so many ICU trials to fail. The through-line of that review is the same as the through-line of this arc. The average obscures heterogeneous effects, and machine learning offers tools to see through it.
The reinforcement-learning frontier, and its hard limit. The most ambitious application is reinforcement learning, which aims to learn an entire sequential treatment policy, what to do at each step as the patient evolves. The AI Clinician for sepsis is the landmark example, learning fluid and vasopressor strategies by modeling the ICU course as a sequence of states and actions. The ambition is enormous, and so is the difficulty. Because the policy cannot be learned by experimenting on patients, it must be learned and evaluated on retrospective data through off-policy evaluation. The Gottesman guidelines paper, published as a direct response, laid out why this is so treacherous. Off-policy evaluation depends on assumptions that are hard to verify, cannot detect confounding in the source data, and can produce confident estimates of a policy’s value that are simply wrong. The critiques of the AI Clinician were not that reinforcement learning is worthless in the ICU. They were that the evaluation method could not support the strength of the claims, and that acting on such a policy without prospective testing could harm patients.
How to evaluate an individual treatment effect model. The field has developed internal checks for CATE models, including Qini curves, targeting operator characteristic curves, and best linear projections, described accessibly in a 2025 causal forest tutorial. These tools assess whether a model’s estimates of who benefits actually track differential response in held-out data. They are valuable, and they are internal validation. They do not answer the ultimate question, which is whether assigning treatment by the rule improves outcomes when the rule is actually used. That question can only be answered by a prospective trial of the rule itself.
The AI Clinician was not a fraud and not a triumph. It was a powerful proof of concept whose evaluation could not tell whether it would help or harm real patients. That uncertainty, not the algorithm, is the lesson.
What Stood Out
Several features of this literature have changed how I think about algorithmic decision support in the ICU.
Treatment benefit is not baseline risk, and this is measurable. The finding that estimated individual treatment effect did not correlate with predicted risk is the empirical capstone of the whole arc. It means that the reflex of targeting therapies to the sickest patients can be exactly wrong, and that identifying responders requires estimating benefit directly rather than inferring it from severity. Every discipline that reasons about who should get a therapy should internalize this.
Causal machine learning escapes the subgroup trap but opens the overfitting trap. The genuine advance of causal forests is that they find heterogeneity across many variables without pre-specifying subgroups, avoiding the multiplicity problem that discredited classical subgroup analysis. The genuine danger is that flexible models can find structure in noise, and their individual-level estimates can be confidently miscalibrated. The methods trade one failure mode for another, and the reader must know which one they are looking at.
Off-policy evaluation is the Achilles heel of reinforcement learning in the ICU. The single most important technical lesson of this week is that you cannot safely let an algorithm learn to treat patients by trial and error, so you are forced to evaluate treatment policies on retrospective data, and that evaluation cannot detect the biases in the data it learns from. This is why the AI Clinician’s evaluation drew such sharp criticism, and why any claim that an algorithm has learned an optimal ICU treatment policy should be read with that limitation front of mind.
Internal validation of a treatment-effect model is not proof of benefit. Qini curves and related tools are real progress, and they are not the finish line. A model can pass every internal check and still fail to improve outcomes when deployed, because the internal checks are computed on the same kind of retrospective data, with the same hidden biases, as the model itself. Only a prospective trial of the rule closes that gap.
The whole arc reduces to one discipline. Across four weeks and three levels of analysis, the average, the subphenotype, and the individual, the same rule holds. A derived classifier, a derived subphenotype, a derived causal forest, and a derived reinforcement-learning policy are all hypotheses until they are prospectively validated. The sophistication of the method does not change the requirement. If anything, the more flexible and powerful the method, the more essential the prospective trial becomes.
Clinical, Research, and Leadership Interpretation
For the interprofessional ICU team, causal machine learning translates into concrete, discipline-specific considerations.
For intensivists and APPs, the central skill is reading claims about AI-derived optimal strategies through the off-policy evaluation caveat. When a paper or a vendor states that an algorithm recommends the best treatment, the questions are whether the recommendation was validated only on retrospective data, whether the evaluation could detect confounding, and whether the rule has been tested prospectively. Until it has, the algorithm is a hypothesis, and standard evidence-based management remains the standard.
For pharmacists, the responders-are-not-the-sickest finding is directly relevant to medication decisions. Causal machine learning is being applied to exactly the drug questions pharmacists own, including corticosteroids, antiemetics, and cardiovascular agents. A pharmacist who understands that a model predicting who is at risk is not the same as a model predicting who will respond, and who asks how any medication-selection algorithm was validated, adds real protective value to the team.
For ICU nurses, the human-in-the-loop role is the essential safeguard. Algorithms do not see the bedside context, the family conversation, the subtle change that has not yet reached the chart, or the situation that falls outside the data the model was trained on. Nurses are also the first to encounter distribution shift, the moment when a patient does not resemble the population an algorithm learned from. Vigilance for the case that does not fit is a clinical contribution no model replaces.
For respiratory therapists, individualized treatment rules are coming for ventilation and oxygenation questions, including oxygen targets, PEEP, and weaning. The same discipline applies. An algorithm that recommends a ventilator setting based on retrospective data is a hypothesis until it is validated prospectively, and RTs who can articulate that distinction will help their units adopt these tools safely rather than credulously.
For perfusionists, the fragility of individual treatment effect estimation in small samples is the relevant caution. Advanced mechanical support populations are small and heterogeneous, which is precisely the setting where causal machine learning estimates are least stable and most prone to overfitting. Skepticism toward confident individual-level predictions in small device-support cohorts is well founded.
For ICU leaders and educators, this is the governance frontier. As algorithmic decision-support tools arrive, the leadership task is to demand prospective validation, external calibration across sites, and clear evidence that a tool improves outcomes before it is allowed to influence care. This connects directly to the clinical AI framework we use on Fridays, where individualized treatment rules sit in the decision-support and agentic buckets that warrant the highest scrutiny. A journal club that works through the AI Clinician story teaches both the promise of the methods and the governance they require.
Bedside and Workplace Takeaways
Five reflexes to install this week.
Separate prediction, causation, and validated benefit. A model that predicts outcomes is not a model that estimates treatment benefit, and neither is a treatment validated to help patients. Ask which of the three any algorithmic claim actually delivers.
Remember that benefit is not risk. The patients most likely to respond to a therapy are frequently not the sickest. Targeting treatment by severity can point to the wrong patients. Identifying responders requires estimating benefit directly.
Ask how the model or rule was validated. Internal checks such as Qini curves are a good sign and not the finish line. The question that matters is whether the rule has been tested prospectively, in real use, against a comparator.
Treat retrospective optimization with special caution. An algorithm trained and evaluated only on retrospective data cannot detect the confounding and missingness baked into that data. Reinforcement-learning policies for the ICU are especially exposed to this, because they cannot be learned by trial and error on patients.
Demand prospective validation before a rule touches a patient. Deriving an individualized treatment rule is the beginning of the work, not the end. The future of precision critical care is not the model. It is the trial of the model.
These five reflexes will compound across every encounter with algorithmic decision support on your unit for the remainder of 2026.
Teaching Pearl
An algorithm that looks optimal on retrospective data is a hypothesis, not a treatment. The more powerful and flexible the model, the more essential the prospective trial becomes, because the model’s own evaluation cannot see the biases it learned from.
What We Should Not Over-Assume
Causal machine learning is a genuine advance, and it carries firm limits.
It does not solve unmeasured confounding. Causal forests and their relatives rest on the same assumption of exchangeability as target trial emulation and every other causal method we examined earlier in this run. If an important confounder is unmeasured, the individual treatment effect estimates are biased, however sophisticated the algorithm.
A model that predicts outcomes well may be useless for predicting treatment benefit. Discrimination for prognosis and accuracy for treatment effect are different properties, and a model optimized for one can fail at the other. The antiemetic finding, that response did not track risk, is a direct demonstration.
Individual-level estimates are often poorly calibrated and unstable, especially in small samples. A causal forest can report a precise-looking individual treatment effect that is confidently wrong, and calibration of these estimates is rarely checked in clinical reports.
Off-policy evaluation cannot rescue a treatment policy from the biases in its training data. This is the specific and severe limitation of reinforcement learning in the ICU, and it is why claims of learned optimal treatment strategies deserve particular scrutiny.
And deriving an individualized treatment rule is not the same as proving it improves outcomes. Almost no ICU treatment rule derived by machine learning has been validated in a prospective trial of the rule itself. Until that happens, these tools inform research and hypothesis generation, not bedside treatment selection.
Limitations
The body of evidence reviewed here carries limitations worth naming.
Most causal machine learning in critical care is retrospective, built on databases such as MIMIC and eICU, which share known biases in documentation, missingness, and the reasons treatments were given. These are precisely the biases that causal estimation struggles to overcome.
Reproducibility and external validation are incomplete. Models trained at one institution frequently degrade at another because of distribution shift, and independent validation across diverse settings remains the exception rather than the rule.
The evaluation metrics for treatment-effect models, including Qini and targeting operator characteristic curves, are unfamiliar to most clinical readers and are not standardized in clinical reporting, which makes it hard to judge the quality of a CATE model from a published paper.
There is no consensus on how individualized treatment rules should be reported, validated, or regulated for clinical use, and the field lacks prospective trials of the rules themselves, which is the single most important gap between the current evidence and the bedside.
Bottom Line
Causal machine learning is the analytic engine of precision critical care. Causal forests and related methods can estimate how much an individual patient stands to benefit from a therapy, they can find heterogeneity without falling into the classical subgroup trap, and they have shown, with data, that treatment benefit is frequently not the same as baseline risk. This is a real advance, and it is the natural endpoint of the journey this arc has traced, from the average, through the subphenotype, to the individual.
For ICCN readers, the closing discipline is the same one that opened this run seven weeks ago. A derived rule is a hypothesis. A model that looks optimal on retrospective data, whether a subphenotype classifier, a causal forest, or a reinforcement-learning policy, has not been shown to help patients until it is validated prospectively, in real use, against a comparator. The AI Clinician is the cautionary emblem of this truth, a sophisticated system that appeared to outperform clinicians and whose own evaluation could not tell whether it would help or harm. The future of precision critical care is not the algorithm. It is the trial of the algorithm.
That closes our four-week precision arc, and with it a seven-week run on evidence mastery that began with target trial emulation, the Fragility Index, and Bayesian reanalysis, and continued through treatment effect heterogeneity, the ARDS subphenotypes, the sepsis subphenotypes, and the causal machine learning that aims to bring it all to the individual. Next Thursday I want to put the entire toolkit to work, taking a single high-impact 2026 trial and appraising it through every lens we have built, the fragility of its significance, the probability of its benefit, the heterogeneity it may conceal, and the honesty of its causal claims. Seven weeks of tools. One trial. A capstone in how a modern intensivist, and a modern interprofessional team, should actually read the evidence that changes practice.
References
Wager S, Athey S. Estimation and inference of heterogeneous treatment effects using random forests. J Am Stat Assoc. 2018;113(523):1228-1242. doi:10.1080/01621459.2017.1319839 [verify DOI before press]
Athey S, Imbens G. Recursive partitioning for heterogeneous causal effects. Proc Natl Acad Sci U S A. 2016;113(27):7353-7360. doi:10.1073/pnas.1510489113 [verify DOI before press]
Athey S, Tibshirani J, Wager S. Generalized random forests. Ann Stat. 2019;47(2):1179-1203. doi:10.1214/18-AOS1709 [verify DOI before press]
Komorowski M, Celi LA, Badawi O, Gordon AC, Faisal AA. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nat Med. 2018;24(11):1716-1720. doi:10.1038/s41591-018-0213-5
Gottesman O, Johansson F, Komorowski M, et al. Guidelines for reinforcement learning in healthcare. Nat Med. 2019;25(1):16-18. doi:10.1038/s41591-018-0310-5
Jeter R, Josef C, Shashikumar S, Nemati S. Does the “Artificial Intelligence Clinician” learn optimal treatment strategies for sepsis in intensive care? arXiv. 2019;arXiv:1902.03271.
Bica I, Alaa AM, Lambert C, van der Schaar M. From real-world patient data to individualized treatment effects using machine learning: current and future methods to address underlying challenges. Clin Pharmacol Ther. 2021;109(1):87-100. doi:10.1002/cpt.1907 [verify DOI before press]
Sverdrup E, Petukhova M, Wager S. Estimating treatment effect heterogeneity in psychiatry: a review and tutorial with causal forests. Int J Methods Psychiatr Res. 2025;34(2):e70015. doi:10.1002/mpr.70015
Enriching patient populations in ICU trials: reducing heterogeneity through machine learning. Curr Opin Crit Care. 2025;31(4):410-416. [author list and full citation to be confirmed before press]
Pirracchio R, Hubbard A, Sprung CL, Chevret S, Annane D. Assessment of machine learning to estimate the individual treatment effect of corticosteroids in septic shock. JAMA Netw Open. 2020;3(12):e2029050. doi:10.1001/jamanetworkopen.2020.29050
Chen Y, Li T. Early metoprolol use in ICU patients with congestive heart failure is associated with increased 30-day mortality: a causal machine learning study. Front Pharmacol. 2026;17:1771969. doi:10.3389/fphar.2026.1771969
Kent DM, Steyerberg E, van Klaveren D. Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects. BMJ. 2018;363:k4245. doi:10.1136/bmj.k4245
Kent DM, Paulus JK, van Klaveren D, et al. The Predictive Approaches to Treatment effect Heterogeneity (PATH) statement. Ann Intern Med. 2020;172(1):35-45. doi:10.7326/M18-3667
Iwashyna TJ, Burke JF, Sussman JB, Prescott HC, Hayward RA, Angus DC. Implications of heterogeneity of treatment effect for reporting and analysis of randomized trials in critical care. Am J Respir Crit Care Med. 2015;192(9):1045-1051. doi:10.1164/rccm.201411-2125CP
Seymour CW, Kennedy JN, Wang S, et al. Derivation, validation, and potential treatment implications of novel clinical phenotypes for sepsis. JAMA. 2019;321(20):2003-2017. doi:10.1001/jama.2019.5791
Clinical Disclaimer
The content above is for educational purposes only and is not intended to replace clinical judgment, institutional protocols, or care delivered by qualified healthcare professionals. Patient care decisions should always be individualized, made in collaboration with the full interprofessional team, and aligned with current local guidelines, regulatory standards, and the patient’s clinical context. ICCN is not responsible for clinical actions taken solely on the basis of this article.
Javier Amador-Castaneda, BHS, RRT, FCCM | Founder & CEO, ICCN