Create 'Humble' AI for Medicine

Artificial intelligence holds promise for aiding doctors in diagnosing patients and personalizing treatment options. However, an international group of scientists led by MIT warns that current AI systems may mislead doctors due to their overconfidence in erroneous decisions. One way to prevent such mistakes is to program AI systems to be more 'humble.' Such systems would indicate when they are uncertain about their diagnoses or recommendations and encourage users to gather additional information when in doubt.

Leo Anthony Celi, a senior research scientist at MIT, states that AI can be used not only as an oracle but also as a coach. This would enhance the ability to retrieve information and increase doctors' involvement in decision-making. Celi and his colleagues have created a framework that they believe can guide AI developers in designing systems that exhibit curiosity and humility. This new approach could allow doctors and AI to collaborate as partners, preventing AI from exerting excessive influence over medical decisions.

The MIT team notes that overconfident AI systems can lead to errors in medical environments. Previous studies have shown that ICU physicians tend to defer to AI systems they perceive as reliable, even when their intuition contradicts AI recommendations. Both physicians and patients are more likely to accept incorrect AI suggestions when viewed as authoritative.

Researchers emphasize that instead of systems providing overconfident but potentially incorrect advice, healthcare facilities should have access to AI systems that work more collaboratively with clinicians. They aim to incorporate human input into these human-AI systems to facilitate collective reflection and reimagining, rather than relying on isolated AI agents.

To create such a system, the consortium designed a framework that includes several computational modules that can be integrated into existing AI systems. One of these modules requires an AI model to evaluate its own certainty when making diagnostic predictions. Developed by consortium members, the Epistemic Virtue Score acts as a self-awareness check, ensuring the system's confidence is appropriately moderated by the inherent uncertainty and complexity of each clinical scenario.

With this self-awareness in place, the model can tailor its response to the situation. If the system detects that its confidence exceeds what the available evidence supports, it can pause and flag the mismatch, requesting specific tests or history that would resolve the uncertainty or recommending specialist consultation. The goal is to have AI that not only provides answers but also signals when those answers should be treated with caution.

This study is part of a broader effort by Celi and his colleagues to create AI systems designed by and for the people who will ultimately be most affected by these tools. Many AI models, such as MIMIC, are trained on publicly available data from the United States, which can introduce biases toward a certain way of thinking about medical issues and exclude others.