3 questions for...

Dr. Rosa E. Martín Peña
Group leader
Dr. Rosa E. Martín Peña investigates the ethical and epistemological dimensions of medical AI, particularly with regard to explainability, algorithmic bias and black box phenomena. A major contribution of her research is to show how AI systems change clinical thinking under uncertainty and what role the responsible handling of different types of data plays in healthcare. The aim of her work is to promote responsible decision-making processes between humans and AI and to promote interdisciplinary collaboration.
1.
How do we deal with the uncertainty that even highly accurate AI systems can make wrong decisions?
Uncertainty in AI is not just a matter of accuracy, it reflects deeper human expectations of perfection and clarity. We often hope that machines will remove the ambiguity we struggle with. But history reminds us that uncertainty has always been part of decision-making. Before X-rays or even basic hygiene practices transformed medicine, doctors operated with limited and often unreliable information.
Over time, we developed tools that extended our perception far beyond what our senses allow, from nanometers of light to deep brain activity. AI continues this trajectory, offering new ways to perceive patterns and make predictions. But even the most advanced models have limits. Some aspects of reality, such as missing data, changing contexts, or human values, remain difficult to capture computationally.
This is especially evident in domains such as oncology or intensive care, where prognoses rely on complex, evolving data, and uncertainty becomes an integral part of clinical judgment. Today, technical methods such as confidence intervals, model calibration, or robustness analysis can help quantify and manage uncertainty. However, no statistical technique can fully resolve the normative ambiguity that arises when AI systems are used in high-stakes decisions affecting human lives.
Rather than trying to eliminate uncertainty, we need to learn to work with it. That means clarifying what AI can and cannot do, and designing decision-making systems that include uncertainty as a feature, not a flaw. By making these boundaries visible, we don’t lose trust in AI, we gain the ability to trust it wisely and to situate it responsibly within human decision-making.
2.
What does justice mean in the context of medical AI, and how can it be operationalized in concrete terms?
Justice in medical AI is not merely about equal treatment or universal access, it requires recognizing asymmetries in data, resources, and lived experience. Operationalizing justice means asking not “Does this system work for most?” but rather “For whom does it fail, and why?”
Concretely, this entails using representative datasets, conducting regular bias audits, applying disaggregated performance metrics across subgroups, and ensuring stakeholder participation from the earliest stages of system design. It also involves building mechanisms for contestability, giving individuals not only the right, but also the practical means to understand, question, and appeal decisions that affect their health and dignity.
Justice further requires the generation of more context-sensitive knowledge. For instance, many cardiovascular risk prediction models perform worse for women or racialized populations because they are trained on homogenous, male-biased datasets. Similarly, the underdiagnosis of autoimmune conditions in women or the misdiagnosis of anxiety in patients with chronic fatigue syndrome illustrate how AI can replicate long-standing diagnostic blind spots unless critically examined.
Designing with justice in mind is therefore not only about the data we
use, but also about the questions we formulate and the patterns we
prioritize. Justice, in this context, is not a fixed outcome, but an
ongoing normative process, one that demands humility, vigilance, and
inclusive thinking, and that must be evaluated as rigorously as any
other system performance metric.
3.
How can we ensure that ethical principles, such as fairness and diversity, are integrated into the design of training data and processes from the start to prevent biases?
The biases of AI systems are not anomalies, they reflect how data has historically been collected, labeled, and prioritized, as well as how other forms of knowledge have been systematically ignored. For instance, a widely cited study found that an algorithm used in U.S. hospitals to allocate care for high-risk patients systematically underestimated the health needs of Black patients, because it relied on historical healthcare expenditures, a proxy that reflected unequal access to care, not actual clinical need. Another example is the persistent underdiagnosis of women in medicine, a field long shaped by male-centric symptom profiles and clinical standards.
These are not isolated flaws. They illustrate how AI systems can reinforce and even magnify existing patterns of injustice if ethical principles are not embedded from the outset. Medical AI, in particular, has enormous normative impact, it does not merely assist in diagnosis, it helps define what counts as normal, healthy, or pathological.
Integrating fairness and diversity therefore requires more than technical fixes. It calls for rethinking foundational assumptions, what counts as evidence, whose experiences are represented, and how difference and uncertainty are treated. From a technical standpoint, this involves applying subgroup fairness metrics, testing for counterfactual fairness, and using data documentation tools such as Model Cards and Datasheets for Datasets to ensure transparency about data provenance, limitations, and representativeness. These tools help move fairness from an abstract value to an operational practice embedded within model development workflows.
The field of machine ethics also plays a key role, not only by applying ethical principles externally, but by encoding normative reasoning into intelligent systems, guiding decisions toward values such as equity, dignity, and respect for difference.
Ultimately, ethical integration is not just about correcting past biases, but about designing with intention, through backcasting: defining the kinds of healthcare futures we want AI to support, and working backwards to inform how we collect data, structure models, and make decisions today.