How Good Is Medical AI, Really? What the MAST Benchmark Tells Us

Preamble

Artificial intelligence can now answer medical questions, generate differential diagnoses and interpret clinical information with impressive fluency.

It is tempting, therefore, to ask a simple question: How good is medical AI now?

A new initiative called the Medical AI Superintelligence Test (MAST) is trying to answer that question rather differently. [1]–[4]

Instead of asking whether an AI can pass another medical examination, MAST evaluates models across several capabilities relevant to clinical medicine.

The results are impressive. But they are also a useful reality check.

On the MAST general leaderboard last updated on 8 October 2026, the leading general-purpose AI models clustered around the mid-50s to approximately 60%, with the highest composite score at 60.2%. [1]

That sounds surprisingly modest for systems increasingly described as approaching—or sometimes exceeding—doctors on selected medical tasks. [7]

So what does the score actually tell us? And perhaps more importantly, what does it not tell us?

Medical AI at an examination desk contrasted with real patient care at the bedside — *Medical AI may excel at answering examination questions, but real clinical care requires much more than getting the diagnosis right.*

1. Medicine is not an examination

AI has become very good at medical questions.

That is important—but medicine is not an examination.

A patient does not usually arrive with five possible diagnoses and ask us to select the best answer.

Clinical medicine requires more than recalling information or identifying a diagnosis. It involves deciding what to do next, interpreting investigations, recognising uncertainty, avoiding harm and changing course when new information appears.

An AI might reach the correct diagnosis and still recommend inappropriate management.

This is one of the problems MAST is trying to address. Researchers involved in the ARISE AI Research Network argue that existing benchmarks do not adequately capture the range of abilities required for clinical medicine. [2][3]

I have previously discussed the broader opportunities and limitations of artificial intelligence in clinical practice in AI Has Entered the Consulting Room. [8] Here, the question is deliberately narrower:

How should we actually measure whether an AI is good at medicine?

2. What is MAST?

MAST stands for the Medical AI Superintelligence Test.

Launched in July 2026, it is a multi-institutional collaborative project designed as a living benchmark platform for evaluating medical AI on clinically meaningful tasks. [2][3]

Despite the ambitious word superintelligence, its developers are not claiming that today's AI systems have achieved medical superintelligence.

Rather, MAST attempts to provide a framework for measuring progress towards increasingly capable medical AI.

MAST-General combines five domains: diagnostic reasoning, management reasoning, safety, radiology and multimodal medical-image performance. The broader platform also evaluates agentic tasks separately; these are not part of the five-domain MAST-General composite. [1][4][5]

That represents an important departure from simply giving an AI another collection of medical examination questions.

Clinical competence is not one ability.

These represent different levels of evidence: answering structured medical questions, performing clinically relevant benchmark tasks, and improving outcomes in real patients. MAST principally addresses the second level—not the third.

3. Why only around 60%?

The MAST general leaderboard provides an interesting snapshot. On the leaderboard as last updated on 8 October 2026 (and checked on 10 October 2026), the highest composite score among the listed general-purpose models was 60.2%, with several leading systems clustered within only a few percentage points of one another. [1] An earlier refresh, on 12 August 2026, put the top score at 62.9%. [6] The exact figures move between snapshots, which is itself a reminder that the leaderboard is still in preview.

At first sight, this seems surprising. How can AI systems that appear so impressive when answering medical questions score only around 60%?

Part of the explanation lies in how MAST is constructed. The overall score brings together performance across several different capabilities. MAST currently uses an equal-weight harmonic mean of its clinical dimensions for its composite score. [5]

Why does that matter? A harmonic mean penalises uneven performance. Being exceptionally good in one area cannot completely compensate for being substantially weaker in another.

For example, in the reported technical snapshot, GPT-5.6 Sol scored 75.7% for diagnostic reasoning but 43.3% for medical images. [4] A high score in one domain does not guarantee comparable performance elsewhere.

Being good at medical questions is not the same thing as being good at medicine.

4. Diagnosis is only the beginning

Consider a patient with pneumonia. Correctly identifying pneumonia is important. But the clinical job has only begun.

Does the patient require admission? How severe is the illness? Which treatment is appropriate? Could another diagnosis explain the findings? What does the imaging show? What happens if the patient deteriorates?

Getting the diagnosis right demonstrates an important capability. It does not demonstrate the ability to manage that patient safely from beginning to end.

This is why a single number describing the “intelligence” of a medical AI can be misleading.

Good at what?

5. What does 60% actually mean?

A MAST score of approximately 60% does not mean that AI gets four out of ten patients wrong.

MAST combines performance across selected benchmarks measuring different capabilities. [1]–[4] It is not a clinical error rate.

It does not directly tell us how many patients an AI would diagnose correctly, treat appropriately, admit unnecessarily, discharge safely—or harm—if it were introduced into everyday clinical practice.

Neither should we make the opposite mistake. A composite score of 60% does not mean the technology is only “60% good”.

The headline number therefore matters less than understanding: Where does this AI perform well—and where does it not?

6. The benchmark is not the bedside

MAST remains a benchmark. Real patients are considerably messier.

Their histories may be incomplete. Symptoms can be vague. Several diseases may coexist. Investigations can conflict. Patients may not take the treatment prescribed. Social circumstances influence what treatment is practical. And sometimes the patient simply does not fit the expected pattern.

No benchmark can completely reproduce that complexity.

MAST represents an important attempt to make medical-AI evaluation more clinically meaningful, but its score should not be mistaken for evidence that the same performance will necessarily occur in routine patient care. [2][3]

There is another reason for caution. The MAST technical leaderboard remains in preview, with validation and tuning continuing. Not every model has results for every benchmark, and the developers note that exact scores may change. [4]

MAST is therefore better regarded as an evolving framework for comparing specific AI capabilities—not as a validated measure of how safely an AI would practise medicine independently.

7. Perhaps we have been asking the wrong question

Much of the excitement surrounding medical AI has been driven by questions such as:

Can AI pass a medical examination?
Can AI diagnose better than a doctor?
Which AI model is best?

These are interesting questions. But they may also be too simple.

A model might be excellent at diagnostic reasoning but less capable at management. Another may perform well with written clinical problems but struggle with medical images. Another may answer an isolated clinical question correctly yet perform less well when required to complete several related tasks.

There may therefore be no single “best medical AI” for every clinical task.

The answer depends upon what we are asking it to do.

Diagnosis? Image interpretation? Treatment recommendation? Drug-interaction checking? Documentation? Clinical decision support? Patient information?

These are very different jobs. An AI validated for one should not automatically be assumed to be equally competent at another.

Before using a system in clinical practice, we should also ask whether its performance has been validated in patients and healthcare settings resembling our own—including the clinical pathways and resources available in Malaysia.

8. So, how good is medical AI, really?

For now, perhaps the fairest answer is:

Very good at some things. Much less good at others. And improving remarkably quickly.

That may sound less dramatic than declaring that AI has surpassed doctors—or that it remains unreliable. But it is probably closer to reality.

Medical AI does not have to know everything to be useful. Doctors do not know everything either.

What matters is understanding the capabilities and limitations of the tool being used.

MAST is useful because it reminds us that medicine is not a single cognitive task—and intelligence is not a single score.

Its most interesting message may therefore not be that the leading general-purpose models scored around 60% on its composite benchmark. It is that the same AI can perform very differently depending on the medical task we ask it to do.

So perhaps we should stop asking: “How intelligent is this AI?”

and start asking:

How good is this AI at the particular medical task I am asking it to do?

That may be the much more useful question.

Sources and further reading

MAST and medical AI benchmarking

Commentary and broader context

Published 47/2026 · 10 October 2026 · No corrections to date · Corrections policy