Why a 99% accurate model can still be useless

Accuracy hides the base rate. The same 99% is a triumph or a disaster depending on how rare the thing you're detecting actually is.

The question

A system reports 99% accuracy. Is that impressive? The uncomfortable answer: 99% tells you almost nothing until you know how rare the thing is. The same number can mean a great tool or a dangerous one.

What's happening

Accuracy — the share of predictions that were right — is the headline metric in almost every model announcement. It is also the metric most likely to mislead, because it silently depends on the balance of the data.

How it actually works

Take a disease that affects 1 in 1,000 people. A test is "99% accurate." What does a positive result mean?

Actually sick Actually healthy
Test positive 99 999 1,098
Test negative 1 98,901 98,902
100 99,900 100,000

Out of 1,098 positive results, only 99 are truly sick — about 9%. So a "99% accurate" test, when applied to a rare condition, produces positives that are wrong 91% of the time.

The mechanism is the base rate. When the thing you're detecting is rare, even a small false-positive rate generates more false alarms than true detections, because there are so many healthy cases to mis-fire on.

precision = true positives / (true positives + false positives)
          = 99 / (99 + 999)
          ≈ 0.09   →   9%

Two more pieces fall out of the same math:

  • Precision vs. recall. Accuracy hides both. A model that flags everything has perfect recall and terrible precision; one that flags nothing has the reverse.
  • Confidence is not correctness. A model's confidence is certainty within its label set, not knowledge about the world. High confidence on a rare, mis-specified case is exactly where models are most confidently wrong.

Ask "how rare is this?" before you ask "how accurate is it?" The second number is meaningless without the first.

What's unknown

  • How often product decisions are actually driven by precision/recall versus the headline accuracy number. My suspicion is: far too often, but I have no data.
  • The best way to communicate this to non-technical readers without losing them in tables.
  • Open: how base rates shift when models are deployed on populations that differ from their training data — the same model can swing from useful to useless without changing a line.

Reading

  • Base-rate fallacy and confusion-matrix material (the standard statistics framing).
  • Precision/recall write-ups from the information-retrieval literature.