Why Confidence Scores Aren't Enough for Safety-Critical Perception
A single confidence score can't distinguish a model that has truly seen a scenario before from one that's extrapolating — and on a factory floor, that gap is where incidents happen.
The Problem With a Single Number
A perception model that reports "98% confident" gives an operator exactly one number to trust. On a factory floor, that number doesn't distinguish between a model that has genuinely seen this exact scenario a thousand times and one that is extrapolating past the edge of its training distribution. Both can report the same score.
Where This Breaks Down in Practice
Lighting changes, a new pallet color, a sensor lens picking up condensation — any of these can push a model outside its comfort zone without its confidence score reflecting that shift. A system tuned purely to maximize accuracy on a validation set has no built-in mechanism for flagging "I haven't seen this before," only for reporting how it would answer anyway.
What We Build Instead
We treat calibration as a first-class requirement, not an afterthought: cross-checking a single sensor's read against an independent modality (thermal against RGB, vibration against acoustic) before a decision reaches an actuator. Disagreement between modalities is itself a signal — often a more useful one than either modality's raw confidence score.
The Standard to Hold Yourself To
A model that is confident and wrong is not a weaker version of one that is right — on a factory floor, it is the more dangerous one. Perception systems for physical environments should be judged on how they behave at the edges of their training data, not just their average-case accuracy.
