What is a Confidence Score?

A confidence score in document AI is a numeric value a model attaches to an output, indicating its own estimate of how likely that output is correct.

The score is the mechanism that makes automation safe, because it is what a system uses to decide which results it can accept without a person looking. Understanding what it does not mean is more useful than understanding what it does.

What a confidence score is not

It is not a calibrated probability. A field returned at 0.90 does not mean it is correct 90 percent of the time. Raw model outputs are typically over-confident, sometimes substantially, and the relationship between the number and real accuracy has to be measured on your own documents before it means anything.

It is not comparable across models or stages. A 0.85 from an OCR engine describes certainty about character shapes. A 0.85 from an extraction model describes certainty that a value belongs to a field. A 0.85 from a classifier describes certainty about document type. They are different quantities on the same scale and cannot be averaged or compared.

It is not a measure of correctness for the right reason. A model can be confidently right about the wrong thing, for example extracting a genuine date from the document that is not the date the field asked for. High confidence with wrong semantics is the failure mode that survives naive testing.

It does not detect the unfamiliar reliably. Models generally return middling-to-high confidence on inputs unlike anything they were trained on, rather than usefully low values. This is why an explicit unclassifiable path matters more than trusting low scores to catch novelty.

Calibration

Calibration is the process of mapping raw scores onto observed accuracy. It is done by taking a representative labeled sample, grouping predictions into score bands, and measuring what proportion in each band was actually correct.

The output is a table saying what a given score means in your environment. It commonly reveals that the useful decision boundary sits somewhere unintuitive, and that the raw number was systematically optimistic.

Calibration is environment-specific and expires. It has to be redone when the model changes, when a major new document source is added, and periodically as the input mix drifts.

The three-zone pattern

Most production systems partition the score range into three zones rather than two.

Auto-accept, above an upper threshold, passes without human involvement. This zone should be set from calibration data at the accuracy the process genuinely requires, not at a round number.

Human review, between the thresholds, queues for a person. The reviewer sees the value alongside the region of the document it came from and either confirms or corrects it.

Reject or re-route, below a lower threshold, does not go to the ordinary review queue. Very low confidence usually indicates a problem with the input rather than a borderline judgment: the wrong document type, an unreadable scan, a blank page. Sending these to the same queue as genuine borderline cases slows reviewers down with work that should have been returned to the sender or re-captured.

Two thresholds rather than one is what separates a queue reviewers trust from one they learn to skim.

How thresholds drive straight-through processing

The straight-through processing rate, the proportion of documents completing with no human touch, is a direct function of where the upper threshold sits. Lowering it raises automation and raises error rate; raising it does the reverse.

That makes the threshold a business decision rather than a technical one, and it should be set from the cost of each error type. A wrong description on a purchase line is a minor annoyance. A wrong bank account on a payment instruction is a loss. Those fields do not belong at the same threshold, which is why thresholds are set per field and per document type rather than globally. See document classification and document data extraction.

What commonly goes wrong

One global threshold. A single value across all fields and types either over-automates the consequential fields or under-automates everything else.

Thresholds set once and never revisited. Input mix drifts, models are updated, and an operating point set at go-live stops being the right one.

Confidence displayed to reviewers as a percentage. Showing an uncalibrated 0.92 as “92% confident” invites reviewers to treat it as a probability and rubber-stamp accordingly.

Aggregating field confidences into a document score. A document is not usefully summarized by the mean of its field confidences, because one critical low-confidence field matters more than twenty high-confidence ones.

Contellect One applies per-field thresholds in the validate stage of intelligent document processing.