> For the complete documentation index, see [llms.txt](https://docs.bitnimbus.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bitnimbus.io/resources/confidence-scores-and-model-familiarity.md).
# Confidence Scores and Model Familiarity
Why a confidence score is only as good as how well the model knows your task
## Introduction
Have you ever wondered how much you can actually trust an AI model's output? Not whether the answer looks right, but whether the model genuinely knew what it was doing when it generated it. Today, we're going to look at a property of language models that most people overlook: confidence scores — and what they really measure.
When you run a classification or decision-support task through a language model, you can ask it not just for an answer, but for a measure of how certain it is. That number feels reassuring. But as we'll show with a concrete experiment, confidence isn't the same as correctness. It's a measure of something more specific: how familiar the model is with the kind of task you're asking it to do.
## What Is a Confidence Score?
A confidence score in a classification model is derived from the probability distribution over its possible outputs. When a model is certain, that distribution is peaked: one answer gets a high probability and the others are near zero. When it's uncertain, the distribution is flat: many answers are roughly equally likely. Confidence is calculated as 1 minus the normalized entropy of that distribution. (Fireship, 2024)
In plain terms: entropy measures how 'spread out' the probabilities are. High entropy means spread out, which means uncertain. Low entropy means concentrated, which means confident. A confidence score of 0.81 means the model had one dominant answer in mind. A confidence score of 0.16 means it was essentially guessing.
Confidence = 1 − H(p) / H\_max
Where H(p) is the Shannon entropy of the model's output distribution and H\_max is the maximum possible entropy for that number of output classes. A score near 1 means the model is very sure. A score near 0 means it has no idea.
## The Experiment: Scoring Assessment Findings with Laya
We ran a straightforward test using Laya, our decision-classification engine. A compliance-style scoring rubric was presented to Laya as a series of yes/no and categorical questions about a written finding. Laya was asked to evaluate the finding across five criteria and return both an answer and a confidence score for each.
The catch: Laya was routed to a general-purpose checkpoint for this task. It hadn't been fine-tuned on compliance assessment, legal review, or any task resembling the structured rubric we were using. Here are the results:
| Question | Model Answer | Confidence | Usable for decisions? |
| -------------------------------------------------------- | ------------ | ---------- | --------------------- |
| Is this finding relevant to the scope of the assessment? | Yes | 0.16 | No |
| Does this finding include all necessary details? | Partially | 0.21 | No |
| Is the finding supported by cited evidence? | Yes | 0.73 | Yes\* |
| Is the finding clearly written? | Yes | 0.32 | No |
| What is the overall quality of this finding? | Medium | 0.20 | No |
\* Evidence citation is a more binary, pattern-recognizable judgment — the model can detect the presence or absence of a citation without deep task familiarity. This explains the relatively higher confidence on that criterion alone.
## Why Confidence Tracks Familiarity, Not Correctness
Four of Laya's five scores sit below 0.5 — the conventional decision threshold for classification confidence. That doesn't necessarily mean the answers are wrong. It means the underlying checkpoint doesn't have a strong internal representation for this kind of question.
Think of it this way. Ask a general-purpose model 'Is the sky blue?' and it'll answer with very high confidence, because that fact pattern is baked deeply into its training. Ask it 'Does this finding adequately cite primary statutory authority in the context of a Tier 2 compliance review?' and the probability distribution flattens out, because the model has never been specifically trained to make that judgment.
Confidence tracks familiarity. When the model knows the territory, it's fast and decisive. When it doesn't, it hedges — and the numbers show it.
## The Dangerous Case: High Confidence on the Wrong Task
Low confidence on an unfamiliar task is actually the better outcome. The model is signaling its uncertainty. You can catch it.
The dangerous case is when a model produces high confidence on a task it's not calibrated for — because the task superficially resembles something it does know well. In that scenario, the scores look authoritative. Thresholds pass. Flags don't trigger. But the underlying judgment is shaky.
This is why confidence calibration matters as much as raw accuracy. A well-calibrated model knows when it doesn't know. An uncalibrated one doesn't.
## What This Means for Your Deployment
If you're using a language model for classification, routing, or decision support, the confidence scores are only as reliable as the model's familiarity with your specific task. A few practical implications:
* Don't treat confidence thresholds as universal. A 0.7 threshold that works well for one task may be meaningless for another if the model wasn't trained on that task type.
* General-purpose models are a starting point, not an endpoint. They're useful for broad tasks, but for high-stakes structured decisions, task-specific fine-tuning is usually necessary to get reliable confidence signals.
* Log and audit confidence distributions, not just scores. If you see a flat distribution across criteria (many scores clustered near 0.2-0.3), that's a signal the model is operating outside its training distribution.
* Test with held-out examples before deploying. Run your rubric through a sample set and look at the confidence histogram. You want a bimodal distribution — high confidence on clear-cut cases and honest uncertainty on edge cases — not a uniform flat one.
## The Path Forward: Fine-tuning on Your Task
The fix is straightforward in principle, though it takes effort in practice: fine-tune a checkpoint on examples that match your exact task.
This doesn't mean training from scratch. Modern fine-tuning approaches let you take an existing capable model and specialize it using a relatively small set of labeled examples — often a few hundred to a few thousand, depending on task complexity. The resulting checkpoint produces peaked distributions on your task type, meaning confidence scores become meaningful decision signals rather than noise.
A few things to keep in mind:
* Your training examples need to reflect the full range of answers you'll encounter in production, including edge cases and borderline examples.
* Evaluate calibration separately from accuracy. A model can be accurate but poorly calibrated (confidently wrong) or well-calibrated but lower accuracy (correctly uncertain on hard cases).
* Revisit calibration when your task distribution shifts. A fine-tuned checkpoint is calibrated to the distribution it was trained on. If the nature of your inputs changes significantly, recalibration may be needed.
## Conclusion
Confidence scores are not a measure of how correct a model is. They're a measure of how familiar a model is with the kind of question you're asking. A general-purpose model presented with a structured assessment rubric will produce low, flat confidence scores — not because it's incompetent, but because it's operating outside the specific task distribution it was trained on.
The implication is practical: if you're making decisions based on model confidence, make sure the model was trained on tasks that match yours. Otherwise, you're reading the signal wrong.
We hope this was a good read. If you're interested in stuff like this, please stay tuned as we continue our journey into the nitty-gritty of large language models.
## References
Fireship. (2024). AI confidence scores explained. YouTube.
