How to Evaluate a Decision Model Before You Ship It

Quick answer
Build a labeled set from real inputs, not invented examples, and include the ambiguous cases your current system gets wrong.
Measure accuracy or F1 for choices, ordinal error for scores, and calibration such as Brier score or reliability bins for probabilities.
Pick a confidence threshold from the cost of a wrong action, not from the model’s average confidence.
Compare against the baseline you are replacing: rules, an LLM prompt, or a human reviewer.
Run in shadow mode on live traffic before letting the model take any action.

What does a good evaluation need?

It needs a labeled set drawn from the inputs the app actually receives, a baseline to compare against, and a metric tied to the decision’s cost. A few hundred examples is enough to see a large difference and not enough to detect a small one, so treat the result as evidence rather than proof. Include the cases that are ambiguous, multilingual, unusually long, or deliberately adversarial, because those are the ones a demo set omits.

Label quality matters more than set size. Two reviewers should agree on most labels; where they do not, the question or the option criteria are probably ambiguous. Fix the definition before collecting more data. A model cannot reliably learn a distinction your own reviewers cannot apply consistently, and a benchmark built on inconsistent labels will reward the wrong behavior.

Step 1 — define the decision and the cost of errors

Write down what the model will decide, what happens on a correct answer, and what happens on each kind of wrong answer. For a triage model, a false negative that sends an outage to the billing queue is worse than a false positive that sends a billing question to engineering. For a moderation model, the asymmetry may run the other way. The costs determine which metric matters and where the threshold belongs.

Then set acceptance criteria before looking at results. Examples: at least 90% accuracy on the held-out set, no more than 10% of live requests below the confidence threshold, p95 latency under 300 ms, and a cost per thousand decisions under a dollar. The numbers are yours; the point is to decide what passing means before you are tempted to rationalize a result.

Step 2 — build the labeled set

Sample from production, not from the documentation. Stratify by the classes you care about so a rare but expensive class is not drowned out by the common one. If you have historical human decisions, use them as labels after a review pass; if you do not, have a domain expert label a sample and record their reasoning for the disagreements. Keep a held-out split that you do not inspect while tuning the questions.

Store the label, the input, the question-set version, and the reviewer. When the question changes, the old labels may no longer mean the same thing, so version the dataset with the question set. This is the same discipline as a test suite: the evaluation is only useful if it is reproducible and if a change that breaks it is visible.

Step 3 — measure the right metrics

Use accuracy or macro-F1 for choice questions, especially when the classes are imbalanced. Use mean absolute error or ordinal error for scores, because a score is ordered and treating it as a plain category loses information. For yes/no questions, use Brier score or log loss, which reward calibrated probabilities rather than confident guesses. Report a confusion matrix alongside the headline number so you can see which classes are being traded against each other.

Calibration deserves its own measurement. Bin the predicted probabilities and compare each bin’s average prediction with the observed frequency. A model that says 0.9 and is right 60% of the time is not calibrated, even if its accuracy is decent. The Brier score combines calibration and resolution into one number, which is useful for comparing models but less useful for explaining a failure than the reliability bins.

import numpy as np

def brier_score(y_true, p_yes):
    """y_true: 0/1 labels; p_yes: predicted probability of 1."""
    y_true = np.asarray(y_true, dtype=float)
    p_yes = np.asarray(p_yes, dtype=float)
    return float(np.mean((p_yes - y_true) ** 2))

def reliability_bins(y_true, p_yes, edges=(0.0, 0.5, 0.7, 0.8, 0.9, 1.01)):
    y_true = np.asarray(y_true, dtype=float)
    p_yes = np.asarray(p_yes, dtype=float)
    for low, high in zip(edges, edges[1:]):
        mask = (p_yes >= low) & (p_yes < high)
        if mask.any():
            print(f"{low:.2f}-{high:.2f}: n={mask.sum():3d} "
                  f"predicted={p_yes[mask].mean():.3f} observed={y_true[mask].mean():.3f}")

# Example: a yes/no question with 6 predictions.
y = [1, 1, 0, 1, 0, 1]
p = [0.95, 0.82, 0.61, 0.73, 0.91, 0.55]
print("Brier:", round(brier_score(y, p), 4))
reliability_bins(y, p)

Step 4 — choose a confidence threshold

A threshold turns a probabilistic answer into a policy: act above it, escalate below it. Choose it by looking at coverage and precision together. Raising the threshold sends more requests to humans and makes the remaining automated decisions more accurate; lowering it does the opposite. There is no universally correct number, because the trade depends on the cost of a review versus the cost of a wrong action.

Compute the trade-off on your held-out set rather than guessing. For each candidate threshold, report the share of answers that pass and the accuracy among those answers. Pick the point where the marginal review cost is worth the marginal accuracy gain. For multi-class choices, consider per-class thresholds when one class is much more expensive to get wrong than the others.

import numpy as np

def coverage_and_accuracy(confidences, correct, threshold):
    confidences = np.asarray(confidences, dtype=float)
    correct = np.asarray(correct, dtype=bool)
    mask = confidences >= threshold
    if not mask.any():
        return 0.0, float("nan")
    return float(mask.mean()), float(correct[mask].mean())

# Replace these with the confidences and correctness from your test run.
confidences = [0.55, 0.62, 0.71, 0.78, 0.81, 0.86, 0.91, 0.95]
correct =     [0,    0,    1,    1,    1,    1,    1,    0]

for threshold in (0.5, 0.7, 0.8, 0.9):
    coverage, accuracy = coverage_and_accuracy(confidences, correct, threshold)
    print(f"threshold={threshold:.1f} coverage={coverage:.0%} accuracy={accuracy:.0%}")

Step 5 — compare against the baseline

The model has to beat what you are already doing, not a random baseline. If the current system is a regex and a queue, evaluate the regex on the same labeled set. If it is an LLM prompt, run that prompt on the same inputs and score its answers the same way. Include the cost and latency of each option, because a model that is two points more accurate but ten times slower may not be an improvement.

Report the difference with uncertainty. A few hundred examples can support a large claim and not a small one; if the gap is a couple of percentage points, you probably need more data or a narrower question. The purpose of the comparison is not to produce a winner on paper but to decide whether the change is worth the integration, evaluation, and operational work.

Step 6 — run in shadow mode and watch for drift

Shadow mode sends live requests through the new decision path without letting it act. Log the model’s answer beside the current system’s answer, then review the disagreements. This catches the failures an offline set misses: a new product name, a language mix, a sudden spike in one request type, a field that is empty in production but populated in fixtures. It also gives you a realistic estimate of the low-confidence rate.

After launch, monitor the confidence distribution and a sampled review set. Drift is not only the model changing; it is also your inputs changing, your question set changing, or your users learning how to phrase requests differently. Re-run the evaluation whenever any of those changes, and keep the old labeled set so you can compare across versions. A decision model without a re-evaluation habit is a model you no longer understand.

Where this bites vibecoders
The demo test is five examples that the assistant wrote to match the prompt. The production test is a few hundred messy inputs, labeled by someone who knows the domain, scored against the system you are replacing. Skipping that work feels like shipping faster, but it moves the discovery to production, where the feedback is a customer complaint instead of a failing assertion. The evaluation set is the test suite for the decision layer.

Where AI coding assistants get this wrong

Testing on examples written by the same assistant that wrote the prompt.
Reporting accuracy without a confusion matrix or the class distribution.
Tuning the confidence threshold on the same set used to report final accuracy.
Treating a confidence value as a probability of correctness without checking calibration.
Comparing against a random baseline instead of the system currently in production.
Launching without a shadow-mode period and a plan to re-evaluate after drift.

Checklist

Write the decision, the correct outcome, and the cost of each error.
Collect 100-300 labeled inputs from production and stratify by class.
Keep a held-out split that is not used for tuning.
Measure accuracy or F1, ordinal error, Brier score, and calibration bins.
Compute coverage and accuracy at several confidence thresholds.
Compare against the current rules or LLM on the same set.
Run in shadow mode, review disagreements, and monitor drift after launch.

FAQ

How many labeled examples do I need?

A few hundred is a practical starting point for a binary or small multi-class decision, and more if the classes are imbalanced or the errors are expensive. The exact number depends on the effect size you need to detect. Start with a stratified sample, measure agreement between reviewers, and add data where the confidence intervals are too wide to support a decision.

What is calibration and why does it matter?

Calibration means the predicted probabilities match observed frequencies: among answers predicted at 0.8, about 80% should be correct. It matters because the confidence threshold is a policy built on those numbers. A model can be accurate on average and badly calibrated in the high-confidence range, which makes the threshold behave differently from what you intended.

Should I evaluate the LLM prompt the same way?

Yes. The decision model is replacing a branch, and the branch has a current implementation, whether that is a rule or an LLM prompt. Run both on the same labeled set and score both with the same metric. Otherwise you are comparing the new model with your hopes for it rather than with the system users actually experience.

What is shadow mode?

Shadow mode runs the new decision path on live requests without letting it take action. You log what the model would have done and compare it with what the current system did. It surfaces distribution problems, missing fields, and low-confidence spikes before users are affected, and it gives you a realistic sample for review.

How often should I re-run the evaluation?

Whenever the model version, the question set, the option definitions, or the input distribution changes, and on a regular schedule even when nothing obvious changed. Keep the labeled set versioned alongside the questions. A quarterly review is a reasonable default for a stable app; a weekly one is appropriate during a launch or a period of rapid product change.