How to Add a Decision Model to a Vibecoded App

Quick answer
Put the decision model in front of a branch you already have: support routing, content moderation, lead scoring, or an agent’s next action.
Keep permissions, limits, arithmetic, and side effects in deterministic code; let the model answer only the semantic question.
Gate on confidence: act above the threshold, fall back to an LLM or a human below it.
Log the state hash, question-set version, model version, answer, confidence, and what the app did next.
Run in shadow mode first, compare against current behavior, then enable for a small percentage.

Where does a decision model belong?

It belongs on a branch, not at the center of the architecture. A branch is a place where the app already chooses between a small number of paths: which queue gets a ticket, whether a comment needs review, which retrieval index to search, whether an agent may run a tool. Those branches are frequent, they need a judgment rather than a paragraph, and a wrong answer is usually recoverable. That is the shape a decision model is built for.

The order is deterministic code, then the model, then a handler. Code enforces the hard policy first: authentication, blocked data classes, spend limits, allowed destinations. The model classifies the semantic part. The handler performs the action and validates it again. If the model is unavailable or unsure, the request falls back to a human or a stronger model. The model never becomes the authorization layer.

Request
  │
  ├─ deterministic policy: auth, limits, blocked data, allowed actions
  │
  ├─ decision model: semantic question (route, urgency, risk, intent)
  │       │
  │       ├─ confidence >= threshold ──► handler validates and acts
  │       │
  │       └─ confidence <  threshold ──► human review or stronger LLM
  │
  └─ audit log: input hash, model version, answer, confidence, outcome

Step 1 — pick one branch and define the questions

Start with one branch and one outcome you can measure. Support triage is a good first project: the input is text, the labels are stable, and a wrong route costs a reassignment rather than a refund. Write down the current behavior first, including the rules or LLM prompt that handles it today. That baseline is what the model has to beat, and it is easy to forget once the new integration is in place.

Write each question as one atomic judgment. “Which team and how urgent?” is two questions, not one; combining them makes the answer hard to threshold and impossible to debug. Give every option a description that distinguishes it from the others, and include an explicit fallback option such as human_review when the input is ambiguous or outside the model’s scope.

questions = {
    "department": {
        "type": "choice",
        "instructions": "Which queue should handle this ticket?",
        "criteria": {
            "billing": "invoices, payments, refunds, subscription changes",
            "technical": "bugs, outages, integrations, API errors",
            "account": "login, passwords, profile access, MFA",
            "human_review": "ambiguous, sensitive, or outside the other options",
        },
    },
    "urgent": {
        "type": "noul",
        "instructions": "Does the ticket describe an outage or lost revenue?",
    },
}

Step 2 — write the decision function

Wrap the provider in one function that returns a normalized decision object. The function should not send emails, issue refunds, or close tickets; it should only return a route, a confidence, and a reason. That separation lets you test the policy without side effects, retry the model call safely, and run the same decision in shadow mode without changing anything for the user.

The example uses Jev, but the shape is the same for Laya or Clef: send the state and questions, read the typed answer, apply the threshold, and return a fallback when the answer is uncertain or the call fails. Keep the model name and the threshold in configuration, and log both with the result.

from dataclasses import dataclass
from typesafe_sdk import Choice, Noul, TypeSafeClient

MODEL = "jev-1.14.0"
CONFIDENCE_FLOOR = 0.75

@dataclass(frozen=True)
class Decision:
    route: str
    urgent: bool
    confidence: float
    reason: str

def decide_ticket(ticket: str) -> Decision:
    try:
        with TypeSafeClient(model=MODEL) as client:
            response = client.system_one(
                state=ticket,
                questions={
                    "department": Choice(
                        instructions="Which queue should handle this ticket?",
                        criteria={
                            "billing": "invoices, payments, refunds",
                            "technical": "bugs, outages, integrations",
                            "account": "login, passwords, profile access",
                            "human_review": "ambiguous or sensitive",
                        },
                    ),
                    "urgent": Noul(
                        instructions="Does the ticket describe an outage or lost revenue?",
                    ),
                },
            )
    except Exception as exc:
        return Decision("human_review", False, 0.0, f"model_error:{type(exc).__name__}")

    department = response.choices["department"]
    urgent = response.nouls["urgent"].noul >= 0.9

    if department.confidence < CONFIDENCE_FLOOR:
        return Decision("human_review", urgent, department.confidence, "low_confidence")

    return Decision(department.choice, urgent, department.confidence, "model_choice")

Step 3 — keep the action in your code

A decision is not an action. Map the returned route to an allow-listed handler, and validate the preconditions again inside that handler. The model can say “billing”, but the billing handler should still check that the ticket exists, that the user is authorized, and that the operation is idempotent. If the model returns an option your code does not recognize, treat it as a failure rather than guessing.

This is also where you enforce the limits that keep a wrong decision cheap. A routing decision can be reversed by moving a ticket; a refund cannot. If the branch triggers a destructive action, put a confirmation step or a human approval between the decision and the action. The decision model’s confidence is a useful signal for where to put that gate, not a replacement for it.

ROUTES = {
    "billing": "billing_queue",
    "technical": "engineering_queue",
    "account": "account_queue",
    "human_review": "triage_queue",
}

def handle_ticket(ticket_id: str, decision: Decision) -> None:
    queue = ROUTES.get(decision.route)
    if queue is None:
        queue = ROUTES["human_review"]

    # The handler validates again; the model does not authorize the action.
    enqueue_ticket(ticket_id=ticket_id, queue=queue, priority="high" if decision.urgent else "normal")
    audit(
        ticket_id=ticket_id,
        route=decision.route,
        confidence=decision.confidence,
        reason=decision.reason,
        model=MODEL,
    )

Step 4 — log and observe decisions

Log the decision, not just the outcome. A useful record includes a hash of the input rather than the full text, the question-set version, the model version, the selected answer, the probability distribution, the confidence, the latency, and what the app did next. The input hash lets you find the case later without storing sensitive text in a log; the question-set and model versions let you explain why a decision changed.

Build three views from those logs: the share of decisions below the threshold, the accuracy on a sampled review set, and the rate at which a route is overridden by a human. A rising low-confidence rate usually means the inputs are drifting or a label is ambiguous. A stable confidence rate with falling accuracy means the model or the question set is stale. Both are visible before users complain if you look at them.

{
  "decision_id": "dec_01J9...",
  "input_sha256": "9f2c...",
  "question_set": "support-triage@2026-10-20",
  "model": "jev-1.14.0",
  "answer": "technical",
  "confidence": 0.93,
  "probabilities": { "billing": 0.04, "technical": 0.93, "account": 0.02, "human_review": 0.01 },
  "latency_ms": 214,
  "outcome": "engineering_queue",
  "reviewed": false
}

Step 5 — roll out in shadow mode

Before the model takes an action, run it beside the current system. Send each request through both paths, log both answers, and keep the existing behavior in control. After a few hundred requests, compare the answers on the cases where they disagree and count how often the new model would have been better or worse. That review is more informative than an offline benchmark because it includes the inputs the app actually receives.

Then enable the model for a small percentage, with a kill switch and a budget cap. Watch the low-confidence rate, the human override rate, and the error rate. Increase only when those numbers are stable. A decision layer is infrastructure; the rollout should look like a rollout, not a flip of a switch on a Friday afternoon.

Where this bites vibecoders
An AI assistant will often build the whole feature around the model: the model picks the action, the model writes the message, the model decides whether to refund. That is the wrong boundary. Ask for the opposite: a decision function that returns data, a policy layer that maps data to allowed actions, and a fallback that a human owns. The result is less impressive in a demo and far easier to operate when the model is wrong.

Where AI coding assistants get this wrong

Letting the model choose and execute an action in one step.
Combining several judgments into one question and then guessing which part failed.
Skipping shadow mode because the offline test set looked good.
Logging the full customer message instead of a hash and the fields needed for review.
Making the confidence threshold a magic number with no recorded rationale.
Treating a provider outage as a normal low-confidence answer instead of a distinct fallback path.

Checklist

Pick one branch with a measurable outcome.
Record the current behavior as the baseline.
Write one atomic question per judgment and include a fallback option.
Return a decision object; keep side effects in a separate handler.
Set a confidence threshold and a fallback for timeouts and errors.
Log input hash, question version, model version, answer, confidence, and outcome.
Run in shadow mode, review disagreements, then enable gradually.

FAQ

Can I use a decision model for the whole app?

No. It answers closed questions; it does not generate text, plan multi-step work, or execute actions. The practical architecture uses it for the branches and keeps an LLM for language and ordinary code for policy. Trying to route every decision through one model produces a system that is harder to debug and no cheaper than the alternative.

How many questions should one call contain?

As many independent judgments as the branch needs, but each one atomic. Jev and Laya evaluate questions in parallel, so adding a question has little latency cost. The limit is interpretability: if two questions overlap, a disagreement between them is hard to resolve. Start with one or two and add a question only when a specific failure calls for it.

What confidence threshold should I start with?

Start with a number you can justify from the cost of errors, then measure. A routing decision that is easy to reverse can tolerate a lower threshold than one that triggers a refund. Run the threshold over your labeled set, look at coverage and precision at each level, and choose the point where the remaining errors are ones a human can absorb.

What happens when the model is down?

Fall back to the behavior you had before the integration, or to a human queue. Make the fallback explicit and test it by forcing a timeout. A provider outage should not become a silent low-confidence answer, because that hides the cause and may route every request through the same path. Treat transport errors and low confidence as different states with different handling.

Can I switch from Jev to Laya later?

Yes, if the decision logic sits behind your own interface and the question definitions live in your repository. Normalize the answer shape, keep the thresholds and routes provider-neutral, and re-run the evaluation after the swap. A direct integration that reads one provider’s response fields throughout the app makes the switch much more expensive.