How to Productionize an AI App

Quick answer
A demo proves the model can produce a good answer once; production requires it to do so predictably, affordably, and safely for every user.
The five gaps to close are reliability, quality, cost, security, and operations — in that order, because each one changes the design.
Timeouts, retries with backoff, and a fallback path matter more than prompt polish, because model providers have outages and slow periods.
Add an evaluation set, a cost ceiling per task, and a feedback signal before launch; without them you cannot tell whether a prompt change helped.
Treat prompt injection and data exfiltration as application security problems, not model problems, and constrain what the feature can reach.

What has to be true before an AI feature goes live?

Five things. It must fail gracefully when the provider is slow or unavailable. It must produce acceptable output on a representative set, not just the demo prompt. It must have a cost ceiling per user and per task. It must not expose data or execute actions beyond its intended scope. And it must be observable enough to answer what happened for a specific request.

Each item changes the architecture. A fallback may need a second provider or a deterministic alternative. A cost ceiling may require routing or a smaller model. A scope limit may require a tool layer with allow-listed actions. Doing these after launch means rebuilding the feature under pressure.

How do you make it reliable?

Set a timeout on every model call and a deadline for the whole request. Retry only transient failures, with exponential backoff and jitter, and cap the attempts inside the token budget. Add a circuit breaker so a failing provider stops receiving traffic instead of consuming the user’s time on every request.

Define a degraded mode: a cached answer, a smaller model, a template response, or a clear message that the feature is unavailable. The worst outcome is a request that hangs until the platform times out, because it consumes resources and gives the user nothing. A fast, honest failure is a feature.

How do you handle quality?

Build an evaluation set of real inputs and expected properties before launch: at least a few dozen cases covering the happy path, the edge cases, and the failures that matter. Run it on every prompt or model change. Keep a sample of production outputs under review, because the set goes stale as users find new inputs.

Add a validator where the output has structure: schema validation, required fields, allowed values, length limits. When validation fails, retry once with the error included, then fall back. For free-text output, use a rubric and human review on a sample. ‘It looked good in testing’ is not a quality process.

How do you control cost and security?

Set a token budget per task and a spend cap per user or tenant, and alert before the cap. Measure cost per successful task rather than per call. Cache stable prompt prefixes and route easy tasks to cheaper models only after evaluating them. Without these, a single abusive user or a runaway retry loop can dominate the bill.

On security, assume the model can be manipulated by any text it reads. Limit the tools it can call, validate arguments server-side, and never let it construct credentials or arbitrary queries. Apply tenant filters in code rather than asking the model to include them. Log tool calls and their outcomes so an unexpected action is visible.

Where this bites vibecoders
The gap between an AI demo and an AI product is mostly the code around the model call: timeout, retry, validation, budget, logging, and a fallback. Generated demos rarely include those because they do not affect whether the happy path works. Add them before the first real user, not after the first incident.

Where AI coding assistants get this wrong

Calling the provider with no timeout and letting the request hang.
Retrying non-idempotent actions and duplicating side effects.
Shipping without an evaluation set, then changing prompts by feel.
Letting the model call tools with the application’s full credentials.
Setting a global spend cap but no per-user limit.
Logging full prompts with customer data and no retention policy.

Checklist

Set timeouts, retry policy, and a circuit breaker.
Define and test a degraded mode.
Build an evaluation set before launch.
Validate structured output on the server.
Set per-task token and per-tenant spend budgets.
Restrict tools and data access, and log every tool call.

FAQ

How is this different from deploying a normal web app?

Three things differ: output is nondeterministic, every request has a variable external cost, and the input can contain instructions that influence behaviour. Those differences require evaluation, budgets, and injection defences that a normal CRUD app does not need.

Do I need a fallback provider?

Not always, but you need a fallback behaviour. If the single provider is down, the feature should degrade in a defined way rather than hang. A second provider is one option; a cached response, a simpler model, or a clear unavailable message can be enough for many features.

How large should the evaluation set be?

Start with twenty to fifty cases that cover the behaviours you would be embarrassed to get wrong. Coverage matters more than size. Grow it from real failures and support tickets, and keep it versioned so a change can be compared against the previous run.

What should I monitor after launch?

Error rate, timeout rate, p95 latency, validation failure rate, escalation rate, cost per successful task, and user feedback. Those together tell you whether the feature is working, affordable, and improving. A dashboard with only uptime cannot answer any of those questions.