How to Set a Token Budget Per Task

Quick answer
A token budget defines the maximum input, output, and retry cost for one task, enforced in code rather than observed after the invoice.
Set the limit from measured p95 usage per task, not from the model’s maximum context window.
Input budget is mostly context assembly: retrieved passages, history, and tool output. Output budget is a max_tokens setting plus a prompt that asks for the expected length.
Enforce hard stops per request, per user, and per tenant, and alert before the hard stop is reached.
Count retries and escalations inside the budget, because a failed cheap call that retries on an expensive model can double the cost of a successful task.

What is a token budget?

It is an explicit allocation of tokens for a task: how much context may be assembled, how long the answer may be, how many retries are allowed, and what happens when the limit is hit. The budget is enforced at the call site and aggregated per user or tenant, so a single request cannot consume an unbounded share of spend.

Without a budget, the effective limit is the model’s context window, which is usually much larger than the task needs. An accidental document dump or a runaway conversation then produces a request that is expensive, slow, and likely to fail — often all three at once.

How do you choose the numbers?

Measure first. Log input and output tokens for a representative sample of each task, then set the budget slightly above the p95. That leaves room for normal variation without financing the tail. For a new task, start with a deliberately small budget, inspect the truncations, and raise it only where quality suffers.

Break the input budget down by component: system prompt, retrieved context, conversation history, tool output, and the user request. The breakdown tells you where to cut. Usually retrieved context is the largest and the easiest to reduce with better retrieval rather than a larger window.

BUDGET = {"input": 6000, "output": 800, "retries": 1}

def build_context(question: str, docs: list[str]) -> str:
    # Reserve room for the question and the answer, then fill with docs.
    reserve = 500
    room = BUDGET["input"] - reserve
    parts, used = [], 0
    for d in docs:                      # already ranked by relevance
        t = count_tokens(d)
        if used + t > room:
            break
        parts.append(d)
        used += t
    return "\n\n".join(parts) + "\n\n" + question

def call_with_budget(fn, prompt: str) -> str:
    attempt = 0
    while True:
        try:
            return fn(prompt, max_output_tokens=BUDGET["output"])
        except RetryableError:
            if attempt >= BUDGET["retries"]:
                raise
            attempt += 1

Where should the budget be enforced?

Enforce it at the boundary where the request is assembled: truncate context before the call, set the output limit on the call, and cap retries in the wrapper. Then aggregate usage per user, tenant, and feature, and apply a second budget at that level. The per-request limit prevents outliers; the aggregate limit prevents a slow burn.

Add a soft threshold and a hard stop. At eighty percent of the aggregate budget, alert or degrade gracefully — shorter answers, a smaller model, cached responses. At one hundred percent, refuse politely rather than letting the spend continue. A budget that only produces a report has no effect on the invoice.

What should happen when the budget is hit?

Degrade, do not crash. A shorter context, a cheaper model, or a cached answer is usually better than an error. Tell the user what happened if the answer is materially reduced, and record the event so you can tell whether the budget is too tight or the task is genuinely too expensive.

The exception is a task where a truncated answer is dangerous, such as a medical or financial summary. There, refuse and explain rather than returning something incomplete. The budget still applies; the fallback behaviour is different because the cost of a wrong answer is higher than the cost of no answer.

Where this bites vibecoders
Generated code passes the whole conversation and every retrieved document into the prompt and sets no output limit, so a single long thread can cost orders of magnitude more than the feature it serves. Adding a context assembler with a token cap and a max output is the highest-leverage cost change in most AI apps.

Where AI coding assistants get this wrong

Setting the budget from the model’s context window instead of measured usage.
Passing full conversation history into every request.
Omitting max output tokens, then receiving a long answer nobody reads.
Counting only successful calls and ignoring retries and escalations.
Enforcing a per-request limit but no per-user or per-tenant limit.
Failing closed with a raw error when the budget is reached.

Checklist

Measure p95 input and output tokens per task.
Set the input budget slightly above p95.
Break the input budget down by component.
Set max output tokens on every call.
Cap retries and count their cost.
Add per-user and per-tenant aggregates with alerts and a soft degradation path.

FAQ

What is a reasonable output token limit?

The length the task actually needs, plus a margin. A classification needs tens of tokens; a summary needs a few hundred. If you do not know, measure a sample. A generous limit is not free, because models often generate up to the cap when the prompt does not specify a length.

Should the budget be per request or per user?

Both. A per-request budget prevents a single pathological call; a per-user or per-tenant budget prevents a slow accumulation that no single request would trigger. They catch different failures, and neither substitutes for the other.

How do I count tokens before sending the request?

Use the provider’s tokenizer library or a compatible counter, and add a margin for message framing and tool definitions. Approximate counts are fine for budgeting as long as they are consistently conservative. Do not estimate by character count alone, especially for non-English text and code.

What about cached tokens in the budget?

They still occupy context, even when they are billed at a discount. Budget them at full context size for truncation purposes, and account for the discount separately in the cost model. Mixing the two leads to context windows that look affordable but are too large to process.