How to Measure Token ROI

Quick answer
Token ROI is the value a task produces divided by the total token cost of producing it, including retries, failures, and cached-input pricing.
The denominator is cost per successful task, not cost per API call; measuring per call rewards retries and hides failures.
The numerator is a business outcome — a resolved ticket, an accepted draft, a completed checkout — not the number of generations.
Compare against a baseline: the cost of the previous workflow, whether a human or a cheaper model. ROI without a baseline is just a number.
Review ROI per feature and per user cohort, because an average across the whole product hides the features that lose money.

What is token ROI?

It is value per unit of token spend: the outcome a feature produces divided by what it costs in tokens to produce one successful outcome. For a support assistant, the outcome might be a resolved ticket; for a drafting tool, an accepted document; for a coding assistant, a merged change that passes review.

The formulation forces two questions that a cost dashboard does not: what counts as success, and what does success cost in total. A feature can look cheap per call and expensive per outcome if it retries often, needs human correction, or produces answers users discard.

How do you calculate cost per successful task?

Sum every token billed for the task: the initial call, cached input at its discounted price, output, retries, escalation to a stronger model, and any follow-up call needed to format or validate the result. Then divide by the number of tasks that reached the success condition. Failures still cost tokens and belong in the numerator of the cost side.

Separate one-time costs such as evaluation runs and prompt development from steady-state cost. They matter for the decision to build the feature, but they distort the operating number. Report both: the cost to reach the first successful version, and the ongoing cost per task.

cost_per_success = (input_cost + output_cost + retry_cost + escalation_cost)
                   / successful_tasks

Example week:
  successful tasks        1,000
  input tokens            4.2M  @ 3.00 / 1M = 12.60
  output tokens           0.6M  @ 15.00 / 1M = 9.00
  retry + escalation      4.10
  total token cost        25.70
  cost per success        0.0257

Compare with the human baseline for the same 1,000 tasks.

What value goes on the other side?

Use the outcome the business already measures: minutes saved, tickets resolved without escalation, documents accepted without edits, conversion rate, or revenue per cohort. Convert to money if you can, and keep the raw unit if you cannot. A ratio in consistent units is enough to compare features and changes.

Include human time honestly. A feature that produces a draft in ten seconds but requires fifteen minutes of correction may have negative ROI against the manual process. Time saved has to be measured after review, not before, or the calculation flatters the automation.

How do you use the number?

Use it to choose where to optimize: a feature with a high cost per success and a large share of traffic is the best target for caching, routing, or a tighter budget. Use it to decide whether a feature should exist at all, and to detect regressions when a model or prompt changes.

Do not use it as a leaderboard for teams. Cost per success depends on task difficulty and user mix, so comparing a support bot to a code assistant says nothing useful. Compare each feature to its own baseline and to its own previous week, with the same definitions.

Where this bites vibecoders
AI features are usually instrumented for latency and error rate, not for cost per successful outcome. Adding three fields to the existing request log — feature, task id, and success — turns the invoice into a per-feature number, which is the difference between knowing the bill and knowing which feature caused it.

Where AI coding assistants get this wrong

Dividing total spend by API calls instead of successful tasks.
Ignoring retries and escalation calls, which can double the real cost.
Counting generated drafts as value before anyone accepts them.
Omitting review and correction time from the human baseline.
Reporting one blended number across features with different economics.
Comparing token prices across providers without comparing output quality.

Checklist

Define the success condition for each AI feature.
Log tokens, retries, escalations, and success per task.
Compute cost per successful task weekly.
Measure the human baseline after review time.
Report ROI per feature and cohort, not as one blended number.
Re-run the comparison after every model or prompt change.

FAQ

Is token ROI the same as cost per request?

No. Cost per request ignores whether the request succeeded and how many attempts it took. ROI uses cost per successful outcome, so a feature with cheap calls and frequent retries can have worse ROI than one with expensive calls that succeed first time.

How do I value time saved?

Use the loaded cost of the person whose time is saved, applied to the time actually saved after review. If the review takes as long as the manual task, the value is zero regardless of how fast the generation was. Measure the end-to-end workflow, not the generation step.

How often should ROI be recalculated?

Weekly at first, then monthly once the number is stable. Recalculate immediately after a model change, a prompt change, a pricing change, or a shift in the user mix. Those are the events that move the ratio, and a stale ROI is misleading in exactly the moment it is consulted.

What if the feature has no direct revenue?

Use a proxy the organisation already trusts: tickets resolved, time to resolution, documents produced, errors prevented. The proxy does not need to be perfect, but it must be measured the same way before and after. A consistent proxy beats an inconsistent dollar figure.