What Is an Agent Runtime?

Quick answer
The runtime is everything around the model: the loop, the tool dispatcher, the sandbox, the state store, and the audit log.
It decides what the agent can do, which is why it is the right place to enforce security rather than the prompt.
A harness in an editor is a runtime with a local sandbox; a production agent is a runtime with a queue and durable state.
Model choice is a runtime configuration; permissions, retries, and budgets are runtime behaviour.
If you cannot describe your runtime, you do not yet have a production agent — you have a script that calls an API.

What does an agent runtime do?

It runs the loop: send context to the model, receive a tool call or an answer, execute the tool under policy, append the result, and repeat until the task ends or a limit is hit. Around that loop it manages state, isolation, credentials, and records.

Tool policy is the part that matters most for security. The runtime knows which tools exist, what arguments are allowed, and which destinations are reachable, so it can refuse an action regardless of what the model asked for.

How is it different from the model API?

The model API is stateless inference. The runtime supplies the state: the conversation so far, the tool results, the plan, the budget, and the identity. Two teams using the same model can have completely different behaviour because their runtimes differ.

That is why “the model refused” and “the agent did it” are not the same statement. In most incidents the model produced a plausible action and the runtime executed it, because the runtime had no policy that said otherwise.

What belongs in the runtime, and what does not?

In: the loop, tool dispatch and policy, sandboxing, credential handling, checkpoints, budgets, approvals, and audit. Out: business logic that can live in a normal service, and data access that can be a scoped tool.

A common mistake is putting security in the system prompt and business rules in the agent. Both belong in code. The model should choose among permitted actions, not define what is permitted.

loop:
  context = load(run)            # task, plan, last observations
  step    = model.next(context)
  if step.tool:
    policy.allow(step.tool, step.args) or refuse
    result = sandbox.run(step.tool, step.args, identity)
    audit.write(run, step, result)
    budget.charge(step.usage)
  checkpoint(run, step)
until done or budget exhausted or cancelled

How do you choose one?

Look for the controls you cannot build cheaply: per-tool policy, sandboxing, durable checkpoints, cancel, and an audit sink. A runtime without those is a library, and you will end up writing them yourself.

Also check the escape hatches. You will need to add a tool, change a policy, and inspect a run. A runtime that only supports its own integrations becomes a constraint on the system rather than a layer in it.

Where this bites vibecoders
An agent built as a prompt plus a loop plus direct API calls has no policy layer, so every guardrail lives in the prompt. That works until the model is convinced. The runtime is where a guardrail stops being a suggestion.

Where AI coding assistants get this wrong

Putting permissions in the system prompt instead of the runtime.
Treating the model API as the agent.
Running the loop in a request handler with no durable state.
Letting the runtime hold broad credentials for convenience.
Skipping the audit sink because the harness has logs.
Choosing a runtime that cannot restrict tools per task.

Checklist

Identify the loop, tool policy, sandbox, state, and audit.
Confirm policy is enforced outside the prompt.
Check that state survives a restart.
Verify cancel and budget enforcement.
Confirm the audit sink is external.
Test adding a tool without changing the loop.

FAQ

Is a coding assistant a runtime?

Yes, a local one: it runs the loop, executes tools in your workspace, and applies whatever policy it has. The difference from a production runtime is the sandbox, durability, and audit, not the concept.

Does the runtime own the model choice?

It can, and often should. Routing different steps to different models is a runtime decision, as is capping token use. The model is a dependency the runtime selects, not the system itself.

Can I build one myself?

Yes, and for a narrow use case it can be small: a queue, a loop, a tool dispatcher with an allowlist, a state table, and a log. The parts that take time are the controls, not the loop, so budget accordingly.

What is the failure mode of a weak runtime?

Every incident becomes a prompt change. Without policy, sandbox, and audit, the only lever is asking the model to behave, and the only evidence is the model’s own account of what happened.