How to Write a Runbook for an AI Agent

Quick answer
Write for the person on call, not for the person who built the agent.
Cover five actions: stop new runs, cancel in-flight runs, revoke credentials, drain the queue, and resume safely.
Include the exact commands, the expected output, and the one signal that confirms each step worked.
Name the failure modes: stuck run, runaway cost, unexpected tool, bad output that passed checks.
Test the runbook on staging, and date it. An untested runbook is a wish list.

What does an agent runbook need to contain?

Identity and ownership first: which agent, which service, who owns it, and where the dashboard is. Then the controls, in the order you would use them during an incident: stop, cancel, revoke, drain, verify, resume.

For each control, the exact command and the output that means it worked. “Disable the agent” is not a step; a command that flips a flag and a query that confirms no runs are active is.

## Stop and contai
1. Stop new runs: `agentctl pause-queue --agent $AGENT`
   Confirm: `agentctl queue --agent $AGENT` shows 0 running.
2. Cancel in-flight: `agentctl cancel --agent $AGENT --all`
   Confirm: every run status is `cancelled` or `completed`.
3. Revoke identity: `agentctl revoke --identity $AGENT`
   Confirm: a test call returns 401.
4. Preserve: copy the audit log to the incident bucket.

Which failure modes should it cover?

A run stuck beyond its time budget, a cost spike, an unexpected tool call or destination, a credential error loop, and the hardest one: a run that completed successfully with a wrong result. Each needs a first action and a way to scope the impact.

Include the “is it still happening?” check for each. Most incidents are discovered after the fact, and the first question is whether the behaviour is ongoing. A runbook that only describes recovery leaves the responder guessing about current risk.

How do you keep it usable?

Keep it short and put the commands in copyable blocks. Link to the dashboard and the audit query rather than describing them. Put the escalation path at the top, because the person reading may not know who owns the agent.

Date it and record when it was last tested. Commands drift, flags get renamed, and a runbook that has not been run since the last release is a source of confidently wrong instructions.

How do you test it?

Run it on staging during a calm week: pause the queue, cancel a run, revoke a test identity, and resume. Time it. The gaps you find are usually permissions the responder does not have and commands that assume the builder’s environment.

Then re-test after any change to the runtime, the queue, or the identity system. The runbook is part of the deployment, not documentation written once and abandoned.

Where this bites vibecoders
The runbook is usually missing entirely, and the recovery plan is “ask the person who built it”. That works until they are asleep. Write the five commands down and test them once; it is an afternoon that pays back on the first bad run.

Where AI coding assistants get this wrong

Writing steps at a level of abstraction that requires the builder’s knowledge.
Omitting the command that confirms each step worked.
Forgetting to revoke the identity after stopping the run.
Not covering the “completed but wrong” failure mode.
Leaving the runbook undated and untested.
Assuming the responder has production permissions already.

Checklist

Name the owner, dashboard, and escalation path.
Give exact commands for stop, cancel, revoke, drain, resume.
State the confirmation signal for each step.
Cover stuck, expensive, unexpected, and wrong-output runs.
Date it and record the last test.
Re-test after runtime changes.

FAQ

Is this different from a normal service runbook?

The controls are similar, but the failure modes are different: a run can be expensive, stuck, or successful and wrong. The runbook also has to cover credential revocation and queue draining, because an agent holds authority that a stateless service usually does not.

Who should own it?

The team that operates the agent, with the builder as a reviewer. Ownership means updating it after changes and testing it, not just writing it. A runbook with no owner is stale within a release cycle.

What belongs in the post-incident update?

The step that was missing or wrong, and the change that fixes it. If the responder had to read code to find the queue name, the runbook needs the queue name. Every incident is a test of the document.

Do I need one for a personal agent?

A short version, yes: how to stop it, how to revoke its token, and where the logs are. The scale is smaller but the failure mode is the same, and future you will not remember which flag pauses the queue.