How to Run a Coding Agent in CI

Quick answer
Start read-only: the agent reads the diff, runs tests, and posts a comment. It does not push.
Use a dedicated identity with the narrowest permissions the job needs, and no repository write access.
Bound the run with a timeout and a token budget so a loop cannot consume the pipeline for an hour.
For fork pull requests, treat the code as untrusted: no secrets, no write token, no network egress beyond the registry.
Graduate to writing only after the read-only version is reliable and reviewed.

What should the first CI agent do?

Read the change, run the test suite, and comment with findings. That is enough to be useful and small enough to be safe. It exercises the same context the agent will need later, without giving it the ability to change the repository.

The review case is the best first step because the output is text. A wrong finding wastes a reviewer’s minute; a wrong push costs a branch. Start where the failure mode is cheap.

# .github/workflows/agent-review.yml
name: agent-review
on: pull_request
permissions:
  contents: read
  pull-requests: write
concurrency:
  group: agent-review-${{ github.event.pull_request.number }}
  cancel-in-progress: true
jobs:
  review:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: npm ci && npm test
        continue-on-error: true
      - name: Agent review
        env:
          AGENT_TOKEN: ${{ secrets.AGENT_READONLY_TOKEN }}
        run: agent review --diff origin/${{ github.base_ref }}...HEAD --max-tokens 120000

Which identity should it use?

A dedicated service identity, not a maintainer’s personal token. Give it read access to contents, write access to pull request comments, and nothing else. If the agent needs to read the issue tracker, add that scope explicitly and record why.

Rotate the credential on a schedule and on any suspicious run. A CI token that lives for years is a standing invitation: it is present in every job and reachable by anything that can influence a step.

How do you bound the run?

A job timeout, a token budget, and a concurrency rule that cancels superseded runs on the same pull request. Without the third one, a busy pull request queues several agents that each work on an older commit and comment on stale code.

Also cap what the agent can read. A job that checks out the repository and runs the suite needs no additional data. If it needs documentation or a ticket, pass a specific file rather than granting a broad integration.

When should it be allowed to write?

After the read-only version has run for a while with findings you trust, and with a separate workflow that can only push to a branch it owns. Keep it out of the default branch, require a pull request, and keep a human approval on the merge.

The writing version needs more controls, not fewer: a branch naming convention, a size limit on the diff, and a rollback that is one action. If the agent can push, it can also be tricked into pushing.

Where this bites vibecoders
The tempting workflow gives the agent a personal token, repository write access, and every secret in the organisation, because that is what makes the demo work. Start with a read-only job that comments, and add capability one reviewed step at a time.

Where AI coding assistants get this wrong

Using a personal access token in a CI job.
Granting contents write before the read-only version is trusted.
Running on fork pull requests with secrets available.
Skipping the job timeout, so a loop burns the runner.
Letting several runs comment on stale commits.
Passing the whole environment to the agent step.

Checklist

Start with a read-only review job.
Use a dedicated identity with minimal scopes.
Set a job timeout and a token budget.
Cancel superseded runs per pull request.
Deny secrets to fork pull requests.
Add write capability only with a branch and a review gate.

FAQ

Do I need a self-hosted runner?

Not necessarily. A hosted runner is fine for a read-only review job. Self-hosting becomes relevant when the agent needs access to internal services, and then the runner’s network position is part of the security model.

How much does a CI review cost?

It depends on the diff size and the model. A review that reads a large diff on every push can cost more than the review saves, so cap the diff size, run on ready-for-review rather than every push, and measure cost per pull request.

What if the agent’s comment is wrong?

Treat it like any review comment: reply, resolve, and correct the instruction if the pattern repeats. A bot whose comments are routinely dismissed should be retuned or removed, because noise trains the team to ignore the channel.

Can the agent run the test suite in CI?

Yes, and it should, because the output is the evidence for its claims. Keep the suite in a normal step, pass the result to the agent, and require the agent to quote the outcome rather than assert it.