Enforce AGENTS.md Rules in CI: Name the Rule That Failed

On 13 November 2026, while building the demo for this post, I grepped this site's repo for one of its own rules. CLAUDE.md hard rule 9 says never use an em dash, and 8 lines of .github/workflows/ci.yml contain one: seven gate step names and a comment. None of the seven blocking gates in that file checks the rule on source, so nothing in CI noticed.

Enforcing AGENTS.md rules in CI has a gap: the rule sits in one file, the checks in another, and no failure says which rule it belongs to. My earlier post argued that rules files are context, not enforcement. Two follow-ups asked what a CI failure that names its rule looks like, and which rules CI can enforce at all. I built a sandbox to answer both: a Docker container with no network, a sample repository and six invented rules. Nothing ran on GitHub Actions.

TL;DR: To enforce AGENTS.md rules in CI so a failure names the rule, give each rule an ID, key the checks by ID, and print the ID, line and rule text on failure, with the ID in the ::error title. In my sandbox, 4 of 6 rules got a check, drift failed the job, and 2 review-only rules passed while broken.

Does a Failing CI Check Name the AGENTS.md Rule It Enforces?

Not by default. In this site's repo, the "no any" clause of rule 2 in CLAUDE.md is enforced by ESLint's @typescript-eslint/no-explicit-any, and a failure prints that ESLint ID, not "CLAUDE.md rule 2". Nothing in ci.yml refers to the rule numbers.

The route-metadata rule is checked by Playwright, and one assertion message in tests/seo/schema.spec.ts names a rule: "schema.md rule 5", from a different file. The em dash rule has one CI check, on llms.txt output only, with the rule reference in a code comment and no custom message. I did not run Playwright, so I do not know whether its output shows that comment.

The repo has no AGENTS.md at the root. Its rules live in CLAUDE.md, numbered 1 to 10 (the project page covers this site's own CLAUDE.md rules and .claude/ setup).

Why Does a Bare grep Step Pass When It Should Fail?

Because a grep in a shell step can exit 0 without having looked at anything. In the sandbox, three setups did it: a container with no UTF-8 locale, the ! shorthand under bash -e, and a pipeline with no pipefail. I wrote a step to test the first two and hit the third by accident.

The sample AGENTS.md has six rules, each with an ID and a tag. The IDs are my convention, not part of the format:

- **R1** [ci] No em dash (U+2014) in content or source. Use a colon, a comma or a full stop.
- **R2** [ci] Never commit secrets: no `.env` file and no access keys in tracked files.
- **R3** [lint] No `any` in TypeScript, explicit or implicit. Use `unknown` and narrow it.
- **R4** [ci] Every route (`src/app/**/page.tsx`) exports `metadata` or `generateMetadata`.
- **R5** [review] Prefer server components. Add "use client" only when a component needs state, effects or browser APIs.
- **R6** [review] Commit messages explain why the change was made, not only what changed.

My first version was one bare grep step for each of R1 to R4, such as if grep -rnP '\x{2014}' content src; then exit 1; fi. On the broken tree each step failed correctly, but the output was only a match (a bare .env for secrets, and nothing at all for the route-metadata loop). None says which rule failed.

The locale. With LANG and LC_ALL unset, grep -P '\x{2014}' cannot parse the code point, and inside an if that error counts as "no match":

$ env -u LANG -u LC_ALL bash -e ci/before/01-no-em-dash.sh
grep: character code point value in \x{} or \o{} is too large
[exit 0]

The same grep outside the if exited 2. The node:22-bookworm image ships with no LANG or LC_* (checked with env). I did not check GitHub-hosted runners, so I claim this only for that container.

The ! shorthand. Written as ! grep ... and ! git ls-files | grep ..., both matches printed and the step still exited 0:

$ bash -e ci/before/05-bang-variant.sh
content/posts/hello.md:5:Shipping on a Friday is fine<U+2014>if the deploy is boring.
.env
content checks done
[exit 0]

The logs show <U+2014> because I piped the output through sed. The behaviour is documented: the GNU Bash manual says -e does not exit when "the command's return status is being inverted with !".

No pipefail. My first sandbox run was my own mistake. cp -a kept the host file owner, so git failed with fatal: detected dubious ownership in repository at '/work/repo', and step 02 (git ls-files | grep) exited 0 anyway, because the if sees only grep's "no match". I kept that log. GitHub's docs say a step with no shell runs bash -e {0}, with no pipefail, and an explicit shell: bash runs -eo pipefail. I ran every step through bash -e and did not test the pipefail variant.

So a check that cannot run has to fail the job. The checker exits 0 for a pass, 1 for rule violations, and 2 for drift or a check that could not run.

Which AGENTS.md Rules Can CI Enforce, and Which Need Review?

A rule is CI-enforceable when a script can decide it without judgement. Of my six sample rules, three met that test (no em dash, no committed secrets, route metadata), one needed ESLint plus the TypeScript compiler, and two stayed review-only because every check passed code that broke them.

The dev.to post "I stopped writing rules for coding agents that CI could not enforce" draws a similar three-way split (its author maintains the tool it shows). My demo adds a rule ID and a failing fixture for each of the six rules:

• R1, no em dash: CI, mechanical. Found the injected character at content/posts/hello.md:5:29. It matches the character, not intent.
• R2, no secrets: CI, mechanical, partial. Caught a tracked .env and an AWS example key ID (AKIAIOSFODNN7EXAMPLE). It matches tracked env files and two key patterns only, so a secret in any other format passes.
• R3, no any: linter plus type checker. ESLint flagged only the explicit : any in parse.ts. tsc strict flagged only the implicit one in legacy.ts (TS7006) and passed the explicit one.
• R4, route metadata: CI, mechanical, partial. Checks that the export exists on src/app/about/page.tsx, not that the title is any good.
• R5, server components: review only. Badge.tsx with 'use client' and no state passed every check.
• R6, commit messages say why: review only. The messages "update stuff" and "fix" passed every check.

How Do You Enforce AGENTS.md Rules in CI by Rule ID?

Keep the ID and the text in AGENTS.md only, and key the checks by ID in the checker. The checker parses each - **R<n>** [tag] text line, runs the check for that ID, and on failure prints the ID, the AGENTS.md line number and the rule text it just read from the file.

The checker is one Python 3.11 file, standard library only. Two lines carry the design:

RULE_LINE = re.compile(r"^- \*\*(R\d+)\*\* \[(ci|lint|review)\] (.+)$")
CHECKS = {"R1": check_r1, "R2": check_r2, "R3": check_r3, "R4": check_r4}

R3 runs eslint -f json and tsc --noEmit and maps @typescript-eslint/no-explicit-any and the implicit-any codes TS7005, TS7006, TS7031 and TS7034 back to R3. Only TS7006 was exercised. On the broken tree, R1 failed like this:

AGENTS.md R1 (line 11): "No em dash (U+2014) in content or source. Use a colon, a comma or a full stop."
  content/posts/hello.md:5:29: found U+2014 (em dash)
::error file=content/posts/hello.md,line=5,col=29,title=AGENTS.md R1::No em dash (U+2014) in content or source. Use a colon, a comma or a full stop. [AGENTS.md line 11; found U+2014 (em dash)]

The run ended agents_rules: exit 1; violations: R1 x1, R2 x2, R3 x2, R4 x1; review-only (not checked): R5, R6. To confirm the text comes from the file, I edited R1 in AGENTS.md only and reran with ci/ unchanged. The first line became AGENTS.md R1 (line 11): "No em dash (U+2014) anywhere. Owner rule since 2026-08-31.".

How Do You Put the Rule ID in a GitHub ::error Line?

Print a workflow command with the rule ID in title=. GitHub's ::error takes title, file, col, endColumn, line and endLine, all optional, and has no rule field. So title=AGENTS.md R1 carries the ID, and file, line and col point at the violation.

For R1 the checker prints ::error file=content/posts/hello.md,line=5,col=29,title=AGENTS.md R1::, followed by the rule text and the AGENTS.md line in the message. Escaping is copied from escapeData and escapeProperty in actions/toolkit: %, \r and \n in the message, plus : and , in properties.

I printed these lines to a log only. I did not observe how the GitHub UI places the annotation or renders the title, and I did not check whether GitHub caps annotations per step. I claim only that the format matches GitHub's documentation.

How Do You Catch Drift Between AGENTS.md and CI?

Fail the job in both directions. Deleting R1 from AGENTS.md while the checker still had a check for it exited 2. Adding an R7 tagged [ci] with no check exited 2 as well. So a [ci] rule cannot exist only on paper, and a check cannot outlive its rule.

AGENTS.md drift: CI has a check for R1, but AGENTS.md has no rule R1. Restore the rule or delete the check.
AGENTS.md drift (line 17): R7 is tagged [ci] but CI has no check for it: "Every image in content has alt text."

With LINT_BIN unset, R3 printed check could not run: LINT_BIN must point at a directory holding eslint and tsc, emitted an ::error titled AGENTS.md R3, and exited 2. That is the fail-closed behaviour the three grep failures called for.

The checker broke its own rule

The second sandbox run failed in a way I had not planned. The first version of ci/agents_rules.py reported an em dash in itself:

AGENTS.md R1 (line 11): "No em dash (U+2014) in content or source. Use a colon, a comma or a full stop."
  ci/agents_rules.py:82:12: found U+2014 (em dash)

I had meant the source to hold a Python escape for code point 2014. The file on disk held the literal character, because the escape was converted when the file was written (od -c showed bytes 342 200 224). I fixed it with EM_DASH = chr(0x2014) and reran.

What Does a Rule ID Checker Still Not Enforce?

A rule ID checker does not enforce rules that need judgement, and this demo covers nothing I did not run. On the fixed tree the run ended agents_rules: exit 0; violations: none; review-only (not checked): R5, R6, with Badge.tsx and the "fix" commit message still in place. I built no heuristic for R5 or R6, so I claim neither that one works nor that one fails.

The image was node:22-bookworm (pulled 13 November 2026) with GNU bash 5.2.15, GNU grep 3.8, Python 3.11.2, TypeScript 5.9.3 and ESLint v9.39.5. One sandbox, sample data. Not tested:

• GitHub Actions itself (act was not installed): the UI rendering of the annotation and title, and the runner's exit line.
• The locale of GitHub-hosted runners, and ugrep 7.8.4 (my host grep) against the locale case.
• A secret scanner such as gitleaks or trufflehog, ESLint plugins for R4 or R5, and a Claude Code PreToolUse hook.
• This repo's Playwright, Vitest, Lighthouse and size-limit gates, which I read but did not run.
• Nested AGENTS.md files, and how any agent reacts to rule IDs.
• Timings. I measured none.

The agents.md FAQ says there are no required fields, so a checker built on - **R1** [ci] lines depends on my format, not the spec.

References

• agents.md: the format; no required fields.
• GitHub Docs, Workflow commands: ::error syntax and parameters.
• GitHub Docs, Workflow syntax: bash -e {0} versus -eo pipefail.
• actions/toolkit, command.ts: escapeData, escapeProperty.
• GNU Bash manual, The Set Builtin: when -e does not exit.
• dev.to, "I stopped writing rules for coding agents that CI could not enforce": the three-way split.

A green run proves only the rules someone wrote a check for, and mine went green with Badge.tsx still in the tree. Which of your AGENTS.md rules have you tagged review-only, and has one of them ever moved to a CI check?