# Enforce AGENTS.md Rules in CI: Name the Rule That Failed

- Author: Abdullah Chaudary
- Category: Engineering Leadership
- Published: 2026-11-13T17:37:01.695Z
- Updated: 2026-11-13T17:37:01.723Z
- Reading time: 8 min
- Tags: CI/CD, GitHub Actions, AI Coding Tools, Engineering Standards, Code Review

> A CI check can name the AGENTS.md rule it enforces. In a sandbox demo, a checker read rule IDs from the file, put the failed ID in a ::error title, and failed on drift.

On 13 November 2026, while building the demo for this post, I grepped this site's repo for one of its own rules. `CLAUDE.md` hard rule 9 says never use an em dash, and 8 lines of `.github/workflows/ci.yml` contain one: seven gate step names and a comment. None of the seven blocking gates in that file checks the rule on source, so nothing in CI noticed.

Enforcing AGENTS.md rules in CI has a gap: the rule sits in one file, the checks in another, and no failure says which rule it belongs to. My earlier post argued that [rules files are context, not enforcement](/blog/agents-md-rulebook-ai-coding-tools). Two follow-ups asked what a CI failure that names its rule looks like, and which rules CI can enforce at all. I built a sandbox to answer both: a Docker container with no network, a sample repository and six invented rules. Nothing ran on GitHub Actions.

**TL;DR:** To enforce AGENTS.md rules in CI so a failure names the rule, give each rule an ID, key the checks by ID, and print the ID, line and rule text on failure, with the ID in the `::error` title. In my sandbox, 4 of 6 rules got a check, drift failed the job, and 2 review-only rules passed while broken.

## Does a Failing CI Check Name the AGENTS.md Rule It Enforces?

Not by default. In this site's repo, the "no `any`" clause of rule 2 in `CLAUDE.md` is enforced by ESLint's `@typescript-eslint/no-explicit-any`, and a failure prints that ESLint ID, not "CLAUDE.md rule 2". Nothing in `ci.yml` refers to the rule numbers.

The route-metadata rule is checked by Playwright, and one assertion message in `tests/seo/schema.spec.ts` names a rule: "schema.md rule 5", from a different file. The em dash rule has one CI check, on `llms.txt` output only, with the rule reference in a code comment and no custom message. I did not run Playwright, so I do not know whether its output shows that comment.

The repo has no `AGENTS.md` at the root. Its rules live in `CLAUDE.md`, numbered 1 to 10 (the project page covers [this site's own `CLAUDE.md` rules and `.claude/` setup](/projects/abchaudary-me)).

## Why Does a Bare grep Step Pass When It Should Fail?

Because a `grep` in a shell step can exit 0 without having looked at anything. In the sandbox, three setups did it: a container with no UTF-8 locale, the `!` shorthand under `bash -e`, and a pipeline with no `pipefail`. I wrote a step to test the first two and hit the third by accident.

The sample `AGENTS.md` has six rules, each with an ID and a tag. The IDs are my convention, not part of the format:

```text
- **R1** [ci] No em dash (U+2014) in content or source. Use a colon, a comma or a full stop.
- **R2** [ci] Never commit secrets: no `.env` file and no access keys in tracked files.
- **R3** [lint] No `any` in TypeScript, explicit or implicit. Use `unknown` and narrow it.
- **R4** [ci] Every route (`src/app/**/page.tsx`) exports `metadata` or `generateMetadata`.
- **R5** [review] Prefer server components. Add "use client" only when a component needs state, effects or browser APIs.
- **R6** [review] Commit messages explain why the change was made, not only what changed.
```

My first version was one bare grep step for each of R1 to R4, such as `if grep -rnP '\x{2014}' content src; then exit 1; fi`. On the broken tree each step failed correctly, but the output was only a match (a bare `.env` for secrets, and nothing at all for the route-metadata loop). None says which rule failed.

**The locale.** With `LANG` and `LC_ALL` unset, `grep -P '\x{2014}'` cannot parse the code point, and inside an `if` that error counts as "no match":

```text
$ env -u LANG -u LC_ALL bash -e ci/before/01-no-em-dash.sh
grep: character code point value in \x{} or \o{} is too large
[exit 0]
```

The same grep outside the `if` exited 2. The `node:22-bookworm` image ships with no `LANG` or `LC_*` (checked with `env`). I did not check GitHub-hosted runners, so I claim this only for that container.

**The ****!**** shorthand.** Written as `! grep ...` and `! git ls-files | grep ...`, both matches printed and the step still exited 0:

```text
$ bash -e ci/before/05-bang-variant.sh
content/posts/hello.md:5:Shipping on a Friday is fine<U+2014>if the deploy is boring.
.env
content checks done
[exit 0]
```

The logs show `<U+2014>` because I piped the output through `sed`. The behaviour is documented: the GNU Bash manual says `-e` does not exit when "the command's return status is being inverted with !".

**No ****pipefail****.** My first sandbox run was my own mistake. `cp -a` kept the host file owner, so git failed with `fatal: detected dubious ownership in repository at '/work/repo'`, and step 02 (`git ls-files | grep`) exited 0 anyway, because the `if` sees only grep's "no match". I kept that log. GitHub's docs say a step with no `shell` runs `bash -e {0}`, with no `pipefail`, and an explicit `shell: bash` runs `-eo pipefail`. I ran every step through `bash -e` and did not test the `pipefail` variant.

So a check that cannot run has to fail the job. The checker exits 0 for a pass, 1 for rule violations, and 2 for drift or a check that could not run.

## Which AGENTS.md Rules Can CI Enforce, and Which Need Review?

A rule is CI-enforceable when a script can decide it without judgement. Of my six sample rules, three met that test (no em dash, no committed secrets, route metadata), one needed ESLint plus the TypeScript compiler, and two stayed review-only because every check passed code that broke them.

The dev.to post ["I stopped writing rules for coding agents that CI could not enforce"](https://dev.to/agentskit/i-stopped-writing-rules-for-coding-agents-that-ci-could-not-enforce-5hn9) draws a similar three-way split (its author maintains the tool it shows). My demo adds a rule ID and a failing fixture for each of the six rules:

- **R1, no em dash: CI, mechanical.** Found the injected character at `content/posts/hello.md:5:29`. It matches the character, not intent.
- **R2, no secrets: CI, mechanical, partial.** Caught a tracked `.env` and an AWS example key ID (`AKIAIOSFODNN7EXAMPLE`). It matches tracked env files and two key patterns only, so a secret in any other format passes.
- **R3, no ****any****: linter plus type checker.** ESLint flagged only the explicit `: any` in `parse.ts`. `tsc` strict flagged only the implicit one in `legacy.ts` (`TS7006`) and passed the explicit one.
- **R4, route metadata: CI, mechanical, partial.** Checks that the export exists on `src/app/about/page.tsx`, not that the title is any good.
- **R5, server components: review only.** `Badge.tsx` with `'use client'` and no state passed every check.
- **R6, commit messages say why: review only.** The messages "update stuff" and "fix" passed every check.

## How Do You Enforce AGENTS.md Rules in CI by Rule ID?

Keep the ID and the text in AGENTS.md only, and key the checks by ID in the checker. The checker parses each `- **R<n>** [tag] text` line, runs the check for that ID, and on failure prints the ID, the AGENTS.md line number and the rule text it just read from the file.

The checker is one Python 3.11 file, standard library only. Two lines carry the design:

```python
RULE_LINE = re.compile(r"^- \*\*(R\d+)\*\* \[(ci|lint|review)\] (.+)$")
CHECKS = {"R1": check_r1, "R2": check_r2, "R3": check_r3, "R4": check_r4}
```

R3 runs `eslint -f json` and `tsc --noEmit` and maps `@typescript-eslint/no-explicit-any` and the implicit-any codes TS7005, TS7006, TS7031 and TS7034 back to R3. Only TS7006 was exercised. On the broken tree, R1 failed like this:

```text
AGENTS.md R1 (line 11): "No em dash (U+2014) in content or source. Use a colon, a comma or a full stop."
  content/posts/hello.md:5:29: found U+2014 (em dash)
::error file=content/posts/hello.md,line=5,col=29,title=AGENTS.md R1::No em dash (U+2014) in content or source. Use a colon, a comma or a full stop. [AGENTS.md line 11; found U+2014 (em dash)]
```

The run ended `agents_rules: exit 1; violations: R1 x1, R2 x2, R3 x2, R4 x1; review-only (not checked): R5, R6`. To confirm the text comes from the file, I edited R1 in `AGENTS.md` only and reran with `ci/` unchanged. The first line became `AGENTS.md R1 (line 11): "No em dash (U+2014) anywhere. Owner rule since 2026-08-31."`.

## How Do You Put the Rule ID in a GitHub ::error Line?

Print a workflow command with the rule ID in `title=`. GitHub's `::error` takes `title`, `file`, `col`, `endColumn`, `line` and `endLine`, all optional, and has no rule field. So `title=AGENTS.md R1` carries the ID, and `file`, `line` and `col` point at the violation.

For R1 the checker prints `::error file=content/posts/hello.md,line=5,col=29,title=AGENTS.md R1::`, followed by the rule text and the AGENTS.md line in the message. Escaping is copied from `escapeData` and `escapeProperty` in `actions/toolkit`: `%`, `\r` and `\n` in the message, plus `:` and `,` in properties.

I printed these lines to a log only. I did not observe how the GitHub UI places the annotation or renders the title, and I did not check whether GitHub caps annotations per step. I claim only that the format matches GitHub's documentation.

## How Do You Catch Drift Between AGENTS.md and CI?

Fail the job in both directions. Deleting R1 from AGENTS.md while the checker still had a check for it exited 2. Adding an R7 tagged `[ci]` with no check exited 2 as well. So a `[ci]` rule cannot exist only on paper, and a check cannot outlive its rule.

```text
AGENTS.md drift: CI has a check for R1, but AGENTS.md has no rule R1. Restore the rule or delete the check.
AGENTS.md drift (line 17): R7 is tagged [ci] but CI has no check for it: "Every image in content has alt text."
```

With `LINT_BIN` unset, R3 printed `check could not run: LINT_BIN must point at a directory holding eslint and tsc`, emitted an `::error` titled `AGENTS.md R3`, and exited 2. That is the fail-closed behaviour the three grep failures called for.

### The checker broke its own rule

The second sandbox run failed in a way I had not planned. The first version of `ci/agents_rules.py` reported an em dash in itself:

```text
AGENTS.md R1 (line 11): "No em dash (U+2014) in content or source. Use a colon, a comma or a full stop."
  ci/agents_rules.py:82:12: found U+2014 (em dash)
```

I had meant the source to hold a Python escape for code point 2014. The file on disk held the literal character, because the escape was converted when the file was written (`od -c` showed bytes `342 200 224`). I fixed it with `EM_DASH = chr(0x2014)` and reran.

## What Does a Rule ID Checker Still Not Enforce?

A rule ID checker does not enforce rules that need judgement, and this demo covers nothing I did not run. On the fixed tree the run ended `agents_rules: exit 0; violations: none; review-only (not checked): R5, R6`, with `Badge.tsx` and the "fix" commit message still in place. I built no heuristic for R5 or R6, so I claim neither that one works nor that one fails.

The image was `node:22-bookworm` (pulled 13 November 2026) with GNU bash 5.2.15, GNU grep 3.8, Python 3.11.2, TypeScript 5.9.3 and ESLint v9.39.5. One sandbox, sample data. Not tested:

- GitHub Actions itself (`act` was not installed): the UI rendering of the annotation and title, and the runner's exit line.
- The locale of GitHub-hosted runners, and ugrep 7.8.4 (my host `grep`) against the locale case.
- A secret scanner such as gitleaks or trufflehog, ESLint plugins for R4 or R5, and a Claude Code `PreToolUse` hook.
- This repo's Playwright, Vitest, Lighthouse and size-limit gates, which I read but did not run.
- Nested `AGENTS.md` files, and how any agent reacts to rule IDs.
- Timings. I measured none.

The [agents.md](https://agents.md/) FAQ says there are no required fields, so a checker built on `- **R1** [ci]` lines depends on my format, not the spec.

## References

- [agents.md](https://agents.md/): the format; no required fields.
- [GitHub Docs, Workflow commands](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-commands): `::error` syntax and parameters.
- [GitHub Docs, Workflow syntax](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax): `bash -e {0}` versus `-eo pipefail`.
- [actions/toolkit, command.ts](https://github.com/actions/toolkit/blob/main/packages/core/src/command.ts): `escapeData`, `escapeProperty`.
- [GNU Bash manual, The Set Builtin](https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html): when `-e` does not exit.
- [dev.to, "I stopped writing rules for coding agents that CI could not enforce"](https://dev.to/agentskit/i-stopped-writing-rules-for-coding-agents-that-ci-could-not-enforce-5hn9): the three-way split.

A green run proves only the rules someone wrote a check for, and mine went green with `Badge.tsx` still in the tree. Which of your AGENTS.md rules have you tagged review-only, and has one of them ever moved to a CI check?