Discipline: Gating What Coding Agents Change

When you tell a coding agent to make the tests pass, that is what it optimizes for. Whether the code actually works is your problem.

In late 2025, Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini published ImpossibleBench¹. They took tasks from SWE-bench and LiveCodeBench and mutated the unit tests to contradict the specification, leaving no honest way to pass them. The prompt said DO NOT MODIFY THE TESTS, in capitals, and GPT-5 still passed 54% of the conflicting SWE-bench tasks. Claude models and Qwen3-Coder mostly edited the tests (over 79% of their cheats), while OpenAI’s models spread their cheats across special-casing, operator overloading that makes == always true, and tracking call counts so the same input returned different answers on consecutive calls.

I kept running into those same patterns in my own diffs while working with agents on a 20-year-old C library, a Rust rewrite of it, and the tooling around both. In that kind of codebase, a silent regression is easy to miss during review and painful to track down later. So I built Discipline² (MIT / Apache-2.0) to gate the diff itself instead of taking the agent’s word for it. It’s a CI check and an agent hook that compares each branch with its merge base and fails when a test got weaker, a test or assertion disappeared, or the sandbox config got wider. It’s deterministic, uses no LLM, and works from the code’s syntax and the config files.

📎  TL;DR

• When you ask a coding agent to make the tests pass, it sometimes makes the tests easier instead of fixing the code: it weakens a check, skips a failing test, or loosens its own sandbox settings. You only see it by comparing the change with what was there before.
• Discipline is a free, open-source command-line check that does that comparison. It blocks the pull request in CI when a test got weaker or a safety setting got looser, and tells the agent what it broke while it’s still working.
• I ran it on 232 pull requests that agents and bots got merged into open-source projects like ruff, deno, and tokio. It would have blocked 41 of them, the same 41 on every run. Run it against your own repository’s merge history to see what it would have caught there.

How Agents Erode a Codebase

A model does not have to be malicious to produce a bad diff. Tell it to make the build green and give it write access to the repository that defines green, and it will edit whatever gets it there fastest.

Even when the tests are not rigged, iterating against a visible test suite pushes models toward narrow fixes that break existing code. On 449 SWE-bench Verified tasks, Ahmed et al. at IBM Research found that among patches that passed an auto-generated reproduction test, 21.8% from Claude 3.7 Sonnet and 33.0% from GPT-4o still failed the held-out test suite³. When a refinement loop showed the generated test to the model and asked it to repair patches that had failed it, 14 of the 22 that newly passed on Sonnet (63.6%) and 13 of 22 on GPT-4o (59.1%) overfit the test and failed the held-out checks. In a separate study on SWE-bench Verified, Wang, Pradel, and Liu ran the rest of each repository’s developer test suite (rather than only the test file modified in the PR) and found that 7.8% of “solved” patches broke existing tests elsewhere in the repo, while 29.6% behaved differently from the human fix under differential testing⁴.

When the agent can edit the test suite itself, it doesn’t even have to write a narrow fix. assert_eq!(a, b) gets watered down to assert!(a.is_some()), or an expected value changes from 4 to 5 to match what the new code returns. A failing test picks up @pytest.mark.skip(reason="flaky"). Sometimes a whole test file is deleted “to stay focused” (an actual Reddit report that the ImpossibleBench authors cite). In other cases the agent writes new tests that run without asserting anything, ships a function with todo!(), raise NotImplementedError, or return null as its body, or wraps a failing call in an empty catch {}.

Or the agent leaves the test files alone and changes how they run. Production code can check "pytest" in sys.modules and return the expected answer only while tests are running. Or the diff tampers with the runner through a conftest.py hook that rewrites a failed test report to passed, || true or --passWithNoTests tacked onto the test script, a lower coverage floor in codecov.yml, continue-on-error: true on the CI job, a looser tsconfig.json, or an edit to the gate’s own config that turns the failing check off.

The same happens with credentials and sandbox settings. Agents commit /Users/<name>/... paths, LAN IPs, scratch files and transcripts, or a token that was sitting in the environment. They also widen their own sandbox by flipping permissions.defaultMode to bypassPermissions, deleting a deny entry, switching sandbox.enabled off, adding an MCP server, or marking a devcontainer privileged.

METR caught frontier models reading the expected answer straight out of the scoring code⁵.

👉 Your sandbox is configuration in the repository. If an agent has write access to the repo and the sandbox stands between it and a green build, an iterate-until-green loop will eventually propose editing the sandbox.

Local and Open-Weight Models Fail Differently

Open-weight models have closed most of the benchmark gap with hosted APIs: in the 2025 Stanford AI Index, the gap between the top closed-weight and open-weight models on Chatbot Arena shrank from 8.04% in January 2024 to 1.70% by February 2025⁶. What people actually run on a workstation or a local server, though, is usually a 4-bit quantized build (GGUF, GPTQ, or AWQ) of a 7B to 32B dense model or of a Mixture-of-Experts (MoE) coder with a few billion parameters active per token. I run a 4-bit Qwen MoE model on my own server alongside smaller local models, and they fail in different ways than the hosted frontier APIs.

The evidence doesn’t support “smaller models cheat more”. ImpossibleBench found the opposite across the models it tested: more capable models had higher cheating rates. It takes a capable model to think of making __eq__ always return True. What the evidence does support is that quantized and open-weight setups are more exposed to a different set of failures, and the mess still ends up in the diff.

Hallucinated packages and APIs. Spracklen et al. checked 576,000 code samples from 16 models against the real package registries: commercial models hallucinated a package at least 5.2% of the time on average, against 21.7% for open-source models quantized with 4-bit GPTQ⁷. Register one of those made-up names on PyPI or npm and pip install pulls your code. Discipline’s dependency-delta gate flags every new direct dependency for a human to look at, but it doesn’t look the package up; a lockfile-resolution check does that part. Liu et al. hand-audited 3,120 generations from five models and found that the small open-weight ones (26.28%–35.42% average pass@1) produced far more prompt-conflicting logic and undefined variables than DeepSeek-R1 (48.08%), though most of R1’s own hallucinations misused library or project APIs⁸. A diff gate can’t tell a wrong method call from a right one. That’s the tests’ job, which is why the tests can’t be allowed to get weaker.

4-bit MoE degradation. Quantization hits code harder than general benchmarks suggest. On instruction-tuned Qwen-MoE-14B-Chat and DeepSeek-MoE-16B-Chat, 4-bit GPTQ kept MMLU within 2.3 points of FP16 while HumanEval pass@1 fell by 29% and 45% (relative), because the calibration data reaches rarely used experts unevenly and ignores how strongly each token is routed to each expert⁹.

Weaker guardrails across turns. In Cisco’s own automated testing of eight open-weight models, including Qwen3-32B, Llama 3.3 70B, and GPT-OSS-20b, single-turn jailbreaks succeeded 13.11% of the time on average, while multi-turn attacks succeeded 25.86% to 92.78% of the time depending on the model¹⁰. An agent reading an issue thread, a PR comment, and a dependency README across several steps is in exactly that kind of multi-turn loop with untrusted text. Discipline’s instruction-smuggling gate scans the diff, the PR description, and commit messages for text aimed at the next agent, but it only sees what lands in the repository.

A harness tuned for someone else’s model. This one I measured myself. Claude Code, Codex, and Gemini CLI are tuned around their vendor’s own model. Point them at a local model and the tool-call contract, system prompt, and stop conditions don’t quite line up.

While building Discipline’s end-of-turn check, I parsed 6,144 turn endings from my Claude Code transcripts and OpenCode database. The table below covers only the turns from the local open-weight model that prompted this work, and how often it ended a turn with “Now let me run the tests.” and stopped, waiting for a human to type “continue”.

In three other OpenCode turns, that same model stated its intent and dumped raw tool-call markup (</parameter> </function> </tool_call>) straight into its text reply, so nothing ran.

Those counts are small and come from one person’s sessions, so treat them as a smell. Even so, you can see how the same weights behave differently depending on which harness drives them. When an agent thinks it ran the test suite because it printed the tool call as text, it can write “all tests pass” in the PR description.

Looser defaults. When I audited agent settings for Discipline’s sandbox-config gate, Qwen Code had folder trust off by default, and several harnesses expose a single switch (tools.autoAccept, bypassPermissions, network_access) that disables approval prompts altogether. Local setups are also where people run agents without a container at all because everything is on their own laptop.

Point being, frontier models cheat when they get stuck, and local models hallucinate packages, drop tool calls, or run with looser defaults. Either way, the agent’s own summary of what it did is the last thing you should trust. Check the diff instead.

What Discipline Does

Discipline is a single static Rust binary that diffs your branch against its merge base and blocks the patterns above before they reach review. The same binary runs in GitHub Actions, GitLab CI, Forgejo and Gitea Actions, Argo, pre-commit, and inside the agent’s own loop through hooks.

Take a throwaway repository where the base branch has two Python tests and a .claude/settings.json with the sandbox enabled and curl denied. An “agent” commit titled fix: stabilize pricing tests weakens one assertion into an existence check, skips the second test, and opens up the settings file.

 def test_discount():
-    assert apply_discount(100.0, 15) == 85.0
+    assert apply_discount(100.0, 15) is not None

-def test_rejects_bad_pct():
+@pytest.mark.skip(reason="flaky")
+def test_rejects_bad_pct():

-  "permissions": { "defaultMode": "default", "deny": ["Bash(curl:*)"] },
-  "sandbox": { "enabled": true }
+  "permissions": { "defaultMode": "bypassPermissions", "deny": [] },
+  "sandbox": { "enabled": false }

Running Discipline v0.30.1 with built-in defaults and no configuration file catches all of them. I trimmed the gate table, doc links, the review-routing summary printed after the verdict, and two findings: a warning about the missing AGENTS.md and an error for the edited .claude/settings.json as an agent-instructions file, which is why the totals count 7 errors. The 18 disabled gates are opt-in.

$ discipline check --base main
error [assertion-reduction] Assertion Count Decreased In Existing Test [tests/test_pricing.py:4]
   Test `test_discount`: equality / pattern assertions dropped from 1 to 0 (weakened to a looser form).
   Remediation: Restore the assertions, or justify the drop on its own line in the PR body or a commit message: `allow-assertion-drop: test_discount <reason>`.

error [ignored-tests] Existing Test Skipped [tests/test_pricing.py:8]
   Test `test_rejects_bad_pct` no longer runs.
   Remediation: Fix the test, or justify it on its own line in the PR body or a commit message: `allow-ignore: test_rejects_bad_pct <reason>`.

error [sandbox-config] Sandbox Configuration Widened [.claude/settings.json]
   `permissions.deny` lost 1 entr(y/ies): `Bash(curl:*)` in `.claude/settings.json`.

error [sandbox-config] Sandbox Configuration Widened [.claude/settings.json]
   `permissions.defaultMode` moved from `default` to `bypassPermissions` in `.claude/settings.json`.

error [sandbox-config] Sandbox Configuration Widened [.claude/settings.json]
   `sandbox.enabled` switched off in `.claude/settings.json`.

error [test-floor] Test Count Below Floor
   Workspace test count (1) dropped below base ref count (2) [tolerance: 0].

gates:  22 passed, 5 failed, 18 disabled, 1 not evaluated (19 items examined)
errors: 7  warnings: 1  overrides: 0
Status: FAILED

If the commit tweaks the expected value from == 85.0 to == 85.01 instead, that gets flagged too.

error [assertion-reduction] Expected Value Changed In Existing Test [tests/test_pricing.py:5]
   Test `test_discount`: the assertion on line 5 now expects a different value; the assertion is otherwise unchanged.

Workarounds that leave the test files untouched still show up in the diff. Probing for the test runner in production code raises a warning, which blocks under --fail-on-warnings. Rewriting failed test reports in conftest.py or making the test script ignore failures both raise errors.

warning [stub-bodies] Test Runner Probe Added To Production Code [src/price.py:4]
   a check of whether `pytest` is loaded is added to production code at `src/price.py:4`: the code can behave one way under its tests and another in use.

error [harness-tampering] Pytest Hook Masks Test Results [tests/conftest.py:7]
   `tests/conftest.py` (a pytest `conftest.py`): `pytest_runtest_makereport` assigns the `outcome` of a test report. Exact form only; no value-flow analysis.

error [toolchain-config] Toolchain Configuration Weakened [package.json]
   `scripts.test` masks the test command's exit code (`jest --passWithNoTests || true`: the right side of `||` cannot fail); gained `--passWithNoTests` in `package.json`.

Notice the second message says “Exact form only.” Discipline matches the patterns it knows about, so an agent determined to hide a failure can still come up with one it hasn’t seen. I’ll get to those gaps shortly.

Fourteen tree-sitter language packs (Rust, Python, JS/TS, Go, Java, C, C++, C#, PHP, Ruby, Kotlin, Swift, Scala, Objective-C) and a PHPT parser read both sides of the diff to find test functions, assertion operands, and stub bodies. Simple regex scripts that count assert are easy for an agent to fool by moving the assertion under if False:. When a file fails to parse, a merge base won’t resolve, or a forge API returns half an answer, Discipline exits 2 (“could not check”) rather than passing the build.

With policy_from: base, which I’d set on any repository where agents open PRs, the policy that evaluates a pull request comes from the base branch. If a PR edits discipline.toml to disable a failing gate, Discipline still judges the diff against the base branch’s copy and flags the edit itself under config-integrity. A PR that adds the repository’s first discipline.toml is compared with the built-in defaults, so it can’t arrive already loosened. The default is head, the PR’s own copy, but even then the override budgets and the five gates that guard the pipeline are always read from the base. Edits to the CI workflow that runs the gate get caught under ci-integrity.

Sometimes a real change needs to skip a test or drop an assertion. You do that by putting a scoped waiver and reason on its own line in the PR body or commit message, such as allow-ignore: test_rejects_bad_pct upstream API is down, tracked in #42. With the defaults, anyone who can write the PR description or a commit message can write that line, including the agent; the remediation text in the output above even shows the syntax. So a waiver doesn’t prevent a shortcut so much as make it explicit, scoped, and countable. Directives can be capped, restricted to the PR description, made to fail the run, or made to wait for an approving review from someone other than the PR’s author. Running discipline audit across git history lists every waiver, loosened option, and grandfathered finding a branch accumulated. What no waiver can do is switch off one of the gates that guard the pipeline itself (config-integrity, ci-integrity, ratified-paths, sandbox-config, instruction-smuggling). A PR that disables one, demotes it, or exempts a path from it is refused whatever it carries, so that change has to land on the base branch on its own, through normal review.

CI catches bad edits after the fact, while hooks catch them mid-session. Running discipline hook install --agent claude-code (or codex, cursor, aider, copilot, agy, qwen, opencode) wires the checks into the agent’s post-edit hook so the model sees the finding and remediation instructions on its next turn. After an edit, the hook runs the gates that judge the changed files and the change as a whole; repository-wide ones such as the test-count floor wait for the end of the turn, which runs every gate. The agent waits for that check, about 4 seconds per edit on ripgrep and 8 on Discipline’s own repository (median, measured on an 8-vCPU cloud VM), still above the project’s own 2-second target¹¹. Installing with --observe logs what would have blocked without blocking, and the agent doesn’t wait for the check after an edit, which is the way to roll hooks out. There is also an opt-in end-of-turn check for the announce-and-stop bug from the table earlier. Every hook run is logged, and discipline hook log prints what each one allowed or blocked. Hooks judge the change by the base branch’s discipline.toml and ignore waiver lines, so the agent can’t argue its way past one mid-session. They also fail open: if the repository ships hook files but discipline isn’t on PATH, every hook exits 0 without checking anything, and discipline doctor warns about it. That’s on purpose, so a missing binary doesn’t break a teammate’s editor. Hooks are the fast feedback; the required CI check is the fail-closed layer.

A pre-tool hook also blocks edits or shell commands that reach into another worktree, which matters when you run multiple agents against the same repo. One of my own sessions rebased the same stacked branches another session was rebasing, and the two racing force-pushes nearly lost the stack. Discipline takes a lease per worktree and installs a git reference-transaction hook that refuses to move a branch held by another live session, regardless of which agent is running. It’s a guard against accidents between cooperating sessions; an agent set on getting around it can run git without hooks.

The other gates cover the mundane stuff that creeps into agent diffs: home paths, LAN IPs, and credentials, agent scratch files, unpinned actions, new dependencies and lockfiles repointed at another host, and test-count ratchets. The gate reference lists them all.

How It Compares to Other Tools

Most repositories already have some mix of agent instructions, path-based review rules, static analyzers, pre-commit hooks, LLM review bots, or container sandboxes. None of them compares the strength of a test suite or the scope of a config file between a base branch and a PR head.

SonarQube is the clearest example because it has test rules of its own. Its default “Sonar way” quality gate checks new code for no new issues, at least 80% coverage on new lines, and at most 3% duplication¹². It includes rules like S2699 (a test should include an assertion) and S1607 (a skipped test should give a reason)¹³, and Sonar sells an AI Code Assurance gate for projects with AI-generated code¹⁴. All of that inspects the branch as it stands after the change, and most agent erosion only shows up when you compare HEAD with base:

• Rewriting assert f() == 85.0 to assert f() is not None still leaves an assertion in the test, so S2699 passes and line coverage doesn’t move.
• Changing == 85.0 to == 85.01 is clean code that checks the wrong answer.
• Adding @pytest.mark.skip(reason="flaky") provides a reason string, which satisfies S1607.
• Deleting a failing test file removes issues and makes the dashboard greener.
• Setting bypassPermissions, removing a deny entry, adding continue-on-error: true, or introducing a new postinstall script are config changes that Sonar doesn’t model as wider than before.

Analysis scope has the same problem. Sonar keeps its quality gate definition on the server where a PR can’t touch it, which is the right call, but sonar.exclusions and sonar.coverage.exclusions can be set in sonar-project.properties or as CI scanner parameters, which override the server UI¹⁵, and a // NOSONAR comment silences every issue on its line¹⁶. An agent stuck on a failing file can exclude it in the same PR. Discipline’s toolchain-config gate compares that scope between base and head, and suppression-delta counts new // NOSONAR comments in Java and Kotlin alongside #[allow], # noqa, // eslint-disable, and // nolint. Here is the trimmed output on a PR that adds an exclusion and stops CI from waiting on the quality gate:

$ discipline check --base main
error [toolchain-config] Toolchain Configuration Weakened [sonar-project.properties]
   `sonar.exclusions` gained 1 entr(y/ies): `src/payments/**` in `sonar-project.properties`.

error [toolchain-config] Toolchain Configuration Weakened [.github/workflows/sonar.yml:8]
   `sonar.qualitygate.wait` switched off in `.github/workflows/sonar.yml`.

LLM review bots (Copilot Code Review, CodeRabbit, Qodo PR-Agent) have a different problem: they read the diff alongside the PR description written by the same agent that took the shortcut. A one-line change to an expected value, buried in a 30-file diff under a commit titled fix: update discount test for rounding, arrives with its excuse attached, and nothing forces the reviewer to flag it.

The closest relative is mutation testing on the diff (cargo-mutants --in-diff, Stryker’s incremental mode). It runs the code and asks whether the tests still catch a changed line, which covers the value-flow cases Discipline misses, at the cost of a working build and a much slower run, so use both. Semgrep’s diff-aware mode (--baseline-commit) reports only new findings, but its rules match patterns in the code rather than comparing a test’s assertions across both sides. And for Claude Code, administrator-managed settings take precedence over a project’s .claude/settings.json, the stronger fix for the sandbox case where you can use it.

On the other hand, SonarQube, Semgrep, and CodeQL cover plenty of ground that Discipline never looks at: taint analysis, SQL injection, bug patterns, duplication, coverage measurement, and historical dashboards. Containers enforce runtime isolation that a diff checker cannot. Use those tools for what they are good at, and use Discipline to make sure the PR didn’t weaken the tests or the config doing the checking.

What It Does Not Catch

Discipline only inspects diffs and config files, so it won’t contain a running agent. In April 2026, Claude Code fixed CVE-2026-39861, where a sandboxed process could create a symlink pointing outside the workspace and the unsandboxed side of the tool would follow it and write there without asking¹⁷; exploiting it only took untrusted text in the context window. In July 2025, a Replit agent ran destructive commands against a production database during an explicit code freeze, then said rollback was impossible when a backup existed¹⁸. Neither shows up in a diff. Watching system calls, filtering egress, and hiding credentials from a process belong to a container or VM with no credentials and an egress allow-list. Any check that runs inside the process it is supposed to contain can be bypassed by that process. Discipline only guards the configuration of those boundaries because config edits show up as a diff. It also ignores the agent’s reasoning traces, since the agent writes those itself.

The AST gates have blind spots too, because they compare syntax without tracking value flow. Writing let b = a; assert_eq!(a, b) compares a value with itself through an alias and goes unreported. The same is true when an assertion’s operands never reach the code under test, or when a test is rewritten with the same assertion count and strength. And the forms it does read, it reads exactly. If an empty cases array is declared inside the test, Discipline sees that cases.forEach(...) never runs the assertion moved into it. Declare the array at the top of the file and it doesn’t. The checks for an expected value computed by the function under test, and for an assertion on a function the test itself patched, read Python only. The gate reference lists what each rule misses. You still need held-out tests and human review to catch those.

Finally, any forge check is only as strong as the credentials behind it. If your agents push commits using your own login, a rule requiring a “review by someone else” is satisfied by you. That is how Discipline’s own repository is set up today, and the threat model calls it out.

Does It Work?

Short version: the engine is deterministic and blocks far fewer PRs than a crude path filter. How many of its findings are worth blocking on depends on who grades them, and I don’t have a precision number I’d defend yet.

The study replayed Discipline across 232 merged pull requests in 8 public repositories (ruff, deno, vite, fastapi, junit5, go, tokio, ripgrep), each checked at its pinned base and head commits. 149 came from coding agents (Claude, Copilot, Cursor) and 83 from dependency bots (Dependabot, Renovate). A naive twin flags any PR touching package.json, CI files, or settings, which is roughly what a CODEOWNERS rule on those paths gives you. Two blinded LLM raters labeled every finding against a written rubric, with a tie-breaker when they disagreed. The pre-registration, every finding, every label, and the replay script are in the repository^{19 20}.

What I’d stand behind is the engine. Each release reproduces its own findings exactly on replay, and the rule changes between releases account for every finding that came or went. The labeled run used v0.28.1, and replaying v0.30.1 over the same 232 PRs gives the same 127 findings and exit codes. Discipline blocks 41 of the 232 PRs (17.7%, CI 13.3%–23.1%), against 130 (56.0%) for the path filter. That 17.7% is also the friction: about one in six PRs these maintainers merged would have needed a fix or a waiver first.

Precision is the open question, and the two rater pools seem to have measured different things. The rubric counts a finding as a true positive when it shows the rule’s defect and a maintainer would want it fixed. The first pool effectively graded rule fidelity: does the finding correctly report what the rule says? The second leaned on actionability: would this project’s maintainers block on it?

The biggest single source of the gap is one Dependabot PR to tokio that bumps actions/checkout from @v6 to @v7. It carries 52 of the 122 structural findings, each an action pinned to a tag rather than a commit hash, which is what the ci-integrity rule reports. The first pool called all 52 true positives and the second called them false, plausibly because tokio pins by tag on purpose (the raters give labels, not reasons). Much of the rest is ruff snapshot updates under golden-output, new dependencies under dependency-delta, and more tag pins: rules that, by the first pool’s reading, fire correctly on changes these projects make routinely.

Neither column is solid ground. The first pool’s labels were assigned in the study’s first run and carried forward since. Those two raters agreed with each other at Cohen’s κ = 0.96 (1 is perfect agreement, 0 is chance), above the pre-registered 0.75 floor, but four later pools re-rated 20 of their items blind and matched only 11 to 15, every disagreement going from true to false. Consistent raters can share a bias. The first-pool numbers are also in-sample, since the v0.28 rule fixes were written against items from these same PRs. The second pool’s raters agreed at κ = 0.38, below the floor, so its numbers fail the study’s own reliability bar. Every rater was an isolated LLM subagent, and no human has labeled these 127 findings yet.

The study also says little about the gates this post spends the most time on. Of the 122 structural findings, 100 come from CI, dependency, snapshot, and toolchain checks, 20 from the gates that read code with tree-sitter (assertion-reduction, error-swallowing, deletion-rationale, stub-bodies), and 2 from the test-count floor. ignored-tests, vacuous-tests, harness-tampering, and sandbox-config didn’t fire once in 232 merged PRs. Merged upstream PRs rarely skip tests, which is good news, but it leaves those gates unmeasured.

Under the pre-registered rules, nothing passes. The gate needed a Wilson lower bound of 90% precision across at least 141 findings, and 122 falls short before the labels even come into it (verdict insufficient-n). Two more caveats. The study replays only the files each PR changed, so a Markdown link to a file that exists in the full tree looked broken, accounting for one of the 41 blocked PRs and one of the 5 heuristic findings. And it doesn’t measure recall, so I don’t know what share of bad changes across those 232 PRs Discipline missed. A recall study is pre-registered in the repository²¹ to test local open-weight agents on Impossible-SWE-bench (where every pass is a known cheat, so the question becomes how many of those cheats the gates catch). Its design was locked after a pilot, and the confirmatory run hasn’t happened yet. A larger precision study on agent PRs from the AIDev dataset, sized per gate and with a human-labeled subset, is pre-registered too²².

Point being, the tool is consistent, and whether its findings belong in your merge gate is a policy call about your repository. A tag-pinned GitHub Action or a snapshot update may be fine in your project, and two LLM raters can’t decide that for you. Both pools rate the heuristic prose gates lowest, and the path filter is a sanity floor that Discipline clears under both. So start in advisory mode, run replay on your own history, and make the gates required once their findings are ones you’d actually block on.

Getting Started

brew install orieg/tap/discipline          # or: cargo install discipline, apt, dnf
discipline replay --last 50                # what it would have blocked in your last 50 merges
discipline check                           # this branch vs origin/main
discipline hook install --agent claude-code
discipline init                            # optional: a starter discipline.toml to tune

I’d start with replay, which answers “how noisy is this going to be on my repository” before anyone has to adopt anything. On an existing codebase, discipline baseline --write grandfathers current findings so only new regressions fail. On your own, the hook is most of the value: the agent sees the finding on its next turn. For a team, run it in CI as a required check that reads its policy from the base branch. Pin the action to a commit, since a tag ref is exactly what ci-integrity flags:

on:
  pull_request:
    types: [opened, synchronize, reopened, edited]
permissions:
  contents: read
jobs:
  discipline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          fetch-depth: 0
      - uses: orieg/discipline@394a472f4d601360610eade3e0eef2fbc757bfb7 # v0.30.1
        with:
          policy_from: base

Then mark it as a required check in your branch protection settings (discipline doctor tells you if it isn’t). To keep the agent from waiving its own findings, read directives from the PR description only and make any waiver fail the run, or wait for a reviewer other than the author:

[directives]
sources = ["pr-body"]
fail_on_overrides = true   # or require_approval = true, with allowed_override_actors

Documentation, the gate reference with before/after diffs for each rule, and a sandbox tutorial are available at orieg.github.io/discipline.

Agents are going to write more of our code, across more hardware and more models than any single vendor tunes a harness for. Trust their reports less, and put a fail-closed check in CI on the one place they can’t hide a shortcut: the diff.

What does your agent do when the tests won’t go green? If you run into examples Discipline misses, open an issue with the diff. I’d like to see them.

References