peltier.io

Spot cooling for hot paths. A profile-guided optimization agent that only ships wins it can prove.

Peltier profiles real binaries and services, isolates hot paths, proposes optimizations, and accepts a change only when two independent bars are cleared:

1. Behavioral equivalence — the change provably preserves behavior (upstream tests, byte-identical golden replay, differential fuzzing, sanitizers) or it is discarded.
2. Statistical significance — the speedup's bootstrap 95% CI lower bound clears a threshold on trusted, calibrated hardware.

Everything else is commentary. A change without both is not a win; it is a ledger row explaining why not.

What it has actually done

Every number here carries its 95% confidence interval and its workload, and every one survived the gates above. These are not projections.

Across five phases and 34 ledger rows, with zero shipped false accepts — including two occasions where the pipeline itself over-accepted and the audit caught it before anything shipped, each becoming a permanent new gate.

The differentiator

Consultants sell surveys; compilers sell flags. Peltier sells verified deltas with the methodology attached — a stopwatch on pinned hardware, a proof of equivalence, and an ROI figure whose confidence interval you can quote to a hostile reviewer. That is the entire product, and this site is how it works.

Start with the thesis, or jump to a case study. To run this discipline on your own code from your own agent, see the peltier skill.

The thesis: trust is the product

Performance work has a credibility problem. "We made it 15% faster" is almost always some mix of a noisy benchmark, a workload that flatters the change, and a behavioral difference nobody checked. The number is real to the person who measured it and worthless to everyone downstream.

Peltier is built on the opposite premise: a performance claim is a liability until it is proven, and the proof ships with the number.

Three non-negotiables

1. No unverified performance claims. A change without passing equivalence gates and a statistically significant benchmark delta does not exist. It is discarded and logged.
2. Measure before optimizing. No patch is proposed until profile data identifies the hot path. A single benchmark run is never trusted.
3. Cheap wins first. Build flags → LTO → PGO → allocator swap → then code changes. The ledger prevents re-attempting a dead end.

What "proven" means here

A win must clear both of two independent bars, because either one alone is gameable:

• Equivalence stops "optimizations" that are actually behavior changes. A JSON parser that's faster because it stopped validating numbers is not faster; it's broken. Byte-identical golden replay and differential fuzzing catch this.
• Significance stops noise from masquerading as a win. The bar is not "the median improved" — it is "the bootstrap 95% CI lower bound clears the threshold," measured by interleaving baseline and candidate on pinned hardware that has passed its own A/A calibration.

A real +5.7% median that couldn't clear the significance bar (its CI lower bound was +0.2%) was rejected — and that rejection is in the ledger, because a rejection is a complete, valid outcome.

Why an agent

The loop — profile, hypothesize, patch, gate, measure, record — is mechanical and tedious, and the temptation to fudge a number is exactly what a tired human succumbs to. An agent proposes; the trust layer disposes. The agent cannot write the ledger, cannot move the baseline, cannot touch the thresholds. It can only propose a patch and ask the harness to judge it. The architecture is designed so that even a compromised or adversarial proposer cannot manufacture a false win.

The detective and the court

The optimizer is a Bayesian detective. The trust layer is a frequentist court.

That sentence is the whole design. The two halves run on opposite statistical philosophies, deliberately:

     BAYESIAN DETECTIVE                        FREQUENTIST COURT
     the optimizer: agent loop,                the trust layer: gates,
     playbook prior, policy ranking            calibrated bench, verdict

   ┌───────────────────────────┐            ┌────────────────────────────┐
   │ generate hypotheses       │            │ prove equivalence first    │
   │ prioritize the docket     │  proposes  │ calibrate the instrument   │
   │ estimate likely payoff    │ ─────────▶ │ measure, interleaved A/B   │
   └───────────────────────────┘            │ reject noise (CI lower     │
               ▲                            │ bound vs. threshold)       │
               │                            └─────────────┬──────────────┘
               │                                          ▼
               │        evidence flows back            verdict
               └───── ledger row + explain ◀──────────────┘

The detective reasons like a Bayesian: it holds a prior over where wins live (the playbook's cheapest-first ordering), updates it from accumulated evidence (the append-only ledger — failures included, because failures are the most informative updates), ranks its next bets (crates/policy, by the Wilson lower bound of each class's proven win rate — even the learning layer bets pessimistically), and reads the post-verdict diagnosis so the next hypothesis is sharper than the last. Its "estimate likely payoff" is a belief, and stays one: dollar figures are minted only by the court's mechanical report, only from accepted rows.

The court reasons like a frequentist, because its audience is a hostile reviewer who shares none of your priors. Its verdicts are decision procedures with measured long-run error rates: the A/A calibration bounds the false-positive rate and the injected-regression test bounds the power — per machine, empirically, before any real measurement is believed. The accept rule is a conservative bound, not a posterior; there is no prior anywhere on the accept path, so there is nothing to argue about except the evidence.

The return edge is what makes the detective Bayesian at all — and note its direction. Evidence flows back; authority never does. The invariant underneath the whole system:

Priors may steer where you look. They may never touch what you conclude.

A wrong prior in the detective costs compute — a few wasted attempts, each honestly ledgered. A prior in the court could cost a false claim, the one unrecoverable failure for a project whose product is trust. The gates are what make speculative, prior-driven exploration safe: the detective can believe anything it likes, because nothing it believes survives into a number.

Architecture

Peltier is split into a trust layer (Rust, the agent cannot write it) and an untrusted proposer (the agent). The split is structural, not advisory: the guarantees hold even if the proposer is adversarial.

crates/            Trust layer — agent has NO write access (SPEC §10)
  bench-runner/    Interleaved A/B timing, bootstrap CIs, A/A calibration,
                   service-mode latency (coordinated-omission correct)
  diff-test/       Equivalence gates, corpus hash-pinning, per-target spec
  ledger/          Append-only SQLite attempt ledger (enforced by triggers)
  report/          ROI: speedup CI → cores → dollars, methodology inline
  explain/         Advisory post-verdict diagnosis of a ledger row (why it
                   won or lost) — never on the accept path
  policy/          Learned class-selection prior from the ledger (advisory)
  verdict/         The pipeline in one command: gates → bench → ledger row
  harnessd/        The one door the agent talks through (7 JSON ops)
agent/             Untrusted proposer (Claude Agent SDK, Python)
playbook/          Optimization classes 1–7, tried strictly cheapest-first
config/            accept.toml (thresholds), pricing.toml (ROI inputs)
targets/           Vendored OSS targets — the only agent-writable path
corpora/           Hash-pinned golden-replay inputs (read-only to agent)
results/           Calibration evidence, case studies, generated reports
.claude/skills/    The trust layer as a consumable Agent Skill — mirrored
                   byte-identically at .agents/skills/ (CI-enforced); see
                   "Consume it: the peltier skill"

The seven-tool boundary

The agent never runs a shell, never writes a file, never touches the ledger directly. It speaks to harnessd — the only door — through exactly seven JSON operations:

Note what is absent: no write outside targets/<name>/workspace, no shell, no way to run the bench or write the ledger except through the verdict binary, which only records a pass after the full gated pipeline.

The data flow of one attempt

profile ─▶ agent hypothesizes ─▶ propose_patch ─▶ [path allowlist + git apply]
                                                          │
   ledger row ◀─ verdict ◀─ bench (interleaved A/B) ◀─ gates (tests, golden,
   (append-only)             vs pristine-rebuilt         fuzz, sanitizers)
                             baseline

The baseline is rebuilt from a pristine checkout every session — never from the agent's workspace — so a patch cannot poison its own comparison. The verdict is decided by the CI-lower-bound rule and written to an append-only ledger the agent cannot mutate.

The next chapters walk each stage: measurement, equivalence gates, the agent loop, and the isolation that makes the boundary load-bearing.

Measurement discipline

The product is a trustworthy number, so the measurement is where most of the engineering went. The reference is Mytkowicz et al., Producing Wrong Data Without Doing Anything Obviously Wrong — the default state of a benchmark is "subtly lying."

Interleaved A/B, never sequential

Baseline and candidate run interleaved (ABABAB…), never in sequential blocks. Thermal drift, a background cron job, a noisy neighbor — anything that changes over the run hits both sides equally instead of biasing one. Warm-up runs are discarded.

The baseline is always pristine

The baseline binary is rebuilt from a clean checkout each session, with the candidate's patch stashed. A patch can never silently become part of its own baseline (a real bug that bit an early smoke test, now closed by verdict --rebuild-baseline).

Bootstrap CIs, and the lower-bound rule

Each side yields ≥30 measured samples. The statistic is the bootstrap 95% CI of the ratio-of-medians (deterministic xorshift seed, so a result reproduces). A change is accepted only if the CI lower bound clears the threshold in config/accept.toml (default 2%) — not the median. The number you commit to is the one that survives the interval.

Beyond wall time: max RSS and PMU counters

Wall time is the only metric a verdict is ever decided on. Two supporting measurements exist alongside it:

• Max RSS is captured for every timed run on Unix (wait4's ru_maxrss) and recorded in the run's environment fingerprint as baseline/candidate medians — so a speedup that quietly pays for itself in memory is visible next to the number it bought.
• PMU counters (cycles, instructions, cache misses, branch misses) are opt-in diagnostics: bench-runner compare --perf-stat wraps each timed run in perf stat and prints per-side medians with the workload caveat attached. PMU access is unavailable in most VMs, and the counters never participate in the accept decision — they explain a wall-time delta, they do not replace it.

The hardware has to earn trust first

Before any measurement session, the harness runs two self-tests, recorded in results/calibration/:

• A/A: run the same binary as both sides. It must produce a null verdict. A "speedup" from identical binaries is a calibration failure.
• Regression injection: a synthetic 5% slowdown (a real busy-wait in the timed window) must be detected ≥95% of the time.

CPU pinning (taskset -c 2) alone tightened the A/A confidence interval ~5× on the dev container. Every calibration anchor — local and on GitHub's shared runners — passed both bars before its numbers were trusted.

Service mode: coordinated omission

Phase 4 measures latency under load, where the classic trap is coordinated omission: a closed-loop client that waits for a slow response simply sends fewer requests during the stall, so the stall never appears in the latency histogram. Peltier's load generator is open-loop — requests are scheduled at a fixed rate and each latency is measured from its intended send time. When the server falls behind, the requests queued behind it each carry the full delay a real client would have seen. Nothing is omitted.

The service measurement is calibrated the same way: 0/10 A/A false positives, 10/10 injected-5%-latency-regression detections (results/calibration/cjson-service-aa.json).

Percentiles are claimed only when the measurement can pin them down. The cJSON service's p50 latency win was CI-tight and accepted (+6.2%, [+5.8%, +7.2%]); its p99 CI was [0.07, 4.97] — uselessly wide on a single-worker loopback tail — so it was rejected, even though the median pointed the right way. A number the measurement can't pin down does not ship.

Equivalence gates

A faster binary that behaves differently is not an optimization — it is a regression with good marketing. Before any speedup is even measured, the candidate must prove it does the same thing the baseline does. These are hard gates: a failure is rejected-gate and the bench never runs.

The layers

0. Pin checks. Before any gate runs, the corpus is verified against its committed MANIFEST.sha256 and the harness refuses to run on a mismatch — nothing is measured on tampered inputs. This ships and is enforced for every target.

   The test suite has the same machinery but is not yet active: diff-test verifies corpora/<t>/TESTSUITE.sha256 when the file is present (mismatch = hard refusal) and prints an explicit "suite unpinned" warning when it is absent. No target ships a suite pin today, so this check currently constrains nothing. It matters because upstream suites are vendored under targets/<t>/workspace/ — an agent-writable path — so "the agent cannot edit the tests" rests on a pin that still has to be generated (scripts/pin-testsuite.sh, a deliberate human action, run after a fetch). Until then, that specific guarantee is a mechanism, not a fact.

1. Upstream test suite. The target's own tests, run against the patched code. For cJSON that is the vendored Unity/CTest suite (19 tests: parse_number, parse_string, print_*, misc_tests, …); for the Rust targets, cargo test.

2. Golden replay — byte-identical. The candidate processes the hash-pinned corpus and its output is hashed and compared to a pinned GOLDEN.sha256. A code counter that miscounts one line, a JSON printer that serializes one float differently — caught here. The corpus itself is pinned via MANIFEST.sha256 and the harness refuses to run on a mismatch.

3. Differential fuzzing. Pristine vs candidate on thousands of mutated inputs, comparing output.

   Where it runs. Fuzz is the one gate that is not a property of the candidate alone: it differs the candidate against a pristine baseline, so it needs both binaries. That means it runs only on the accept path (just verdict, the one flow that rebuilds a pristine baseline). The standalone just gates flow has no baseline, so the gate reports Skipped, with that reason recorded — it is never faked by comparing the candidate against itself. just gates is a fast equivalence check, not a fuzz run. The gate is graded strictly: diff-test runs the target's declared [gates].fuzz command and parses a FUZZ-RESULT iters=<n> divergences=<m> line; a run that never prints one fails, rather than passing by silence.

   The hard rule. A machine accepted verdict is now impossible without this gate having actually passed. A skipped or failed fuzz gate caps the verdict at needs-human-review — because an accept minted without differential fuzz is an unverified claim.

   Honest history. Through Phase 5 this ran as an out-of-band per-attempt script (scripts/diff-fuzz-<target>.py), not inside the pipeline — so 8 of the 10 accepted ledger rows record fuzz_iters=0 even where a 10,000-input audit run is documented in the case study. The two exceptions are the Phase 0 tokei rows, which record 4,332. Those out-of-band runs produced real findings — the tokei wins passed only after canonicalizing benign parallel-walker output ordering (differential gates must compare semantics, not bytes, where the target is legitimately nondeterministic in presentation), and the cJSON win cleared quote/number/escape edge cases with 0 divergences.

4. Sanitizers. ASan+UBSan (ASan/LSan on the Rust targets) is built and run against the pinned workload on every would-be accept — the sanitizer gate is machine-enforced, not manual (see below). A flagged run, or a target that declares no sanitizer build at all, caps the accept at needs-human-review.

   A TSan lane exists on the same accept path, but it only runs when a target declares [build].tsan. No target declares it today, so there is currently no TSan coverage — the lane is wired and dormant, and a patch touching threads needs that opt-in added before it can be checked. (MIRI, named in the spec, was never implemented.) The concurrency safety net that is live is the review tier below, which routes any concurrency-token patch to a human.

The tier rule (SPEC §8)

Some changes can pass every gate and still be unsafe to auto-ship. A change that touches floating-point ordering, concurrency primitives, unsafe, or carries a sanitizer flag is never auto-accepted — its verdict is capped at needs-human-review. The tier is decided by the nature of the change, never by the size of the number. The routing is machine-enforced: verdict runs a conservative lexical risk classifier over the patch's changed lines (concurrency / unsafe / floating-point token lists — deliberately over-triggering, not a semantic analysis) and any signal, or the use of fp-tolerance equivalence mode at all, forces a would-be accept to review. The mimalloc allocator swap (case study) is the canonical example: a real +4.6% win, correctly held for a human ruling because a global allocator change's safety is not something one benchmark can clear.

When a gate caught the pipeline itself

The sanitizer gate exists in its current, machine-enforced form because it once wasn't. The pipeline auto-accepted a comrak patch that skipped AST teardown for a +10.7% win — and the audit's manual LeakSanitizer run found it leaks the arena at exit. It was overturned before shipping and the fix made verdict build with ASan/UBSan and run the pinned workload on every accept, capping any flagged win at needs-human-review. The full story is the caught false-accept.

The agent loop

The proposer is a Claude Agent SDK loop (agent/hotpath_agent/) driving the seven harness tools over a standalone MCP server. It has no built-in file, shell, or web tools in its allowlist — only the harness door.

One attempt, start to finish

1. Read the ledger. What classes and hypotheses were already tried on this target? The ledger is the anti-double-attempt memory: a class may be re-entered with a materially new hypothesis, never a duplicate.
2. Read the profile. callgrind cache-sim ranks hotspots with per- function cache-miss and branch-mispredict columns, so the agent can tell an allocation-churn hotspot from a branch-bound one and pick the right playbook class.
3. Pick the cheapest untried class whose profile preconditions are met, and read its playbook entry.
4. Read the source (windowed, at the pinned commit) and propose a patch with a stated hypothesis.
5. Run the verdict and poll for the result.
6. Report it verbatim. A rejection with a clean ledger row is a successful outcome. Do not iterate more than twice on a rejected hypothesis without a new idea.

What the loop proved about itself

Across an unattended grind of 20+ audited attempts, the loop demonstrated the behavior the architecture is designed to produce:

• It chose the cheapest untried class each time, reading the ledger to avoid re-grinding a dead end.
• Every patch went through propose_patch (path-allowlist + git apply) and the full gate+bench pipeline. Malformed diffs were refused by the harness, not the agent's goodwill.
• It twice refused to fabricate a verdict it could not read back. When a multi-minute pipeline outran the MCP transport's timeout, the agent checked the ledger, confirmed the row wasn't written, and reported the verdict as unreadable, not invented — asking a human to read the ledger rather than guessing a number. That refusal, under a real failure mode, is the anti-reward-hacking design working.

The honest-failure record

The ledger records every attempt, including the rejections, because a rejection is training data and a demonstration of the bar. Of the audited unattended attempts, most were honest rejections — real mechanisms that simply didn't clear the 2% CI-lower-bound bar — a handful were verified wins, and one was a rejected-gate where golden replay caught the agent's patch producing wrong output that a green test suite would have shipped.

The load-bearing number is zero shipped false accepts across the whole project.

Isolation & anti-reward-hacking

The guarantees only matter if the proposer cannot cheat. Peltier's defense is structural: the agent is denied the capabilities it would need to manufacture a false win, at the OS level, not by prompt.

What holds even if the proposer is adversarial

• Writes go through one door. propose_patch is the only harness- mediated write path: a diff is path-allowlisted (relative paths only, no .., no .git, nothing outside targets/<name>/workspace) and then git apply-ed rooted in that workspace. A patch aimed at config/ or crates/ is refused before any apply. Verified against a malicious accept-threshold diff.
• The ledger is append-only. UPDATE and DELETE are refused by SQLite triggers; accepted rows are written only by the verdict binary after the gated pipeline. The agent cannot forge a passing verdict through the harness.
• The baseline is unbypassable. It is rebuilt from a pristine checkout; the agent's workspace never becomes the comparison.

OS-level isolation (SPEC §10)

The SDK's tool allow/deny list is defense-in-depth, not the boundary — scripts/agent-isolated.sh provides the real one, verified by just isolation-check (19 checks, both modes):

• mountns mode — the whole agent process tree runs in a private mount namespace with the repo bind-mounted read-only and CAP_SYS_ADMIN/ CAP_SYS_PTRACE dropped from the bounding set. Repo writes fail with EROFS; a forged INSERT, a direct verdict invocation, a mount -o remount,rw, and even userns re-entry to undo the mount are all refused.
• user mode — the agent runs as an unprivileged user whose only write path into the repo is the harnessd Unix socket; the trust layer is owned by another uid. This is the production shape (docs/infra.md).

Network isolation on the verdict path

Target code should not be able to phone home during a measurement. What is actually enforced, stated honestly:

• The agent path is wrapped. When harnessd launches the verdict pipeline, it execs through scripts/no-net.sh, which runs it inside an unprivileged network namespace (unshare --net --map-current-user). HOTPATH_VERDICT_WRAPPER overrides the wrapper.
• It fails closed. Where namespaces are unavailable the wrapper exits 97 without running the pipeline — it does not degrade silently. The one bypass, HOTPATH_ALLOW_UNISOLATED=1, runs the pipeline with full network access; it warns loudly on stderr and the resulting ledger row records the isolation note no-net.sh (HOTPATH_ALLOW_UNISOLATED=1 — network NOT isolated). A run cannot claim an isolation it did not have.
• In CI, the bench workload runs under docker run --network=none.

Two limits, stated plainly rather than buried:

• Only the harnessd (agent) path is wrapped. A human who invokes just verdict directly runs it unwrapped on the host; those rows are stamped isolation: "unwrapped-host" in the ledger. The wrapper is a property of the agent pipeline, not of the verdict binary.
• Full-container isolation is still an open gap. The namespace isolates the network — not the filesystem, not the syscall surface. The fully containerized, seccomp-restricted bench container of SPEC §10 is not built. The agent-side OS boundary above is a separate mechanism and is shipped.

The two times the pipeline was wrong

Anti-reward-hacking is not "it never fails" — it is "when it fails, the audit catches it and the system gets harder." That happened twice, and both are documented, not hidden:

1. A leaking teardown patch the bench loved was auto-accepted and then overturned by the 100% human audit's LeakSanitizer run. The fix made the sanitizer gate machine-enforced on every accept.
2. During the adversarial review, the ROI report generator was found to mint a clean dollar figure for that same overturned row (it trusted the immutable ledger's historical accepted verdict). The fix made the report treat an accepted-but-not- sanitizer-clean row as not shippable.

Both defects were found by attacking the system's own output, fixed, and turned into permanent gates. Zero false accepts shipped.

The playbook

Optimizations are tried in a fixed order — cheapest and safest first — because cost-per-attempt and risk both rise as you go down the list, and the ledger stops you from re-grinding a class that already has a verdict.

Six classes were exercised across the case studies, with verified wins on classes 1, 2, 3, and 5 — spanning both a Rust and a C target (the C win is the class-5 cJSON rewrite; cJSON's class-3 attempt was an honest rejected-bench, its CI lower bound short of the 2% bar).

Why the ordering is load-bearing

The cheapest class produced the project's single largest individual win: LTO on tokei, +10.4%, a pure build-config change with no source edit. The lesson the ledger enforces is the one every performance engineer learns the hard way — you do not hand-optimize a hot loop before you have turned on the optimizer's own switches. An agent that jumped straight to SIMD would have missed a bigger, safer, free win sitting one config flag away.

Each class entry (playbook/0N-*.md) carries its preconditions (the profile signature that justifies trying it), its procedure, its verification requirements, and its known failure modes. The agent reads the profile, matches the signature, and picks the cheapest class whose preconditions are met and that isn't already spent on this hotspot.

ROI: from stopwatch to dollars

The report is the pitch artifact, and it answers exactly one question: what did the stopwatch say, and what does that buy? It is generated mechanically — just report <run-id> reads a ledger row and the calibrated bench and renders the numbers; nothing is hand-typed.

The three conversions

• Throughput → cores → dollars. A speedup s on a CPU-bound fleet of N cores returns N × (1 − 1/s) cores; at a $/core-hour rate that is an annual saving. Both flow through the speedup's confidence interval, so the report quotes a range.
• Latency → percentile deltas → value. Under replayed load, the p50/p99 latency speedups with their CIs; a p50 service-time reduction maps to throughput capacity for a CPU-bound service.
• Batch windows → wall-clock hours returned to a schedule.

The rule that makes it survive review

Every figure carries its CI and its workload, and the methodology ships inside the report — the interleaved A/B, the pristine baseline, the bootstrap CI, the gates, the calibration. And the number the report tells you to quote is the CI lower bound, never the median:

On a 500-core fleet at $0.04/core-hour, the cJSON service win (+6.2% p50 latency) returns 27.5 cores / $9,621 per year at the CI lower bound (median 29.4 / $10,290). The point estimate is not a promise.

What the report refuses to do

It will not mint a clean dollar figure for a change that wasn't shipped. A row whose verdict isn't accepted, or an accepted row that isn't sanitizer-clean (a pre-gate accept the audit overturned), gets a bold Not shipped banner before any number. That guard was itself added by the adversarial review, which caught the generator pitching the overturned comrak patch. The pitch artifact is held to the same standard as everything else: it cannot claim what wasn't proven.

Case studies

Five verified case studies, each reproducible from a pinned corpus through the same gated pipeline, each number carrying its 95% CI and workload.

• tokei — compounding algorithmic wins (Rust) — three class-5 wins on the same hot function, each unattended and audited.
• Cheap wins first: build flags & allocators — LTO accepted (+10%), mimalloc held for a human ruling then accepted-scoped.
• The caught false-accept (comrak) — the leaking teardown the bench loved, overturned by audit. The most important study.
• cJSON — the cross-language proof (C) — a sub-threshold win rejected, then a byte-identical win accepted, on a C target.
• Services & scale: latency under load — the cJSON win measured as a latency delta under coordinated-omission-correct replay, turned into a dollar figure mechanically.

Case study 1 — tokei: compounding algorithmic wins (Rust, class 5)

Target: tokei v14.0.0 — a source-code line counter. Workload: 221 MB / 4360 files (8 replicas of the Pro Git book + comrak's own tree), single rayon thread, CPU pinned. Result: three independent, CI-verified class-5 wins on the same hot function — each proposed by the unattended agent and 100%-audited. Multiplied together they are ~1.26× over the Phase-0 baseline (1.107 × 1.037 × 1.099), but that product is an approximation, not a measured figure: each speedup carries its own bootstrap CI against its own baseline, and confidence intervals do not compose through multiplication. The trustworthy claims are the three individual increments below.

The hot path

tokei spends the overwhelming majority of its instructions in one function — LanguageType::parse_lines and the per-byte loop inside SyntaxCounter::perform_multi_line_analysis. The cache-sim profile put it at 55.9% of instructions, 62% of all branches, 44% of branch mispredicts on the pinned corpus: for every byte of every line, the counter tested a stack of quote/comment token matchers.

The three wins

Each is a redundant-work elimination: the first-byte gate replaces five per-byte matcher loops with one table load; the whitespace-break change drops a full backward scan of the file tail per byte; the streaming scan replaces a per-line aho-corasick restart (which kept memmem's SIMD prefilter in its short-haystack slow path) with a single leftmost search advanced by integer comparisons.

Why the numbers are trustworthy

• Byte-identical golden replay on the pinned corpus for every win — a code counter that miscounts one line fails the gate.
• 10,000-input differential fuzz (scripts/diff-fuzz-tokei.py), pristine vs candidate, 0 divergences — after canonicalizing tokei's benign parallel-walker output ordering (a real finding: raw-byte differential comparison over-counts when the target is legitimately nondeterministic in presentation).
• ASan/LSan clean on the patched tree over the full corpus.
• Interleaved A/B vs a pristine-rebuilt baseline, bootstrap 95% CI lower bound ≥ the 2% bar. phase2-tokei-008 was the first accept through the machine-enforced sanitizer gate (see case study 3).

What it demonstrates

The richest wins on a mature Rust binary were not exotic — they were "this loop does work it doesn't need to, on every byte." The agent found each by reading the cache-sim profile, checking the ledger to avoid re-attempting a class on the same hotspot, and proposing the cheapest untried mechanism. Three separate hypotheses, three separate verified increments, all on the same 40%-of-runtime function — and one honest rejection in between (phase2-tokei-009, a real +5.7% median whose noisy CI lower bound of +0.2% could not clear the bar, so it does not count).

Note (2026-07-13). The 10,000-input differential fuzz cited above ran out-of-band (scripts/diff-fuzz-tokei.py, per-attempt), not inside the verdict pipeline — so these ledger rows record fuzz_iters=0 in their machine record. The fuzz gate is now pipeline-integrated; rows written after this date record the iteration count actually executed.

Case study 2 — cheap wins first: build flags and allocators

The playbook's ordering principle (SPEC §3.3, CLAUDE.md non-negotiable #3): build flags → LTO → PGO → allocator swap → then code changes. The cheapest classes are tried first because they carry the lowest cost-per-attempt and the lowest risk. Two of them produced Phase-0 results that frame the whole project's discipline.

Class 1 — link-time optimization (tokei) — ACCEPTED

phase0-tokei-003: set lto = true (fat LTO, up from thin) and codegen-units = 1 in tokei's release profile.

• Speedup: 1.104 median, 95% CI [1.085, 1.120] vs the pristine thin-LTO baseline.
• Byte-identical golden replay; no source change, so equivalence is by construction (same code, different codegen). Auto-accept tier.
• A pure build-config win worth +10% — the cheapest possible class, and on this target the single largest individual increment.

The lesson the ledger enforces: you do not go hand-optimize a hot loop before you have tried turning on the optimizer's own switches.

Class 2 — allocator swap (comrak) — NEEDS-HUMAN-REVIEW

phase0-comrak-002: swap the global allocator to mimalloc.

• Speedup: 1.046 median, 95% CI [1.033, 1.058] — a real, CI-clean +4.6% on comrak's Pro Git render.
• Byte-identical golden replay. And yet the verdict is not accepted — it is needs-human-review.

Why: an allocator swap changes the process's memory behavior globally. It is exactly the class of change SPEC §8 routes to a human — not because this instance is wrong, but because "the benchmark got faster" is not sufficient evidence that a global allocator change is safe under every workload, fragmentation pattern, and concurrency scenario the target will meet in production. The pipeline measured a real win and then refused to auto-ship it, tagging it for a human ruling instead.

Human ruling (results/rulings/phase0-comrak-002.md): ACCEPTED, scoped to the comrak CLI single-stream markdown-render workload — a real win for the measured use, deliberately not a blanket "always use mimalloc" mandate. The tier forced the judgment; the judgment is bounded and written down. The ledger row stays needs-human-review (append-only records what the pipeline decided); the ruling is the human layer on top.

What the pair demonstrates

The cheap classes are not lesser — class 1 produced this project's biggest single win. But "cheap to try" is not "safe to auto-accept." LTO is deterministic codegen: accept it. An allocator swap is a global behavioral change: measure it, prove the speedup, and still hand it to a human. The tier is decided by the nature of the change, never by the size of the number — the same rule that later caught a leaking teardown patch the bench loved (case study 3).

Case study 3 — the caught false-accept (comrak, class 3)

This is the most important case study in the project, because it is the one where the machine got it wrong and the process caught it.

What happened

phase2-comrak-010: the unattended agent, reading comrak's cache-sim profile, saw that AST-arena teardown at process exit — drop_in_place::<Vec<Node<RefCell<Ast>>>> plus its share of the glibc free path — cost real instructions and cache misses in a run-to-completion CLI. Its hypothesis: comrak already ends in process::exit, so the OS reclaims memory in bulk anyway; wrap the arena in std::mem::ManuallyDrop to skip the per-node drop walk, and add a std::hint::black_box(&arena) before the scope closes so the leak stays reachable and a leak checker won't flag it.

The pipeline ran it:

• Upstream tests: PASS
• Golden replay: byte-identical
• Interleaved A/B: +10.7% median, 95% CI [+7.9%, +12.1%]
• Verdict: accepted

A clean, large, CI-significant win. And it was wrong.

The audit overturned it

Phase 2 runs a 100% human audit of every accepted win (SPEC §10). The audit ran the sanitizers the pipeline had not: LeakSanitizer flagged a 384-byte arena chunk as unreachable at exit. The agent's black_box(&arena) keeps the arena live only until its scope closes — but process::exit runs after that scope, so at the moment of exit the arena is genuinely unreachable. A real leak. The agent's own hypothesis had even named the risk ("if the sanitizer still flags the leak, rejected-gates is the honest outcome") — the pipeline simply had no sanitizer gate to trip, because in Phase 2 ASan/LSan were a per-attempt manual step.

Called per SPEC §10: a pipeline false accept, caught by audit before anything shipped. The win was never banked. Shipped false accepts across the entire project remain zero.

The fix made the machine enforce the tier

The corrective action was not "be more careful." It was structural: verdict now builds the patched tree with ASan+LSan (ASan+UBSan for C/C++) and runs the pinned workload on every would-be accept, capping any sanitizer-flagged win at needs-human-review. The tier rule SPEC §8 always specified is now enforced by the machine, not the auditor. The very next Rust accept (phase2-tokei-008) and the first C accept (phase3-cjson-002) both passed through this automated gate.

The 10k-input differential fuzz on the overturned patch found 0 divergences — the output equivalence was genuinely real. Only the teardown-leak tier was wrong. That is the subtle failure mode the layered gates exist for: a change can be behaviorally equivalent on output and still unacceptable on a dimension the bench and the golden replay can't see.

What it demonstrates

The product is not "an agent that finds speedups." The product is trust — and trust is demonstrated most by the case where the optimistic path failed and the structural controls held. An agent graded only by accept-rate would have banked this win. The ledger is append-only, so the accepted row still stands, immutable — with this document as the permanent corrective record beside it.

Case study 4 — cJSON: the cross-language proof (C, classes 3 + 5)

Target: cJSON v1.7.19 — an ultralightweight C JSON parser/printer, MIT. Workload: a 5.6 MB deterministic JSON corpus (20k records: nested objects/arrays, escaped/unicode strings, integers, dense floats), parsed and serialized, single-threaded, CPU-pinned. Result: the first verified win on a C/C++ target — proof the trust layer, the gates, and the ROI story carry across languages, not just across Rust crates.

Getting C through the same pipeline

Nothing in the trust layer assumes cargo anymore. cJSON is "just a target.toml" pointing at clang build commands, with:

• a hermetic Unity/CTest suite (19 tests: parse_number, parse_string, print_*, misc_tests, …) as the upstream-test gate;
• a trust-layer harness driver (targets/cjson/harness.c) compiled with the patched cJSON.c — the agent can patch the parser but not the measurement rig (it lives outside the workspace allowlist);
• ASan + UBSan as the sanitizer gate (the C analogue of the Rust nightly ASan path);
• its own A/A calibration: 0/20 false positives, 20/20 injected-slowdown detections — the C workload is a trusted anchor before any verdict runs.

The profile

callgrind cache-sim put ~22% of instructions in glibc float I/O — __printf_fp (number serialization) and strtod/str_to_mpn (number parsing) — plus parse_value, parse_string, and heavy malloc/free churn. A number-dense JSON document spends most of its time turning numbers into strings and back.

Attempt 1 — the honest rejection (phase3-cjson-001, class 3)

parse_number heap-allocated a temporary NUL-terminated buffer (malloc + memcpy + free) for every number, just to hand a clean string to strtod. With ~100k+ numbers in the corpus that is a large share of the allocation churn. The fix: a 64-byte stack buffer for the common short-number case, heap only for pathologically long number strings — byte-identical (strtod sees the same bytes).

• Speedup: 1.012 median, 95% CI [1.004, 1.022] → rejected-bench.

A real mechanism, a real +1.2%, but the CI lower bound (+0.4%) does not clear the 2% bar — because the corpus's wall-clock is dominated by float formatting, not by malloc. The statistical bar rejected a genuine win, exactly as designed. It is in the ledger.

Attempt 2 — the accepted win (phase3-cjson-002, classes 5 + 3)

Adding the class-5 lever: print_number verified every fractional number's "%1.15g" round-trip by calling sscanf("%lg") — a full strtod plus format-string interpretation, per number. Replacing it with a direct strtod is the identical parse with none of the format-string overhead. Combined with the stack buffer:

• Speedup: 1.089 median, 95% CI [1.075, 1.100] → accepted.
• Upstream Unity suite: PASS. Golden replay: byte-identical.
• ASan + UBSan: clean — through the machine-enforced sanitizer gate.
• 10,000-input differential fuzz (scripts/diff-fuzz-cjson.py), pristine vs candidate, 0 divergences.

A +8.9% verified win on a C target, banked.

A finding worth keeping

The first cJSON accept nearly wasn't recorded: the ASan+UBSan build failed to link because the clang sanitizer runtime (libclang_rt.asan) was not installed, and sanitizer_check treated that as a hard error — crashing the verdict and losing the measured bench. The fix hardened the gate: an infrastructure build failure (missing runtime, broken toolchain) now caps the accept at needs-human-review and logs the failure for the operator, rather than discarding a real result. "Cannot verify" is a review tier, not a crash.

What it demonstrates

The progression — a real win rejected for being too small, then a larger win accepted only after golden replay, fuzzing, and sanitizers all held — is the whole method in miniature, now shown on C. Nothing about the trust story is Rust-specific. The cheapest safe lever (skip a per-number allocation) wasn't enough; the algorithmic lever (stop doing a redundant heavyweight parse) was — and the pipeline proved it the same way it proves everything else.

Note (2026-07-13). The 10,000-input differential fuzz cited above ran out-of-band (scripts/diff-fuzz-cjson.py, per-attempt), not inside the verdict pipeline — so the phase3-cjson-002 ledger row records fuzz_iters=0 in its machine record. The fuzz gate is now pipeline-integrated; rows written after this date record the iteration count actually executed.

Phase 4 — services & scale

Exit criteria (SPEC §5): one real service workload optimized under replayed traffic; report generated mechanically. Both met.

Phase 4 takes the trust story off the batch stopwatch and onto a running service: the same verified optimization, measured as a latency delta under replayed load, then turned into a dollar figure by a generator that reads only the ledger and the calibrated bench — no hand-typed numbers.

What was built

• Service-mode latency bench (crates/bench-runner/src/service.rs, SPEC §3.1 mode c): a coordinated-omission-correct, open-loop, fixed-rate load generator. Each request's latency is measured from its intended send time, so when the single worker falls behind, the requests queued behind it carry the full delay a real client would see — the exact error closed-loop tools hide. Interleaved A/B of two server binaries, exact percentiles (bounded sample; no HdrHistogram bucketing error), bootstrap p50/p99 CIs through the same ratio-CI machine the wall-clock bench uses.
• A real service target: targets/cjson/service.c, a minimal HTTP server wrapping the (patched) cJSON.c — trust-layer code outside the workspace allowlist, so a patch changes the parser under load but not the measurement rig.
• Service-mode calibration (bench-runner service-calibrate): the latency measurement is a trusted anchor only after it passes A/A (0/10 false positives) and injected-5%-latency-regression detection (10/10), recorded in results/calibration/cjson-service-aa.json.
• Mechanical ROI report (crates/report binary, SPEC §9): reads a ledger row + the service-latency JSON and renders throughput→cores→dollars and latency percentiles, every figure with its 95% CI, the pricing source, and the methodology printed inline.

The result (phase4-cjson-service, ledger)

The accepted batch win phase3-cjson-002 (+8.85% throughput) was measured as a service under a 150 rps-target coordinated-omission-correct replay, baseline (pristine cJSON v1.7.19) vs candidate (the banked win), 20 interleaved rounds, server pinned, 0 drops — the ledger row records sessions=20, rate_rps=150, drop_rate=0.0; the total request count was not machine-recorded (see errata below):

The p50 win is the claim: +6.2% service-latency reduction under load, CI [+5.8%, +7.2%] — the batch parse speedup translating to lower per-request latency. It is a touch smaller than the +8.85% batch number because under load the latency also carries fixed connection and HTTP overhead the parse optimization doesn't touch — an honest dilution, not a measurement error.

The p99 was not claimed. Its CI is [0.07, 4.97] — uselessly wide, because tail latency on a single-worker loopback service is dominated by rare OS-scheduling spikes that swamp the ~0.2 ms parse difference. The harness resolved this correctly: p99 verdict: rejected-bench. A number the measurement can't pin down does not get reported as a win, even though the median points the right way. That refusal is the whole project in one line.

The ROI, generated mechanically

just report phase4-cjson-service --service-json results/phase4/cjson-service.json → cjson-service-roi.md. (Note, 2026-07-13: the raw cjson-service.json was not preserved — see errata below — so this command is not reproducible as written; the generated report and the ledger row are the surviving evidence.) On a 500-core fleet at $0.04/core-hour, the +6.2% p50 win returns 27.5 cores / $9,621 per year at the CI lower bound (median 29.4 cores / $10,290) — and the report quotes the lower bound, because the point estimate is not a promise. Every figure carries its CI, its workload, and the methodology that produced it, so the number survives hostile review.

Exit criteria (SPEC §5) — met

• One real service workload optimized under replayed traffic: the cJSON HTTP service under CO-correct open-loop replay; the verified win shows a CI-significant +6.2% p50 latency improvement.
• Report generated mechanically: the ROI report is produced from the ledger row and the calibrated service JSON with no hand-editing; throughput→cores→dollars and latency percentiles, CIs and methodology inline.

Errata (2026-07-13)

• "40,000 requests" retracted. This report originally stated the replay comprised 40,000 requests. The phase4-cjson-service ledger row records only sessions=20, rate_rps=150, and drop_rate=0.0 — the per-run request count was never machine-recorded, so the 40,000 figure (an arithmetic reconstruction, not evidence) is withdrawn. The prose above now states the ledger-backed facts only. The claimed result is unaffected: p50 latency +6.2%, 95% CI [+5.8%, +7.2%], under the 150 rps-target CO-correct replay, 20 interleaved rounds, 0 drops.
• results/phase4/cjson-service.json was not preserved. The .gitignore rule results/**/*.json swept the raw service-latency JSON that the report command consumes, so it no longer exists in the repository. A !results/phase4/*.json carve-out has been added so future service-mode evidence JSONs are committed alongside the calibration JSONs.

Research forks: kernel lane & learned policy

The core loop was complete at Phase 4 — profile → hypothesize → patch → gated-verdict → ROI, running unattended across Rust and C, batch and service, with zero shipped false accepts. SPEC §13 named two frontier ideas that the roadmap deliberately excluded, precisely because they change the shape of the work rather than extend the loop. Phase 5 built both, as stress tests with one question each: does the trust machinery still hold?

Both times it does. Neither fork relaxes a non-negotiable; each routes its headline result to needs-human-review, and the learned prior is advisory only. What follows is the argument for each.

The kernel lane: when byte-identical is the wrong gate

Every target through Phase 4 was gated by byte-identical golden replay — the optimized output must match the baseline bit for bit. For a parser or an HTTP service that is exactly right. For a numerical kernel it is wrong, and the kernel lane is the case that proves it.

targets/matmul/kernel.c is a single-precision matrix multiply in two forms:

• matmul_ref — naive i,j,k, one sequential accumulator. The oracle.
• matmul_opt — transpose B so the inner reduction walks contiguous memory (the cache win), plus eight independent accumulators tree-combined at the end (the ILP win, breaking the float-add dependency chain). The second lever reorders the reduction, so the last few ULPs differ by construction. Bit-identical output is impossible.

The demonstration (sh scripts/kernel-lane-demo.sh 512) walks the argument:

1. byte-identical FAILs — 244,901 of 262,144 values differ. Under the Phase 0–4 gate this correct 3× speedup would be rejected.
2. FP-tolerance is EQUIVALENT — the target declares mode = "fp-tolerance", abs = 1e-4, rel = 1e-3; every numeric token agrees within abs + rel·|ref|.
3. a wrong result is still caught — perturbing one element by +0.5 is REJECTED by the same tolerance. Loose enough for last-ULP reordering, tight enough for a real bug.
4. measured with the same machinery — interleaved A/B, bootstrap 95% CI: median 3.23×, CI [3.16, 3.26].

The new rule lives in one place — EquivalencePolicy::compare in diff-test. Byte-identical stays exact; fp-tolerance tokenizes, compares numeric tokens within tolerance, handles NaN explicitly, and keeps non-numeric tokens and token counts exact. The fp-compare binary applies a target's equivalence.toml to two output files from the command line.

The verdict is needs-human-review. Using the FP-tolerance tier at all is a human-review signal (SPEC §8): a person confirms the declared tolerance is defensible for the workload, exactly as with the mimalloc allocator swap. The machine measures and gates; the human ratifies the tolerance.

Scope, stated honestly. The FP-tolerance mechanism is in the gate pipeline — diff-test honors an equivalence.toml policy (it needs a committed [corpus].golden_reference, and fails closed without one), and verdict auto-routes any fp-tolerance run to needs-human-review. But matmul is not wired into that pipeline: it has an equivalence.toml and no target.toml, which is the file diff-test/verdict load a target from. So the kernel lane is demonstrated script-driven (scripts/kernel-lane-demo.sh, using the fp-compare binary and bench-runner compare directly), and the phase5-matmul-opt ledger row was recorded outside the automated verdict path. Promoting matmul to a real pipeline target is open work — the policy is closed; that target's wiring is not.

This is the GPU lane in miniature. The environment has no GPU, so the GPU fork is shown on the CPU as what it actually is — the same trust machinery with a different timer. A Triton or CUDA kernel versus a reference kernel is the identical shape: reference-kernel differential testing within a tolerance, interleaved timing with a bootstrap CI, a ledger row routed to human review. Only the profiler and the bench clock become kernel-time-aware, and the hardware changes. The correctness story, the statistics, and the audit trail are already built and exercised here on real code.

The learned policy: the ledger as a training set

The playbook orders optimization classes cheapest-first — a fixed prior. But the ledger is now dozens of rows of (class, target) → verdict. That is a dataset, and it can turn the fixed ordering into a learned one.

crates/policy ranks each class by the Wilson score lower bound of its observed shippable-win rate. The Wilson lower bound is the same instinct as the bootstrap CI lower bound the bench uses to accept a speedup: be pessimistic about thin evidence. A class that went 1-for-1 should not outrank one that went 3-for-16 — and it does not, because the lower bound penalizes small samples automatically. Ties break untried-before-failed (an unexplored class keeps its cheapest-first prior; a class with evidence against it sinks), then cheapest-first.

Two guards keep it honest, both printed in the output footer:

• A "win" counts only a machine-sanitizer-verified accept. The overturned comrak-010 false-accept and every pre-sanitizer-gate accept are excluded — the policy trusts the ledger's machine record, not the audit narrative that caught them.
• The tier-gated mimalloc win shows as held for human review — a third category, neither a loss nor a shippable win.

On the current 34-row ledger the ranking recommends algorithmic first (best evidence, Wilson lb 0.066), then the next-ranked class whose profile preconditions match. It is advisory: the equivalence and significance gates still decide every verdict. This is the MLGO / AlphaDev / LLM-Compiler direction at the scale the ledger currently supports — a ranked prior that sharpens as the dataset grows and never overrides a gate.

Run it with cargo run -p policy (add --target <name> to scope the evidence to one target).

What the forks settle

• A learned prior can be extracted from the ledger without weakening a single gate — it only reorders what to try first.
• A numerically-reordered optimization — the class byte-identical replay cannot handle — slots into the same harness by swapping the equivalence policy, with the correctness bar preserved and the tolerance itself put under human review.

The full write-up, with reproduction commands and the raw tool output, is in results/phase5/.

Consume it: the peltier skill

Everything this site describes — equivalence before timing, A/A calibration, interleaved A/B, the CI-lower-bound rule — is packaged as an Agent Skill you can drop into any repo and drive from any agent harness that reads the SKILL.md standard: Claude Code, OpenAI Codex, GitHub Copilot, VS Code, Cursor, Gemini CLI, opencode, Goose, Amp, Zed, Factory, ZeroClaw, and others.

The skill does not reimplement anything. It drives the same bench-runner binary the case studies used, from a peltier checkout you point it at. That is deliberate: a second copy of the statistics would be a second thing that can lie. The corollary is stated in the skill itself — if it cannot find the trust layer, or the host cannot run it, it refuses to produce a number rather than fall back to a hand-rolled timing loop. A refusal ends the claim, not the work.

Install

# from your repo, with a peltier checkout at /path/to/peltier.io
sh /path/to/peltier.io/scripts/install-skill.sh .
export PELTIER_HOME=/path/to/peltier.io

One command stamps both discovery paths, because harnesses look in different places:

Two copies is a drift risk, so this repo treats it the way it treats every other claim: CI enforces the copies stay byte-identical (diff -r in the skill-preflight job), and the same job checks both against the Agent Skills spec rules (name/directory match, description and body limits) on every pull request.

ZeroClaw: the script-blocking case

ZeroClaw's skill security policy blocks script files inside skills by default (skills.allow_scripts is off) — its auditor rejects the full skill on sight. The installer emits a variant for it:

sh /path/to/peltier.io/scripts/install-skill.sh --zeroclaw-variant /tmp/peltier-skill
zeroclaw skills install /tmp/peltier-skill

The variant simply omits scripts/. Nothing is lost and nothing is forked: preflight.sh still exists exactly once — in the peltier checkout the skill already requires — and the skill's instructions run it from there. Transcribing or re-implementing the preflight logic inline is explicitly named a refusal condition in the skill, so a harness's restrictions cannot pressure an agent into forking the trust layer.

What the skill enforces

Two modes. verify works anywhere, on any two shell commands, in any language; attempt drives the full in-repo loop this site's case studies document. In verify mode, the gate order is the content:

1. Preflight — find the trust layer, build the real bench-runner, refuse on unsupported hosts. STATUS=ok means the harness can run, not that the host can measure — only calibration decides that.
2. Equivalence before timing — a faster program that computes something different is a bug with good benchmarks, not an optimization. Comparing two empty outputs is a failed gate, not a passed one.
3. A/A calibration — the same command on both sides must yield a null verdict. A harness that "finds" a speedup between a binary and itself is measuring noise, and every number it produces afterwards is fiction.
4. Interleaved A/B, bootstrap CI — and the claim is the CI lower bound, never the median. A median of 1.088 with 95% CI [1.074, 1.100] (the shipped cJSON batch win, on its 5.6 MB / 20k-record parse+serialize workload) supports "≥7.4% faster" — not "8.8% faster".
5. Risk routing — concurrency, unsafe, and floating-point-reordering changes are needs-human-review regardless of the numbers. In verify mode the agent is instructed to be the classifier, because the standalone harness only ever emits accepted/rejected-bench.

The platform truth

Measurement requires a Linux/POSIX host — the same requirement as the rest of the pipeline. On Windows (or with no checkout reachable) the skill refuses and says why. The refusal itself is cross-OS: preflight ships as both preflight.sh (POSIX) and preflight.ps1 (Windows PowerShell 5.1 / pwsh), so a Windows-native harness with no POSIX shell still gets a structured STATUS=refuse instead of a launch error it might misread as license to hand-roll a timing loop. Two implementations of one contract is a drift risk, so CI byte-diffs their output on every path (resolve, refuse, PELTIER_HOME) on the same host, and a Windows runner asserts the native refusal on every pull request. That is asserted, not assumed: on every pull request, CI runs the preflight's refusal path (skill copied outside any checkout) and its resolve paths (in-checkout and via PELTIER_HOME from a foreign repo), alongside a live plumbing run of the harness it drives.

The skill's own files are the reference: SKILL.md, references/verify.md, references/attempt.md.

Reproduce it yourself

Nothing here is a screenshot. Every number regenerates from a pinned corpus through the same commands.

Prerequisites

• A Linux/POSIX host. The pipeline is Unix-only at runtime: harnessd (Unix sockets, setsid), service mode, the isolation wrapper, and every sh-based gate assume it. Windows compiles the workspace and runs the portable unit tests, but cannot run the pipeline.
• Rust (stable) + just; clang/cmake for the C target; valgrind for callgrind profiling. Nightly Rust for the sanitizer gate.
• Targets are vendored fresh (their workspace/ is gitignored); each targets/<name>/target.toml has the [fetch] recipe.

The commands

just build / test / lint       # trust-layer workspace
just aa [cmd]                  # A/A self-test — must be a null verdict
just calibrate <cmd> <out>     # A/A false-positive + injection detection
just gates <target>            # corpus pin + upstream tests + golden replay
just verdict <t> <bin> ...     # gates + bench vs pristine baseline + ledger row
just report <run-id>           # ROI report from a ledger row

just gates does not fuzz. Differential fuzz differs the candidate against a pristine baseline, and only just verdict builds one — so on the gates flow the fuzz layer reports Skipped, with the reason recorded, rather than comparing the candidate against itself. Use gates as a fast equivalence check; the fuzz gate (and the sanitizer lanes) run on the accept path, where an accept is impossible without a passed fuzz gate.

Service mode (Phase 4):

just service <baseline-bin> <candidate-bin> <doc>     # interleaved p50/p99 CIs
just service-calibrate <server> <doc> <out>           # latency A/A + injection
just report <run-id> --service-json <json>            # latency + ROI report

The agent loop, behind OS isolation:

just isolation-check                    # 19 boundary checks, both modes
scripts/agent-isolated.sh <target> <run-id>   # one unattended attempt, confined

Auditing a result

Every accepted win can be re-audited independently:

python3 scripts/audit-attempt.py <run-id>        # mechanical re-check
python3 scripts/diff-fuzz-<target>.py <dir> 10000 <baseline-bin> <candidate-bin>

The ledger itself is the record: results/ledger.sqlite, append-only, every attempt (win or loss) with its hypothesis, gates, bench evidence, and verdict.

Use it on your own code

The whole discipline is installable into any repo as an Agent Skill, driven from any SKILL.md-reading harness — see Consume it: the peltier skill.

Explain: why a row won or lost

Verdicts say what. just explain <run-id> says why — an advisory diagnosis of one ledger row, built so it can never contaminate the thing it explains:

• Ledger-only inputs, deterministic. It reads exactly one row and nothing else — no config, no clock, no re-benching. The same row produces byte-identical text, so a regenerated explanation is checkable.
• Strictly off the accept path. It runs after the verdict is written; nothing it says feeds gates, the bench, or the ledger.
• Record vs. inference is explicit. Restatements of the machine record are printed plain; anything beyond that is prefixed inference: — so an explanation can never be quoted as if the machine measured it.
• The narrative must agree with the verdict. An accepted row is described by its CI lower bound (the defensible claim). A rejected bench is classified honestly: measured regression, null result, or real-but-below-bar. A needs-human-review row names the recorded cap and says plainly that the numbers were not the deciding factor.

Real output, real rows

The shipped cJSON win (phase3-cjson-002):

verdict (machine record): accepted
bench: baseline median 0.4263s, candidate median 0.3916s — speedup 1.0885 (+8.8%), 95% CI [1.0745, 1.1001]
workload: 5.6 MB synthetic JSON (20k records: nested objects/arrays, escaped/unicode strings, ints, dense floats), parsed+serialized 2x, single thread
inference: the defensible claim is the lower bound — "at least +7.4%" on this workload, not the median +8.8%

The caught false-accept (phase2-comrak-010) — explain flags it instead of laundering it:

inference: the defensible claim is the lower bound — "at least +7.9%" on this workload, not the median +10.7%
CAUTION: accepted with sanitizers_clean=false — this row predates the machine-enforced
sanitizer lane; under current doctrine it could not be machine-accepted. See results/rulings/

The kernel-lane row (phase5-matmul-opt, needs-human-review):

verdict (machine record): needs-human-review
bench: … speedup 3.2301 (+223.0%), 95% CI [3.1600, 3.2594]
inference: the numbers were not the cap — the review routing below decided this verdict

That last line is the point: a 3.2× speedup with a tight interval still does not argue with the review routing. Explanations restate the record and reason from it — they do not re-litigate it.

Why this exists

The loop's compounding value is not any single verdict — it is that the next hypothesis is better than the last. A rejection that says only rejected-bench teaches nothing; one that says "real improvement, below the ship bar" or "indistinguishable from no change — a null result, not a small win" or "a measured regression, report it as one" points at three different next moves. Explain is that teaching layer, kept deliberately outside the measurement it learns from — see SPEC §3.7 for the invariants, including the one that keeps it out of the agent-facing surface: better hypotheses must never become a feedback channel into the accept path.

Rows written from 2026-07-20 on also record the accept bar in force for that run (accept_threshold in the env fingerprint), so an explanation never has to guess a historical threshold from today's config — for older rows it says so explicitly instead.

Bench infrastructure — what Phase 1 still needs from hardware

The trust layer is code-complete for Phase 1 (calibration, gates, verdict, agent IPC), but SPEC-grade bench numbers need hardware this repo's CI cannot provide. Status by requirement:

OS-level agent isolation (SPEC §10) — shipped

scripts/agent-isolated.sh launches one unattended attempt with the agent process tree behind an OS boundary; just isolation-check verifies the boundary from the agent's side (19 negative/positive checks, both modes). harnessd runs OUTSIDE the boundary as the trusted uid, serving its seven ops on a Unix socket (harnessd --socket) — the agent's only write path into the repo.

Two modes (HOTPATH_ISOLATION):

• mountns (default in this container). The agent tree runs uid 0 inside a private mount namespace with the repo bind-mounted read-only, CAP_SYS_ADMIN/CAP_SYS_PTRACE dropped from the bounding set. Verified refused: repo writes (EROFS), forged ledger INSERT, trust-binary replacement, mount -o remount,rw, and userns re-entry to undo the mount. Chosen because the nested CLI's auth is bound to the parent session's uid — no credentials are copied anywhere.
• user (bench-metal shape). The agent runs as unprivileged hpagent; the repo is root-owned with no world-writable paths (the launcher refuses to start otherwise). Verified refused: all of the above plus running the verdict binary directly. Requires the agent user to bring its own API credentials; a parent session's root-owned auth is deliberately not copied across the boundary.

Residual (both modes, this container): the agent tree can still read the repo (by design — it needs source), reach the network through the container's egress proxy, and signal same-uid processes in mountns mode. The bench-metal user mode plus no-net target containers remains the production shape.

The bench-metal box (human setup, ~15 min + account)

Any dedicated bare-metal server works; the budget default is a Hetzner AX-line dedicated server (other options: OVH/Equinix Metal/AWS *.metal). Setup:

1. Ubuntu LTS; apt install linux-tools-$(uname -r) docker.io git build-essential.
2. Measurement discipline: cpupower frequency-set -g performance; disable SMT (echo off > /sys/devices/system/cpu/smt/control) and turbo; reserve two cores via isolcpus= kernel arg; set pin_prefix in config/accept.toml to those cores.
3. Register as a GitHub self-hosted runner with label bench-metal (repo Settings → Actions → Runners). Run the runner as an unprivileged user; make crates/, config/, corpora/ owned by a different user (this is the separate-uid enforcement from SPEC §10).
4. First act: just calibrate <real workload> results/calibration/bench-metal-01.json — the box is only trusted after its own A/A + injection evidence is committed. Then flip verdict bench jobs to runs-on: bench-metal.

Until that box exists, verdict-grade benches keep running wherever a human has verified calibration evidence first (as done for this container in results/calibration/).

Roadmap & scope

Done — the core loop, end to end

The profile → hypothesize → patch → gated-verdict → ROI loop runs unattended, OS-isolated, across Rust and C, batch and service, from a pinned corpus to a dollar figure — with zero shipped false accepts across all phases, and the two pipeline over-accepts both caught by audit and turned into permanent gates.

Phase 5 — the research forks (SPEC §13)

Deliberately out of the core roadmap, these are the two hardest cases the spec named — both now built and exercised, both routed to human review. Full write-up: Research forks.

• Kernel lane — done as a CPU demonstration. A matmul optimization that reorders floating-point accumulation (transpose-B + eight-accumulator ILP) runs 3.23× faster and is impossible to gate byte-identically. It slots into the harness by swapping the equivalence policy for a declared FP-tolerance (abs + rel·|ref|) — which still catches a genuine wrong result — and routes to needs-human-review because using the tolerance tier at all is a §8 signal. This is the GPU lane in miniature: reference-kernel differential testing, interleaved timing, ledger row. Only the timer and the hardware change; the actual GPU run needs GPU hardware, absent here.
• Learned optimization policies — done. crates/policy reads the ledger and ranks optimization classes by the Wilson lower bound of their shippable-win rate, turning the fixed cheapest-first ordering into a learned prior (advisory only — the gates still decide). Prior art: MLGO, AlphaDev, Meta's LLM Compiler.

The real target

The end goal is not vendored OSS case studies — it is a real production system. The same gates that matter for a JSON parser matter far more for a live CMS: an equivalence failure there is a customer-facing bug, and the network-isolated verdict path and append-only audit trail are what would make it safe to point an optimization agent at code that a business depends on.

Known gaps (what would have to close first)

Honesty about the remaining edges, since the product is trust:

• Full-container isolation. The verdict pipeline is network-isolated on the agent path (scripts/no-net.sh); the seccomp-restricted bench container of SPEC §10 is not built, and a human running just verdict directly is unwrapped.
• Test-suite pins. The verify-or-refuse machinery exists, but no target ships a TESTSUITE.sha256 yet — so "the agent cannot edit the upstream tests" is currently a mechanism awaiting its pins.
• TSan. The lane runs on the accept path only when a target declares [build].tsan; none does today, so there is no TSan coverage. Class-7 (concurrency) work depends on that opt-in landing.
• Kernel lane wiring. matmul demonstrates FP-tolerance equivalence script-driven; it is not yet a target.toml pipeline target.