Before and after
Where the three repos meet: a task from rails/ai-evals, seeded into basecamp/writebook, graded by the rails/lemans verifier. Three worked examples from the evidence directory. Sources: docs/PROGRAMME_MAP.html §3, docs/SCOREBOARD.md §2b/§2e/§2g, benchmarks/…/<run>.diff.
Suite22 ai-evals tasks against Writebook. 21 are api-knowledge; hello-world proves the pipeline. 4 touched, 18 untouched.
TaskFour files: instruction.md (frontmatter carries rails_anchor, stripped), environment.patch (seeded defect), solution.patch (oracle), verification_test.rb (hidden).
ArmOne run = task × one intervention. The only variable. Named Bphase4, Bprobe, KBrep1__a3…
PipelineA ▸ B ▸ C, guard watching. The thing being built. Everything above and below this row is upstream's.
Gradinglemans' verifier, mirrored by factory_harness.sh: preverify app suite ▸ restore test/ bin/ config/environments/test.rb ▸ run hidden test. Exit status is the reward.
sup-legacy-conversions — same seven lines, three outcomes
7 sites · 6 files · 1 trap · local Qwen 27B 4-bit both columns
The seeded defect is a Rails 7.0 rename left unapplied: to_s(:format) became to_fs(:format) and to_default_s was removed. The task describes it the way a bug report would — the nightly archive job crashed — and never names the API. Both runs below had a correct work order and made their edits in 13 calls at 0% duplication. One had been handed 1.6 KB of verified upgrade notes in Phase A. The other had not.
| Writebook file | Seeded (before) | Bphase4, no sheet | Bprobe, with sheet = oracle |
|---|
books/reports_controller.rb:16 | updated_at.to_s(:number) | ✓ strftime("%Y%m%d%H%M%S") | ✓ updated_at.to_fs(:number) |
books/reports_helper.rb:3 | to_date.to_s(:long) | ✓ strftime("%B %d, %Y") | ✓ to_date.to_fs(:long) |
report_digest_mailer/weekly.text.erb:6 | to_date.to_s(:short) | ✗ %b %d — verifier wants %d %b | ✓ to_date.to_fs(:short) |
book/reportable.rb:5 | length.to_s(:delimited) | ✗ number_with_delimiter — a view helper, errors in a model | ✓ length.to_fs(:delimited) |
publishing/report_card.rb:24 | to_a.to_s(:db) | ✗ map(&:title).join(", ") — wants ids | ✓ to_a.to_fs(:db) |
publishing/report_card.rb:28 | format ? value.to_s(format) : value.to_s | ✗ strftime without the time component | ✓ value.to_fs(format), else-branch untouched |
report_archive_job.rb:12 | last_edited_at&.to_default_s | ✗ strftime("… %z") — wants %Z | ✓ last_edited_at&.to_s |
| Hidden verifier | — | 4 failures + 2 errors · FAIL | 0 failures, 0 errors · PASS · diff byte-identical to solution.patch |
Bprobe diff, app/jobs/report_archive_job.rb
private
def header(book)
- "# archived #{book.last_edited_at&.to_default_s}"
+ "# archived #{book.last_edited_at&.to_s}"
end
end
Verified runtime output, probe against real records, rolled back
report_stamp : September 07, 2026
filename : shelf-report-20260907072256.txt
report card : Updated: 07 Sep 07:22
Covers: 1,2,3
archive header : # archived 2026-09-07 07:22:56 UTC
digest line : * Probe Book, last edited 07 Sep
Scope was never the problem in the left column: Bphase4 found all seven sites. It believed to_default_s had no Rails 8.2 replacement and hand-rolled formats; five of seven missed the verifier's exact strings. The right column's survey found only six sites and misfiled the ternary as do-not-touch; Phase B changed it correctly anyway. One fact — renamed, not removed — is the entire difference.
as-purge-embedded-images — one line, and the line it didn't write
Opus via Claude Code · matched pair · plain vs +11,704 chars of RAG · both FAIL
The controlled RAG test. Two fresh Opus runs, identical harness, only the instruction varied: one plain, one with the retriever's top chunks appended, purge_later at rank 0. Both arms produced the same one-line change in app/models/page.rb:
- dependent: :destroy
+ dependent: :purge_later
Both failed the same hidden test — a page revised after its image went in frees the revision's image too. Both runs saw the second leak. Both wrote it up under uncertainties and declined to fix it:
"I judged this a pre-existing upstream design quirk rather than the injected bug… Flagging it in case the hidden tests exercise the edit-then-destroy path."
The contaminated run — the one that received rails_anchor: purge_later because the frontmatter had not been stripped — found both bugs and passed. Being handed part one appears to have freed it to look further. That is a scoping judgement, not a knowledge gap, and it is why eleven kilobytes of correct documentation changed nothing. The result is the cleanest null in the programme and the reason the RAG line was closed.
ac-throttle-search — pass by hand-rolling
local Qwen 27B 4-bit · 3 of 3 with the stopping rule · Rails API recall: 0 of 3 · the verifier rewards behaviour
The task turns on Rails 8's built-in rate_limit. No local run ever found it — and the run you're about to see didn't either. It passed anyway, which is the point. Run C left this on disk:
- M
app/controllers/books/searches_controller.rb
- ??
app/controllers/concerns/rate_limited.rb
- ??
lib/rate_limiter.rb
A concern and a library class where the oracle adds one macro to the controller. The behavioural verifier accepted it, as lemans' methodology says it should: the checks test behaviour, not implementation, and API recall is scored separately from passing. It is the mirror image of example 1 — there, a behavioural verifier would have forgiven the strftime hand-rolls too; an exact-output verifier did not. Whether thin Rails 8 knowledge is fatal depends on what the test asserts.
