Snapshot of the programme as of 8 September 2026, 18:25 AEST, reconciled against the programme record on 9 September. The live record lives in the model-ops repo (mrichardsJBOIT/model-ops, private).

Update — 10 September · the overnight test, and the pivot. The 9–10 Sep night answered the question this page ended on — and the answer changed the plan:
  • Held-out generalisation: 1 of 8. Eight unseen tasks through the blind topic-keyed sheet, first attempt: 1 passed (2 of 12 over both rounds, both ar-compact-positions — a one-token typo). Only one survey stopped on its own; every other plan came from forced emission.
  • The sheet is real, and it is not the constraint. The coverage check handed the sheet's own facts to three known misses — 0 of 3. The one pass came from a task whose fact the sheet did not carry.
  • Cloud calibration, same harness: deepseek-v4-flash-0731, 21 tasks, 10 of 20 graded, $3.20 of the US$10 cap, 20–25 min a run clean. Matches its own leaderboard profile — no lift, no harm, at a quarter of the time and cost.
  • Misses classified: 13 of 17 were process (plan / behaviour / scope); 2 were knowledge — both general Active Record facts, not release-note items.
  • The pivot (09:00, 10 Sep). The generation tier is now the cloud model; the Macs become free replication capacity; the harness work — stopping, orientation, phase B discipline, a verify queue — continues in avalon-speedway, measured on that model. The cutover run CUTlegacy passed from the new tree, with phase C1 exercised green for the first time in the programme.

The story, in words: The model-ops programme, part 2. Details: docs/SCOREBOARD.md §2k.

Update — 9 September. The open question this page ended on (“why the survey doesn't finish”) was answered that evening and confirmed overnight:
  • The blank plans were never blank. Every killed survey already held the full, correct plan in its transcript — phase A doesn't fail to converge, it fails to stop. A 40-call cap plus forced emission turned 6-of-8 blanks into 3 of 4 graded passes (the fe queue, including the cutover run FE1).
  • The sheet mechanism is confirmed, not just observed. Placebo control (same-size sheet, decisive facts removed): 0 of 2. Generality: sup-log-to-terminal — 0 of 4 locally, 0 of 3 for cloud frontier upstream — passed 2 of 2 with one verified paragraph on the broadcast logger.
  • Overnight step 4 — ten tasks upstream ≥ 88%: 2 of 10 passed (ac-throttle-search, sup-legacy-conversions again with the oracle diff), zero blank plans in ten, and six of the eight misses were each one verified fact away.
  • Now running: knowledge keyed by Rails topic, not task — a blind topic-indexed sheet (knowledge/rails8_topics.md, 8 sections, every entry verified with bin/rails runner), the survey names the topics, the harness injects those sections. Eight held-out tasks run overnight 9–10 Sep, with a cloud calibration lane (deepseek-v4-flash, capped at US$10) in parallel.
Details: docs/SCOREBOARD.md §2i, §2j, §2k.

Apple Silicon · 25 August – 10 September 2026

A 4-bit model on a laptop passed a real Rails task three times. Four lines of prompt did it. The next task showed exactly where it stops.

Fifteen days of measuring local coding agents against the 21-task rails/ai-evals corpus. The retrieval layer the programme was built to test turned out not to be the constraint. A stopping rule was. Then a boundary. Then a 1.6 KB sheet of verified upgrade notes.

Read the findings See the work order

What the programme was

One person, two MacBook Pros, a fixed corpus, and an agent that ran the experiments overnight.

The Model-Ops Evaluation Programme set out to test one thesis: local models can do accurate, timely Rails work if a RAG layer injects the right docs at dispatch. It measured that against the 21-task rails/ai-evals corpus (Writebook on Rails 8.2.0.alpha), first through the upstream lemans/miniswen harness, then through pi with a hand-built grader. The host was a 14-inch MacBook Pro M1 Max with 64 GB, serving Qwen3.8-27B-4bit via oMLX; a second M1 Max was commissioned mid-programme.

The thesis did not survive contact. Real retrieval reached ~56% recall against the oracle's 100%, and a matched pair on Opus showed RAG making no difference at all. What produced the first local pass was a four-line stopping rule; what produced the second was a two-phase survey-then-execute pipeline handed a 1.6 KB sheet of empirically verified Rails upgrade notes.

What did not move: multi-locus tasks without that sheet. And the survey phase's 8-blank-plans-in-10 stretch turned out to be a missing stopping rule, not a missing plan — capped and asked to emit, the same surveys produced first-attempt passes that evening.

Prompt structure beat model choice. Harness beat retrieval. A small verified sheet beat general RAG. And the survey phase was never failing to think — it was failing to stop. Forced emission fixed it in an afternoon.

21tasks in rails/ai-evals, against 1,170 published upstream trials
22.0 sper step, local 4-bit via openai/ adapter, vs 37.7 s cloud
3 / 3ac-throttle-search passes with the stopping rule, a different implementation each time
0 / 6multi-locus passes with the stopping rule alone, across two tasks
1.6 KBverified upgrade notes into Phase A: first sup-legacy-conversions pass, diff byte-identical to the oracle
8 / 10Phase A attempts on Sep 8 that produced no work order — fixed that evening by a call cap + forced emission (3 of 4 graded passes)
10 / 20cloud calibration through the same harness — deepseek-v4-flash, $3.20, matching its own leaderboard profile
1 / 8held-out tasks passed on first attempt (2 / 12 over both rounds, both the same task)

Read the denominator carefully. Per graded run, the pipeline looks good — when Phase A produces a work order, the rest works and the answer is exactly right. Per attempt, Phase A fails to produce a work order more often than it succeeds: KBrep1 needed 3 attempts for its 1 pass; two produced no plan and cost ~45 minutes each. Until 2026-09-08 the retry logic recorded only the successful attempt, so the flattering rate was the only one visible. Every attempt is now written to the results table. For comparison against the published leaderboard, per-attempt is the honest figure — upstream runs do not get retries.

The held-out number tells the same story: 1 of 8 unseen tasks passed on first attempt (2 of 12 over both rounds, both the same task — a one-token typo), and 13 of 17 misses were process (plan, behaviour, scope), not knowledge. Per-attempt is the only denominator that makes that visible.

Architecture and design

A senior→junior split with the same local model in both seats, a harness-level loop guard, and a retrieval layer that ended up replaced by a curated sheet. Source: docs/PHASED_RUN_DESIGN.md, docs/AGENT_EXECUTOR_PAIR_PROMPT_MODEL.md.

Phased run pipeline Task body plus knowledge sheet feed Phase A survey, which is read-only and emits a work order. Phase B execute receives the work order verbatim and applies it with full tools. Phase C verify runs the suite. A harness-level loop guard and stall detector watch phases A and B. Every attempt is recorded, including blank plans. Input Task body + 1.6 KB knowledge sheet Phase A · senior Survey (read-only) read, grep, find, ls · emits work order Phase B · junior Execute full tools · "the search is finished" Phase C · verify bin/rails test → grade C1 green once — CUTlegacy, 10 Sep loop guard: duplicate tool call in last N → interrupt · stall detector: 480 s silence → terminate Per-attempt record: calls, uniq, dup %, edits, SURVEY_EXIT, bytes, grade
The design is A + B + C + guard. What actually produced the passes is narrower: task → knowledge sheet → A → B → grading. Phase C1 was exercised for real for the first time on the 10 Sep cutover run (CUTlegacy, green); the loop guard has fired in real runs (KBrep2__a2 at 20.0%, FE2's diffuse trip, FE4's tight trip) — and since 8 Sep evening a trip ends in a forced plan rather than a blank. C1 is now a single green data point, not a proven component.

The work order Phase A is asked to write

You are in phase 1 of 2: SURVEY. You have read-only tools.
You cannot edit anything, and you will not be asked to.

Your only job is to produce the plan the next phase will execute.

1. Find every site that must change. Do not stop at the one
   described above — the same defect may appear in several
   places, and may appear in more than one form.
2. If a search returns nothing, accept that and move on.
   Do not repeat a search you have already run.
3. Write the plan incrementally. Emit each site as you
   confirm it. A partial plan is useful; a plan you never
   finish is not.

Then output ONLY this, and nothing else:

SITES: <n>
- <path>:<line> — <exact current code> → <exact replacement>
- ...
RISKS: <anything that looks similar but must NOT be changed,
        or "none">
Point 1 is the definition of done. Point 2 is the dead-end rule. Point 3 was added on Sep 8 after two surveys timed out with zero bytes. RISKS exists because the task contains a trap: a ternary where only one branch may change.

The one that passed — Bprobe, 28 calls, 0% duplication

All six sites confirmed across the whole tree.

Findings:
- 5 sites use `to_s(:format)` — renamed, now `to_fs`.
- 1 site uses the removed `to_default_s` in the nightly
  archive job — that's last night's crash.
- One near-miss: `report_card.rb:28` has a bare `value.to_s`
  in the else branch — must stay.

SITES: 6
- app/controllers/books/reports_controller.rb:16 —
  `updated_at.to_s(:number)` → `updated_at.to_fs(:number)`
- app/helpers/books/reports_helper.rb:3 —
  `time.to_date.to_s(:long)` → `time.to_date.to_fs(:long)`
- app/views/report_digest_mailer/weekly.text.erb:6 —
  `to_date.to_s(:short)` → `to_date.to_fs(:short)`
- app/models/book/reportable.rb:5 —
  `length.to_s(:delimited)` → `length.to_fs(:delimited)`
- lib/publishing/report_card.rb:24 —
  `to_a.to_s(:db)` → `to_a.to_fs(:db)`
- app/jobs/report_archive_job.rb:12 —
  `last_edited_at&.to_default_s` → `last_edited_at&.to_s`
RISKS: report_card.rb:28 `format ? value.to_s(format) :
  value.to_s` — else-branch no-arg `to_s` must NOT change.
Scope slightly wrong, remedy right. It listed 6 of 7 sites and misfiled the ternary as do-not-touch; Phase B changed it correctly anyway. The same task without the sheet (Bphase3) got all 7 sites and prescribed strftime hand-rolls with 5 of 7 formats wrong. Scope right, remedy wrong. That diff is the whole story of Sep 8.

The loop guard, and what it can't see

Prompts failed four times to stop a local model repeating itself — 38× bin/rails runner 'puts Book.count' in one run, one grep 47 times in another. So the guard is harness-level: it watches the session JSONL and interrupts on a duplicate tool call in the last N. Removing tools moved the metric where prompts hadn't: the read-only survey ran at 5% duplication where the bash-equipped arm ran at 42%.

Then the Sep 8 queue showed the guard's blind spot. Of eight blank plans, only one was a loop. Four had duplication at or below 3.1%, three at exactly 0.0%. Phase A doesn't loop; it reads honestly and never converges on a plan. A duplication guard cannot see that, which is why a 480-second stall detector was added the same afternoon, threshold set above the longest legitimate gap observed (349 s).

The retrieval layer, and what replaced it

Hybrid dense + lexical retrieval with RRF fusion and LLM query expansion over Rails 8.2.0.alpha at the pinned SHA, turbo-rails, and the app's own source. It reached ~56% recall at five, and on the one task where it changed an outcome it cost 52% more steps on a task the model already passed. What actually closed the last gap was a hand-curated sheet of Rails upgrade notes, every entry verified with bin/rails runner, handed to Phase A. Knowledge was the constraint; untargeted retrieval was the wrong delivery mechanism.

Hardware and harness stack

  • Eval hostMBP-14, M1 Max, 64 GB. oMLX 0.6.4, ceiling 64.4 GB
  • Second hostMBP-16, M1 Max, 64 GB. oMLX 0.6.3rc1, ceiling 55.7 GB — not interchangeable for controlled comparisons
  • Modelmlx-community/Qwen3.8-27B-4bit, ~15 GB resident, v22.4 chat template. Also run: Gemma 12B (agentic fine-tune, MLX 8-bit), IQuest 40B
  • Local harnesspi v0.84.4 in lemans-writebook-pi:v1, graded by factory_harness.sh
  • Upstream harnesslemans / miniswen (bash-only), used for cloud controls via OpenRouter
  • Frontier controlClaude Opus 5 via Claude Code harness
  • Corpusrails/ai-evals, 21 api-knowledge tasks, Writebook on Rails 8.2.0.alpha at 3a4961048ad2
  • Prehistoryexo cluster over the same Macs, pre-oMLX; August OpenRouter spend $101.48 USD; see docs/exo/

Two M1 Max machines at $2,400 each sitting idle was the prehistory's stated concern. The programme is the answer to "can they do anything useful once running."

Key findings

Six results, each with the number that supports it. Sources: docs/SCOREBOARD.md, docs/RAG_EFFICACY_EXPERIMENT.md, docs/LOCAL_MODEL_BENCHMARK_RECONCILIATION.md, RESULTS_TABLE.md.

1

Prompt beat model beat retrieval

Experiment 1 gave the local model a full editing harness — pi with read/edit/write — and it never called an edit tool in an hour. 84 of 87 calls were bash; one grep ran 47 times. Experiment 1b changed nothing but the prompt: four lines telling it when to stop investigating. It passed in 21 minutes and stopped voluntarily — the first local run ever to judge itself finished. The "10 tool calls" in the rule never bound anything; both passes first edited at call 25. The working ingredient is the transition imperative.

Exp 1, no rule
87 calls, 0 edits, FAIL, killed at 60 m
Exp 1b, + rule
28 calls, 3 edits, PASS, 21.3 m, PI_EXIT=0
Replication
3/3, three different implementations
2

Static scores did not predict agentic completion

On Aug 25's single-turn rubric, IQuest 40B 8-bit scored 95/100 and local Qwen 88/100. In the agentic harness, Qwen became the primary executor for tool-schema adherence and IQuest hallucinated another vendor's tool names. Then the sharper version: the same Qwen family passes 76.2% of upstream trials from the cloud and passed 0 of 7 locally in miniswen. 100% of our failures never reached the verifier; only 12.5% of upstream failures fail that way. We were not failing at Rails. We were failing before Rails became relevant.

Static winner
IQuest 40B, 95/100
Agentic executor
Qwen 3.8 27B 4-bit
Cloud vs local, same family
76.2% vs 0/7 (miniswen era)
Never submitted
100% ours vs 12.5% upstream
3

Local inference is not slow. The adapter was.

Same weights, same server, same task: 41–48 s/step via the ollama/ provider path, 22.0 s/step via openai/. That is faster than the cloud model that passed (37.7 s via OpenRouter). It also exposed a fairness bug: the bench sets 100 steps and 60 minutes, cloud runs exhaust steps, and the slow local runs exhausted the clock at 70–88 steps. The 0-for-7 versus 76.2% comparison was never made under the same budget.

Local, ollama/
41–48 s/step
Local, openai/
22.0 s/step (104 requests, 2292 s)
Cloud, OpenRouter
37.7 s/step (40 steps, 1507 s)
Token shape
~1.70 M prompt : 14.4 K completion
4

RAG efficacy is narrow, and general RAG was the wrong delivery

Real retrieval reached ~56% R@5. On cloud Qwen it converted sup-legacy-conversions from 0/3 upstream to 1/3 — and on the guardrail task it already passed, injection cost 52% more steps for the same outcome. The properly controlled pair on Opus was a clean null: plain and RAG arms produced byte-identical diffs and failed the same hidden test. Both saw the second bug and reasoned themselves out of fixing it — a judgement failure, not a knowledge gap. What worked was the opposite of general retrieval: 1.6 KB of curated, empirically verified notes into Phase A.

Retrieval recall
~56% R@5 vs oracle 100%
sup-legacy, cloud Qwen
0/3 → 1/3 with RAG (61 steps, 44 m)
Guardrail cost
40 → 61 steps (+52%), same pass
Opus matched pair
both FAIL, identical diff
Knowledge sheet, local
PASS, diff = solution.patch
5

The multi-locus boundary

The stopping rule is 3-for-3 on ac-throttle-search (rated easy, one locus of change) and 0-for-6 on two tasks that need edits in several places. Opus passed the harder task with no docs in 27 calls. The local model, once phasing gave it correct scope, still failed on remedy: it never found to_fs, declared the removed API had no replacement, and hand-rolled formats that missed 5 of 7. Whether thin Rails 8 knowledge is fatal depends on the verifier: behavioural assertions forgive a hand-rolled fix, exact-output assertions do not.

Single-locus
3/3 local
Multi-locus, rule only
0/6 local
Opus, no docs
PASS, 27 calls, 6.6 m
Local remedy errors
5 of 7 formats wrong
6

Duplication as the discriminator — and its limit

In the single-phase arms, every failure ran 20–82% duplicate tool calls and the pass ran 0%. It is cheap to compute, and it became the loop guard's trigger. But the Sep 8 queue qualified it: of eight blank Phase A plans, only one looped. The survey mostly fails by reading exhaustively and never emitting — 104 calls, 72 reads, 30 ls, 0% duplication. Timing separates outcomes better than duplication does: every plan ever produced arrived by 19.7 minutes; every blank ran past 24.

Single-phase fails
20–82% dup
Single-phase pass
0% dup
Phase A blanks that looped
1 of 8
Plans arrived by
10.8, 12.9, 19.7 min
Blanks ran to
24.1, 37.9, 42.5, 44.9 min
After forced emission
3 of 4 graded passes (fe); 0 blanks in 10 (step 4)

Before and after

Where the three repos meet: a task from rails/ai-evals, seeded into basecamp/writebook, graded by the rails/lemans verifier. Three worked examples from the evidence directory. Sources: docs/PROGRAMME_MAP.html §3, docs/SCOREBOARD.md §2b/§2e/§2g, benchmarks/…/<run>.diff.

Suite22 ai-evals tasks against Writebook. 21 are api-knowledge; hello-world proves the pipeline. 4 touched, 18 untouched.
TaskFour files: instruction.md (frontmatter carries rails_anchor, stripped), environment.patch (seeded defect), solution.patch (oracle), verification_test.rb (hidden).
ArmOne run = task × one intervention. The only variable. Named Bphase4, Bprobe, KBrep1__a3…
PipelineA ▸ B ▸ C, guard watching. The thing being built. Everything above and below this row is upstream's.
Gradinglemans' verifier, mirrored by factory_harness.sh: preverify app suite ▸ restore test/ bin/ config/environments/test.rb ▸ run hidden test. Exit status is the reward.

sup-legacy-conversions — same seven lines, three outcomes

7 sites · 6 files · 1 trap · local Qwen 27B 4-bit both columns

The seeded defect is a Rails 7.0 rename left unapplied: to_s(:format) became to_fs(:format) and to_default_s was removed. The task describes it the way a bug report would — the nightly archive job crashed — and never names the API. Both runs below had a correct work order and made their edits in 13 calls at 0% duplication. One had been handed 1.6 KB of verified upgrade notes in Phase A. The other had not.

Writebook fileSeeded (before)Bphase4, no sheetBprobe, with sheet = oracle
books/reports_controller.rb:16updated_at.to_s(:number)✓ strftime("%Y%m%d%H%M%S")✓ updated_at.to_fs(:number)
books/reports_helper.rb:3to_date.to_s(:long)✓ strftime("%B %d, %Y")✓ to_date.to_fs(:long)
report_digest_mailer/weekly.text.erb:6to_date.to_s(:short)✗ %b %d — verifier wants %d %b✓ to_date.to_fs(:short)
book/reportable.rb:5length.to_s(:delimited)✗ number_with_delimiter — a view helper, errors in a model✓ length.to_fs(:delimited)
publishing/report_card.rb:24to_a.to_s(:db)✗ map(&:title).join(", ") — wants ids✓ to_a.to_fs(:db)
publishing/report_card.rb:28format ? value.to_s(format) : value.to_s✗ strftime without the time component✓ value.to_fs(format), else-branch untouched
report_archive_job.rb:12last_edited_at&.to_default_s✗ strftime("… %z") — wants %Z✓ last_edited_at&.to_s
Hidden verifier—4 failures + 2 errors · FAIL0 failures, 0 errors · PASS · diff byte-identical to solution.patch

Bprobe diff, app/jobs/report_archive_job.rb

   private
     def header(book)
-      "# archived #{book.last_edited_at&.to_default_s}"
+      "# archived #{book.last_edited_at&.to_s}"
     end
 end

Verified runtime output, probe against real records, rolled back

report_stamp   : September 07, 2026
filename       : shelf-report-20260907072256.txt
report card    : Updated: 07 Sep 07:22
                 Covers: 1,2,3
archive header : # archived 2026-09-07 07:22:56 UTC
digest line    : * Probe Book, last edited 07 Sep

Scope was never the problem in the left column: Bphase4 found all seven sites. It believed to_default_s had no Rails 8.2 replacement and hand-rolled formats; five of seven missed the verifier's exact strings. The right column's survey found only six sites and misfiled the ternary as do-not-touch; Phase B changed it correctly anyway. One fact — renamed, not removed — is the entire difference.

as-purge-embedded-images — one line, and the line it didn't write

Opus via Claude Code · matched pair · plain vs +11,704 chars of RAG · both FAIL

The controlled RAG test. Two fresh Opus runs, identical harness, only the instruction varied: one plain, one with the retriever's top chunks appended, purge_later at rank 0. Both arms produced the same one-line change in app/models/page.rb:

-  dependent: :destroy
+  dependent: :purge_later

Both failed the same hidden test — a page revised after its image went in frees the revision's image too. Both runs saw the second leak. Both wrote it up under uncertainties and declined to fix it:

"I judged this a pre-existing upstream design quirk rather than the injected bug… Flagging it in case the hidden tests exercise the edit-then-destroy path."

The contaminated run — the one that received rails_anchor: purge_later because the frontmatter had not been stripped — found both bugs and passed. Being handed part one appears to have freed it to look further. That is a scoping judgement, not a knowledge gap, and it is why eleven kilobytes of correct documentation changed nothing. The result is the cleanest null in the programme and the reason the RAG line was closed.

ac-throttle-search — pass by hand-rolling

local Qwen 27B 4-bit · 3 of 3 with the stopping rule · Rails API recall: 0 of 3 · the verifier rewards behaviour

The task turns on Rails 8's built-in rate_limit. No local run ever found it — and the run you're about to see didn't either. It passed anyway, which is the point. Run C left this on disk:

  • Mapp/controllers/books/searches_controller.rb
  • ??app/controllers/concerns/rate_limited.rb
  • ??lib/rate_limiter.rb

A concern and a library class where the oracle adds one macro to the controller. The behavioural verifier accepted it, as lemans' methodology says it should: the checks test behaviour, not implementation, and API recall is scored separately from passing. It is the mirror image of example 1 — there, a behavioural verifier would have forgiven the strftime hand-rolls too; an exact-output verifier did not. Whether thin Rails 8 knowledge is fatal depends on what the test asserts.

Timeline of breakthroughs

From static rubrics to a measured boundary. Sources: docs/LOCAL_MODEL_BENCHMARK_RECONCILIATION.md, docs/JOURNEY.md, docs/plans/2026-09-08_PLAN.md.

  1. Static benchmarks

    Single-turn prompts scored 0–100 on Rails 8 omakase, inverted maths, locking. IQuest 40B 8-bit wins at 95/100. Bottleneck named as architectural knowledge.

  2. The paradigm shift

    Evaluation moves to live multi-turn tool execution. Static scores stop predicting anything; Qwen 3.8 27B 4-bit becomes the primary local executor on tool-schema adherence and clean exits. The pi coding agent is adopted as the harness. Bottleneck renamed: tool dialect and loop termination.

  3. The retrieval layer, and the A/B that measured nothing

    Hybrid retriever built and measured at ~56% R@5. Six agentic trials, zero solved, every one killed by the wall clock. One run received the correct answer at step 17 and re-ran the identical probe 61 more times.

  4. The cloud control, and the throughput finding

    Cloud Qwen passes ac-throttle-search in 40 steps through our exact harness: infrastructure sound, fault isolated to the local stack. Local via openai/ measured at 22.0 s/step, faster than the cloud. RAG converts sup-legacy 0/3 → 1/3 on cloud Qwen; the Opus matched pair is a clean null.

  5. First local pass

    Experiment 1b. Same harness as the hour-long failure, plus four lines of stopping rule. 28 calls, three edits, 21 minutes, voluntary stop. It never found Rails 8's rate_limit and hand-rolled a rate limiter instead; the verifier checks behaviour, so it passed.

  6. The boundary

    Seven runs while Mark slept. ac-throttle-search passes a third time with a different implementation. Everything multi-locus fails: 0-for-6. A definition of done makes it worse. The two-phase arm times out at 25 minutes with 0 bytes — recorded INVALID, not FAIL.

  7. Phase A works. Phase B still loops. Then it doesn't.

    Bphase3: give the survey a session directory and ask for incremental emission, and it finishes in the budget that twice produced nothing — all 7 sites, 0% duplication. Execute then loops on a correct plan, hunting a timestamp format that is not in the repo. Bphase4, byte-identical plan, guard on: 13 calls, 6 files edited, guard never fired. Phase B's variance is too large for single-run comparisons.

  8. The knowledge sheet

    Bprobe: one variable against Bphase4, a 1.6 KB sheet of verified upgrade notes into Phase A. First local pass on sup-legacy-conversions, 6 runs, 21 assertions, 0 failures, diff byte-identical to the oracle. 0-for-6 becomes 1-for-7.

  9. Forced emission — the blank plans were never blank

    The replication queue's failures (8 blanks in 10) got the forensic they needed: every killed survey, resumed with no tools and asked for its plan, produced it in minutes — the three sup-legacy ones with the oracle's exact remedy. Phase A wasn't failing to converge; it was failing to stop. The harness gained a 40-call cap and a forced plan; the fe queue passed 3 of 4 graded runs the same night, including the cutover run FE1 from the new repo.

  10. Boundary confirmed from the other side

    Replication queue: 10 attempts, 2 graded, 1 pass, 8 blank plans. Only one blank was a loop. The revised diagnosis — phase A fails to stop, not to think — was written down that evening and confirmed by experiment hours later. The programme's next question was answered before it was asked.

  11. The sheet holds — placebo, generality, and a 10-task sweep

    Placebo control: same-size sheet without the decisive facts, 0 of 2 — the mechanism is the specific facts. Generality: sup-log-to-terminal, 0 of 4 locally and 0 of 3 for cloud models, passes 2 of 2 with one verified paragraph. Step 4: ten tasks, 2 of 10 passed, zero blank plans, six of eight misses each one verified fact away.

  12. Topic-keyed knowledge, and the hosts at full capacity

    The sheet is now keyed by Rails topic, not task — eight sections, every entry verified with bin/rails runner, and the survey names the topics it needs. MBP-16's hidden blocker (the ANE prefill setting) was found and disabled, making it the faster host; eight held-out tasks and a cloud calibration lane (deepseek-v4-flash, capped at US$10) run overnight 9–10 Sep.

  13. The number that changed the plan

    Eight unseen tasks through the blind topic-keyed sheet: 1 of 8 on first attempt (2 of 12 over both rounds, both ar-compact-positions — a one-token typo). The coverage check handed the sheet's own facts to three known misses: 0 of 3. The cloud lane — deepseek-v4-flash through the same harness — passed 10 of 20 at $3.20, matching its own leaderboard profile. 13 of 17 misses were process, not knowledge. On the morning of 10 Sep the generation tier moved to the cloud; the Macs become replication capacity; the harness work continues in avalon-speedway, where the cutover run CUTlegacy passed with phase C1 green for the first time.

Roadmap

The goals as written in docs/plans/2026-09-08_PLAN.md, with their status, and the gates that follow from §2h of the scoreboard.

G1 · done 8 Sep

Finish the two-phase run

Phase A produces a complete, correct work order in 25 minutes. Both prerequisites — a transcript and recoverable partial output — turned out to be one change: --session-dir instead of --no-session.

G1b, the diagnosis it revealed: "the junior loops for want of scope" is dead. It had the scope. The constraint moved to Rails 8 API knowledge.

G4 · PASS 8 Sep · G5 · failed

Targeted knowledge, then replication

G4: a small curated verified delta sheet closes the knowledge gap. Cheap, static, derivable from release notes for upgrade-shaped work.

G5's replication queue failed 8-of-10 on blanks — but the blanks turned out to be plans waiting to be asked for. The mechanism itself held up under placebo (0 of 2 without the decisive facts) and generalised (2 of 2 on sup-log-to-terminal, a task five frontier models score 0 on).

Answer — forced emission · the open question is closed

Why the survey doesn't finish: it never knows it's done

  • Forced emission (8 Sep evening). Every killed survey, resumed with no tools and asked for its plan, produced it in minutes — the three sup-legacy ones with the oracle's exact remedy. Phase A was failing to stop, not to converge.
  • Harness fix. Survey capped at 40 calls; any survey that ends without a plan is asked for one with the tools removed. Result: 3 of 4 graded passes on the same arm that was 2-of-8 the day before, and 0 blank plans in ten runs overnight.
  • Sep 9. The sheet is keyed by Rails topic, not task; eight held-out tasks and a cloud calibration lane run overnight 9–10 Sep. Results land in the scoreboard §2k.
Decided 10 Sep — cloud-first

The pivot: the generation tier is now the cloud

The overnight test gave the answer the routing rule needed. Eight unseen tasks through the blind sheet: 1 of 8 (2 of 12 over both rounds, both the same task — a one-token typo). The coverage check handed the sheet's own facts to three known misses: 0 of 3. The cloud lane through the same harness passed 10 of 20 at $3.20, matching its own leaderboard profile — no lift, no harm, at a fraction of the time and cost. And 13 of 17 misses were process, not knowledge: the local model does not stop, loops on thin plans, edits the right file the wrong way.

The decision that follows is the pivot already recorded here: the generation tier is the cloud model; the Macs become free replication capacity; and the harness work — stopping, orientation, phase B discipline, a verify queue — continues in avalon-speedway, measured on that model. The cutover run CUTlegacy passed from the new tree, with phase C1 green for the first time.

Deferred, and still open

  • Generality across the suite — the eight held-out tasks through topic-selected sheet sections ran overnight 9–10 Sep: 1 of 8 on first attempt, and the sheet's own facts did not convert three known misses (0 of 3). The routing rule is now: single-locus, local, well-specified — with the cloud as the generation tier for everything else.
  • Greenfield A/B/C — now cloud-first: local Qwen vs cloud Flash vs cloud SOTA with pair prompting, when a real set of customer requirements exists. Kickoff drafted, not run.
  • Quantisation ladder — IQuest 40B 4-bit vs 8-bit, sequentially on one box. Staged, not run.
  • Fine-tuning a Rails-specialist model — only once enough platforms have shipped to make a concrete cost, latency, or privacy case. Two-of-eight is a reason to fix the loop, not to train a model.
  • Phase C1 exercised for real once on the 10 Sep cutover (CUTlegacy, green) — a single data point, not proof. The sweet-spot experiment (how much context is enough) is queued behind SPEC-1 Part C.

Trials and tribulations

Five failures, each of which paid for itself in what it revealed. None of them was a model failure.

The bash tool-casing bug

Gemma 12B emitted Bash; the lemans harness expected bash. Six trials failed at the harness before the model's competence was ever tested, and they sat in the scored count until the forensic pass separated harness failures from model results. Fixed since.

What it taught: tool-schema conformance is a harness/format interaction, not a knowledge gap, and no amount of retrieval addresses it. The programme's first lesson in not scoring its own bugs as model results.

The definition-of-done backfire

Bdod added "fix every site this defect can occur at" to the stopping rule. It was the only arm that made the model act — 8 edits where every other sup-legacy arm made 0. It then wrote lib/rails_ext/active_support_formatting.rb, monkey-patching the removed Rails API back into existence globally, and broke the app's own test suite before the hidden tests ran.

What it taught: instructing for completeness without constraining method converts inaction into sweeping wrong action. The remedy went into the work-order format, not the prompt: exact current code → exact replacement, per site.

The wall-clock trap

The bench sets step_limit: 100 and timeout: 60m. Cloud models burn 100 steps in minutes. Local runs via ollama/ at 41–48 s/step hit the clock at 70–88 steps and never received their budget. A three-hour, 231-step run produced exactly what the one-hour runs produced, so more time was not the answer either.

What it taught: set the clock so the step limit binds, or the result measures throughput rather than capability. At 22 s/step that means at least 40 minutes; at 45 s/step, 80. Every earlier upstream-vs-local comparison was retracted as not like-for-like.

The blank plan epidemic

Two surveys timed out with 0 bytes at 25 and 45 minutes because pi -p prints only on completion. Two later blanks were retried silently before the stepper recorded them, so a run reported a clean PASS while concealing a 1-in-3 denominator; one transcript was overwritten and is unrecoverable. Then the replication queue: 8 blanks in 10 attempts, every one with zero assistant text.

What it taught: a blank is a result and must be recorded as one — and the follow-through matters: session directories made timeouts recoverable, a stall detector caught the 33-minute silences, and the forced-emission experiment that evening proved the plans were in the transcripts all along. Phase A was failing to stop, not to think. The survey is no longer the open question.

The host-down silent failure

oMLX on MBP-16 exited cleanly mid-run (Finished server process, empty crash log). The container issued 5 tool calls into a dead endpoint, then sat silent for 14 minutes, and was graded as a blank plan. Three gaps let it run that long: the health gate only runs at launch, the stall detector had not landed in that container, and the KeepAlive LaunchAgent had been removed that morning to make the GUI sole supervisor.

What it taught: excluded from the denominator by Mark's call — counting a dead host as a model failure would overstate the blank-plan rate, exactly the drift per-attempt accounting exists to prevent. The control arm it belonged to therefore answers nothing, and is still owed a run.

Harness, prompt, budget, instrumentation, operations. The leverage is in the loop, and so are the bugs. The scoreboard is harsher than it was a week ago because the accounting got better, not because the model got worse.

The One-Person Software Factory

The blueprint this programme exists to serve — and which, until the Sep 8 migration, no model-ops document cross-referenced. Source: docs/architecture/ONE_PERSON_FACTORY.md, docs/LOCAL_MODEL_BENCHMARK_RECONCILIATION.md §3. The routing rule below was rewritten on 10 Sep when the pivot landed.

The vision

One person ships and supports multiple platforms without a team. A hub-and-spoke orchestrator holds only routing, the credential vault, and escalation; everything else is delegated to sub-agents for spec, code, infra, marketing and support. Every action report carries exactly three fields — status, data, next — so the orchestrator reads state, not prose. The constraint the blueprint names is curation fatigue, not token cost.

The evaluation programme is the first hard measurement of the implementer seat. Its result is a routing rule with numbers behind it: well-specified single-locus work can go local today (ac-throttle-search, 3/3, plus the overnight wins); multi-locus work goes local only with a curated knowledge sheet and a survey phase that is made to finish by the harness; everything else goes to the frontier. On 10 Sep the frontier seat stops being a fallback and becomes the generation tier: the cloud calibration passes 10 of 20 at $3.20 through the same harness, and 13 of 17 held-out misses were process, not knowledge.

The orchestrator pattern

Ralph Loop: one agent, one goal per loop. The tiering from the reconciliation doc puts cloud models on design and decomposition into bounded "one feature, one verification" slices, and local executors on the slices. The programme's contribution is what a slice has to contain for a local executor to finish it.

  • Tier 1Cloud orchestrator and design strategist. Decomposes features into bounded slices with acceptance criteria.
  • Tier 2Local executors on Apple Silicon via oMLX. Qwen 3.8 27B 4-bit primary: logic, tests, ActiveRecord, bug fixes. Since 10 Sep: free replication capacity, not the generation tier.
  • Phase AThe same local model as senior: read-only survey, emits the work order.
  • Phase BThe same local model as junior: "the search is finished," apply the work order, leave a change on disk.
  • AgentsDuncan Claw authored the reconciliation and the blueprint; the Sandi agent drove the Sep 6–8 evaluation runs; the programme record now lives in the mrichardsJBOIT/model-ops repo.
intent ──▶ Tier 1: spec, bounded slice, acceptance criteria
                 │
        slice shape?
                 ├─ single-locus ─────────────▶ local, stopping rule       (3/3)
                 ├─ multi-locus + upgrade-shaped ▶ local, A→B + verified sheet (2/8, Phase A gated)
                 └─ multi-locus, no sheet ─────▶ frontier · now the generation tier  (cloud: 10/20)
                                                       │
                                 per-attempt record ◀──┘  pass, fail, blank, host-down
The routing rule the programme validated — revised 10 Sep: the frontier branch is now the generation tier, with the cloud calibration (10/20 at $3.20) as its evidence.

Why local models matter

Three reasons, in the order the prehistory ranks them. Cost: August's OpenRouter bill was $101.48 USD, and two M1 Max machines bought at $2,400 each were the "idle" concern that started the exo work. A local loop at 22 s/step costs electricity. Privacy: client repositories never leave the machine. Sovereignty: the loop keeps running when an upstream API changes terms or availability.

The standing constraint of the eval sessions followed from this: everything stays local, both phases, no frontier model as architect or fallback. A pipeline that only works with a frontier planner tells the factory nothing new — Opus already passes most of the corpus unaided.

Decision log

Every decision and its driver, numbered for review. Source: doco/model-ops/DECISION_LOG.md.

Compiled: 2026-09-10 (from the programme record; cross-referenced below) Purpose: one reviewable page of decisions and their drivers — the "why" behind each pivot. Results and evidence pointers are inline; the full narrative lives in 00_INDEX_AND_JOURNEY.md and SCOREBOARD.md. Interpretation is deliberately out of scope here.

Legend: every row is numbered for cross-reference (e.g. "see D08" in a review).

# Date Decision Driver(s) — why Evidence Result / consequence
D01 2026-08-27 Adopt pi (earendil-works/pi-coding-agent) as the local-model executor instead of OpenClaw-native subagents OpenClaw subagents fed XML/text tool dialects (call:write{...}, <function=...>) that oMLX rejected/stripped; pi speaks standard OpenAI tools JSON → 100% native tool execution on oMLX/LM Studio doco/model-ops/PI_HARNESS_LOCAL_MODELS.md (VALIDATED) Qwen 3.8 27B 4-bit proven as champion local worker (0 tool errors)
D02 2026-08-28 Re-rank models by agentic execution, not static rubric Static winners (IQuest 95/100) failed agentically; harness mechanics, not code intelligence, were the bottleneck LOCAL_MODEL_BENCHMARK_RECONCILIATION.md §1–2 Qwen 27B = primary local executor; IQuest/Gemma demoted
D03 2026-09-06 Run the real lemans / rails-ai-evals harness (open-sourced that day) instead of our custom 4-task pilot lemans became open-source + shipped a Docker backend — the Daytona-account blocker disappeared; pilot was single-turn, no hidden tests, not leaderboard-comparable LEMANS_LOCAL_HARNESS_PLAN.md §1 First real, public-methodology-comparable local numbers (Table B: 0/11, all walls)
D04 2026-09-06 Wire lemans via the ollama provider, not openai openai_api_base silently never forwarded (miniswen's configuration_requirements omits it) → agent called real api.openai.com; ollama auto-forwards OLLAMA_API_BASE + zero-cost pricing LEMANS_LOCAL_HARNESS_PLAN.md §3.2 Clean provider path; no RUBYOPT hacks
D05 2026-09-06 Accept network.mode: public in the sandbox Docker backend doesn't support allowlist (Daytona-only) LEMANS_LOCAL_HARNESS_PLAN.md §3.3 Documented rigor reduction; local runs could reach the internet
D06 2026-09-06 Stop benchmark after 2 stages — enough lemans signal Gemma 0/3 (tool-casing), Qwen 0/2 (both walls); pattern established, move to RAG layer memory/2026-09-06.md 21:12; LOCAL_DEV_AGENT_THESIS.md Next phase = RAG layer build
D07 2026-09-07 Test the harness hypothesis: miniswen → pi (with edit tools) 8 miniswen runs, zero edit calls, bash-only; one run re-ran the same grep 61×; "failure = converting a known answer into an edit" JOURNEY.md §4; harness-hypothesis-session-kickoff.md Exp 1: FAIL even with edit tools (still 0 edits!)
D08 2026-09-07 Add the 4-line stopping rule to the pi prompt Exp 1 proved tools alone changed nothing; the model never stops investigating (47× one grep) 00_INDEX_AND_JOURNEY.md §11 First local PASS (Exp 1b, 21 m 16 s, voluntary stop); 3/3 on ac-throttle-search
D09 2026-09-07/08 Overnight queue with stopping rule + 4 named prompt additions Replicate §11; find what breaks the repetition loop (20–82% dup in failures) JOURNEY.md §12; evidence/RESULTS_TABLE.md 1/7 PASS; sup-legacy 0/4, sup-log 0/2 → stopping rule is single-locus only
D10 2026-09-08 Two-phase pipeline (survey → execute → verify) Survey with read-only tools looped far less (2.6–5% dup); separate planning from editing PHASED_RUN_DESIGN.md Phase A became the bottleneck: 8/10 blank plans
D11 2026-09-08 Forced emission — force a plan even on survey timeout Phase A never converges — 8 blank plans, 3 at 0% duplication; a survey that reads honestly must still emit (salvaged 7/7 plans, all complete) SCOREBOARD.md §2i FE1/FE2/FE3 PASS (3/4 graded); cutover FE1 = first graded run from new repo
D12 2026-09-08 Build loop guard + stall detector Duplication 6.8–52.9% in failures vs 0% in passes; one stall sat silent 33 min (three-quarters of a 45-min budget) SCOREBOARD.md §2h Guard tripped once (KBrep2__a2); 4/8 blanks were ≤3.1% dup — loops were the minority case
D13 2026-09-08 Inject 1.6 KB curated Rails upgrade notes into phase A Bphase4: correct scope, wrong remedy — model never found to_fs; a named missing fact, not retrieval SCOREBOARD.md §2g Bprobe PASS, byte-identical to oracle; 2 passes from 8 attempts (KBrep2 blanked)
D14 2026-09-09 Cloud calibration (deepseek-flash, 21 tasks, US$10 cap) Local results provisional; need a harness-lift number on a known model; leaderboard had 63 miniswen trials of the same model BACKLOG.md M9–M11; SCOREBOARD.md §2k 10/20 PASS, $3.20 — no lift, no harm; matches its own leaderboard profile
D15 2026-09-09 SPEC-2 experiment programme (held-out tasks, coverage check, plumbing) Answer the one question: can the local model + harness + blind sheet be trusted, at what pass rate, at what curation cost SCOREBOARD.md §2k Held-out 1/8; sheet 0/3 with fact present, 1/3 absent; 2 of 17 misses were knowledge
D16 2026-09-10 Pivot to cloud-first — stop active local investment Local 10/41 graded (1/8 unseen tasks) vs flash cloud 50% at $0.15/run; failures mostly process, and the harness that fixes them lifts any model 2026-09-10_EVIDENCE_LOCAL_VS_CLOUD.md; STATUS.md 09:50 Local hosts become free replication capacity; harness work continues cloud-first
D17 2026-09-10 Migrate to avalon-speedway (SPEC-3) Pivot confirmed; keep model-ops frozen as the private record REVIEW-3.md Migration reviewed PASS; CUTlegacy PASS from new tree (C1 green first time)

Source map

File Holds
doco/model-ops/00_INDEX_AND_JOURNEY.md chronological narrative, mistakes, thesis status
doco/model-ops/RESULTS_SCOREBOARD.md every graded run (§2g/§2i/§2k = knowledge injection, forced emission, closing record)
doco/model-ops/LEMANS_LOCAL_HARNESS_PLAN.md lemans setup decisions (D03–D05)
doco/model-ops/PI_HARNESS_LOCAL_MODELS.md original pi validation (D01)
doco/prompts/harness-hypothesis-session-kickoff.md · doco/prompts/pi-harness-container-setup.md the pi experiment design (D07–D08)
memory/2026-09-06.md → memory/2026-09-09.md day-level decisions (D06, D11–D14)
model-ops/docs/SCOREBOARD.md (migrated repo) §2i–§2k · BACKLOG.md · STATUS.md SPEC-2 programme, cloud calibration, pivot (D14–D16)
avalon-speedway/results/ · REVIEW-3.md migration + cutover (D17)

Full results tables

Environments, the day-by-day timeline, the upstream leaderboard, and every graded run — the complete reference. Source: doco/model-ops/TIMELINE_TABLES.md.

Compiled: 2026-09-10, from the programme record (workspace + migrated model-ops repo). Companion: doco/model-ops/DECISION_LOG.md (decisions + drivers, D01–D17).

ID map — every row is numberable for review:

  • ENV-01…06 technical environments
  • T01…T41 timeline by day (2026-09-06 → 2026-09-10)
  • A1-… upstream leaderboard, all 18 models · A2-… upstream per-task pass rates · A3-… same-task ours-vs-upstream · A4-… same-model head-to-head
  • B01…B13 first non-harnessed runs (vanilla lemans on Colima/Lima)
  • C01…C12 first pi harness results
  • D01…D19 iterations & experiments
  • E01…E27 our cloud results (phased harness)
  • L01…L17 decision log (mirror of DECISION_LOG.md; re-prefixed here to avoid colliding with Table D)

Note: in the chat version, the decision log used D01–D17, which collides with this file's Table D (experiments). In this combined file the decisions are L01–L17, identical rows. DECISION_LOG.md keeps its original D numbering.

ENV — Technical environments

ID Environment
ENV-01 Upstream leaderboard (rails.org / Evil Martians) — rails/ai-evals: 18 frontier/cloud models × 3 trials × 21 tasks; lemans 0.1.0–1.1.0 + miniswen; deterministic verifier (bin/rails test + verification_test.rb); 1,170 runs, 1,134 graded
ENV-02 First local runs ("lima") — lemans 1.3.1 + miniswen-installed, --backend=docker on Colima/Lima VM (Mac mini M1); 60-min wall + 100-step limit; bash-only tools; Qwen3.8-27B-4bit instruct (oMLX, MBP-14), Gemma 12B agentic 8-bit (oMLX)
ENV-03 pi harness — lemans-writebook-pi:v1 (pi v0.84.4 + Node 22 in-container), oMLX over Tailscale, factory_harness.sh grader, MBP-14; read/edit/write/bash tools
ENV-04 Phased pipeline — survey→execute→verify (phased_run.sh), loop guard + stall detector + forced emission; Qwen 27B 4-bit (MBP-14); MBP-16 second host (Tailscale); Colima 4→6 CPU / 10 GiB
ENV-05 Cloud calibration — deepseek-v4-flash-0731 via OpenRouter, same phased harness, US$10 cap (spent $3.20)
ENV-06 avalon-speedway — 2026-09-10 migration of ENV-04/05 code, cloud-first, REVIEW-3 reviewed

T — Timeline by day

2026-09-06

ID Event
T01 Thesis set: local models + RAG for Rails work; oracle pilots 4/4 (ceiling tests)
T02 Retrieval layer built (c92a71f): Rails 8.2.0.alpha corpus, hybrid dense+lexical, RRF. Real retrieval 56% R@5 vs oracle 100%
T03 RAG A/B: 6 trials, 0 solved, all killed by 60-min wall
T04 Forensics (d2d9b94): bash-only, zero edit calls, 61× identical probe in one run
T05 Gemma tool-case fix Bash→bash (aa27d40)
T06 Long run: Qwen thinking, 180 min / 400-step budget, 231 steps, no result (75161eb)
T07 Leaderboard comparison (6f0522a): cloud qwen3.8-27b 76.2% vs local 0-for-7; A/B task = hardest upstream (15/18 models 0/3)

2026-09-07

ID Event
T08 Cloud control: Qwen 27B full, openrouter → PASS (40 steps, 25 m, $0.26)
T09 RAG test: sup-legacy-conversions 1 of 3 PASS (61 steps, 44 m); guardrail cost +52% steps / +49% wall
T10 Opus 5 control (Claude Code, no docs): sup-legacy PASS, 27 calls, 6.6 m
T11 Matched pair (as-purge, frontmatter stripped): 0/2, byte-identical diffs — RAG = no effect
T12 Throughput: local 4-bit via openai/ 22.0 s/step vs ollama/ 41–48 s vs cloud 37.7 s
T13 pi container built; scorer validated known-good AND known-bad
T14 Exp 1 (edit tools, no stopping rule): FAIL, 87 calls, 0 edits, 47× one grep, clock-killed
T15 Exp 1b (+4-line stopping rule): PASS, 28 calls, 3 edits, 21 m 16 s, voluntary stop — first local pass ever
T16 Overnight queue (7 arms) launched 21:00

2026-09-08

ID Event
T17 Overnight results: ac-throttle-search 3/3 total (C PASS); sup-legacy 0/4; sup-log 0/2 (§2d boundary)
T18 Bphase3: survey works (SURVEY_EXIT=0, all 7 sites) but phase B loops on a correct plan
T19 Bphase4: same work order → phase B edits 6/6 files exactly, wrong remedy (no to_fs)
T20 Bprobe: +1.6 KB verified upgrade notes → PASS, diff byte-identical to oracle
T21 Replication queue: 1 pass / 8 attempts; 8/10 blank plans; phase A = binding bottleneck; KBctl0 = dead host
T22 Dup-metric correction: recomputed on full args (Bdod 20%→6.8%); call counter bug fixed
T23 Loop guard + stall detector + forced emission built
T24 Migration to model-ops repo; smoke + strip-task proofs byte-identical
T25 FE queue (forced emission): cutover FE1 PASS 22:10; FE2, FE3 PASS; FE4 FAIL → 3/4 graded
T26 S3: placebo 0/2 (sup-legacy+decoy), generality 2/2 (sup-log+sheet)
T27 E4 (step 4, ten tasks upstream ≥88%): 2/10 passes, zero blank plans (E4throttle, E4legacy)
T28 MBP-16 joins Tailscale; died mid-run (no supervisor) → KBctl0 lost

2026-09-09

ID Event
T29 SPEC-2 defined; blind sheet authored from changelogs (~2 h); selection recall 0.87, micro precision 0.24
T30 Held-out queue ho/ho16 launched (8 unseen tasks)
T31 Cloud calibration cal1–cal3 launched 12:32; paused 13:10 (4-CPU Colima contention)
T32 Queue 5 (rr/rr16) armed (six known misses through tier 2)
T33 Candidate screens: LM Studio MoE (5 tasks, MBP-14), IQuest-40B-Loop (floor, MBP-14), Gemma 12B (MBP-16)
T34 Model-ops lab page + blog live on gr8ful.dev (12:09–13:00), reconciled to record (12:56), GoatCounter analytics (17:31). Pre-pivot record still said "survey not converging" — corrected on the page

2026-09-10

ID Event
T35 All queues complete, hosts idle; Colima resized to 6 CPU / 10 GiB (08:30)
T36 Gemma screens closed: 0/5; thinking-off pair FAIL (over-edited 3 files → syntax error); 3 held-out runs host-lost to MBP-16 15 W adapter, rerun as …B
T37 Closing record §2k: held-out 1/8 (2/12 both rounds), queue 5 1/6, plumbing 0/2, cloud 10/20, every miss classified, one-paragraph answer
T38 09:00 pivot decision (Mark): cloud-first; no more code changes in model-ops
T39 Migration to avalon-speedway (de67d87; strict scrub clean, smoke identical); REVIEW-3 PASS; R7/R10 recorded
T40 Cutover run CUTlegacy launched 09:46 → PASS 09:59 (94 calls, 7 edits, 6.4% dup; phase C1 green first time in programme)
T41 Remaining: push private (both repos local-only), R7 sign-off, R10 deferred, 3 retired automations to delete

A — rails.org/Evil Martians leaderboard vs our cloud results

A1 — Upstream leaderboard, all 18 models (21 tasks × 3 trials)

ID Model Pass Rate
A1-01 claude-fable-5.1 58/63 92.1%
A1-02 claude-opus-5 58/63 92.1%
A1-03 claude-fable-5 57/63 90.5%
A1-04 kimi-k3 57/63 90.5%
A1-05 gpt-5.6-sol 53/63 84.1%
A1-06 ox-alpha 52/63 82.5%
A1-07 grok-4.6 52/63 82.5%
A1-08 glm-5.3-flash 52/63 82.5%
A1-09 glm-5.3 50/63 79.4%
A1-10 claude-opus-4.8 50/63 79.4%
A1-11 gpt-5.6-terra 49/63 77.8%
A1-12 muse-spark-1.2 48/63 76.2%
A1-13 qwen3.8-27b (same family as our local) 48/63 76.2%
A1-14 gpt-5.6-luna 46/63 73.0%
A1-15 gemini-3.7-flash 45/63 71.4%
A1-16 claude-sonnet-5 44/63 69.8%
A1-17 glm-5.2 42/63 66.7%
A1-18 deepseek-v4-flash-0731 (the model we later calibrated) 41/63 65.1%

A2 — Upstream per-task pass rates (18 models × 3 = 54 runs/task)

ID Task Difficulty Upstream pass rate
A2-01 as-purge-embedded-images hard 7.4% (4/54)
A2-02 ar-erase-account hard 18.5% (10/54)
A2-03 hw-scoped-broadcast medium 64.8% (35/54)
A2-04 ar-bulk-access-grants medium 70.4% (38/54)
A2-05 sup-log-to-terminal medium 70.4% (38/54)
A2-06 ac-deep-link-return medium 74.1% (40/54)
A2-07 as-variant-processed-once medium 75.9% (41/54)
A2-08 ar-compact-positions medium 79.6% (43/54)
A2-09 av-toc-cache-per-role medium 83.3% (45/54)
A2-10 sec-audit-sweep medium 83.3% (45/54)
A2-11 ar-announce-once easy 85.2% (46/54)
A2-12 aj-enqueue-after-commit medium 88.9% (48/54)
A2-13 sup-legacy-conversions easy 88.9% (48/54)
A2-14 ar-tenant-isolation medium 88.9% (48/54)
A2-15 aj-resumable-cleanup easy 94.4% (51/54)
A2-16 tst-error-page-flake easy 98.1% (53/54)
A2-17 ar-archive-book-access easy 98.1% (53/54)
A2-18 ar-atomic-import easy 100% (54/54)
A2-19 sup-cache-library-digest easy 100% (54/54)
A2-20 ar-release-recap easy 100% (54/54)
A2-21 ac-throttle-search easy 100% (54/54)

A3 — Same-task comparison: our cloud (deepseek flash, our harness) vs upstream rate

ID Task Upstream rate Ours (flash) Outcome class (§2k)
A3-01 as-purge-embedded-images 7.4% FAIL same (both ~0)
A3-02 ar-erase-account 18.5% FAIL same
A3-03 hw-scoped-broadcast 64.8% FAIL mixed
A3-04 ar-bulk-access-grants 70.4% FAIL same
A3-05 sup-log-to-terminal 70.4% FAIL same
A3-06 ac-deep-link-return 74.1% FAIL same
A3-07 as-variant-processed-once 75.9% FAIL mixed
A3-08 ar-compact-positions 79.6% PASS same
A3-09 av-toc-cache-per-role 83.3% FAIL mixed
A3-10 sec-audit-sweep 83.3% UNGRADED (no plan ×2) mixed
A3-11 ar-announce-once 85.2% PASS (1 retry) ours better
A3-12 aj-enqueue-after-commit 88.9% PASS (1 retry) ours better
A3-13 sup-legacy-conversions 88.9% PASS same
A3-14 ar-tenant-isolation 88.9% FAIL mixed
A3-15 aj-resumable-cleanup 94.4% PASS same
A3-16 tst-error-page-flake 98.1% FAIL ours worse (plumbing loss)
A3-17 ar-archive-book-access 98.1% PASS same
A3-18 ar-atomic-import 100% PASS same
A3-19 sup-cache-library-digest 100% PASS same
A3-20 ar-release-recap 100% PASS same
A3-21 ac-throttle-search 100% PASS same

A4 — Same-model head-to-head: upstream deepseek-flash vs ours via our harness

ID Config Pass rate Notes
A4-01 Upstream (miniswen, lemans) 41/63 = 65.1% (rank 18/18) —
A4-02 Ours (phased harness, 21 tasks, ENV-05) 10/20 graded = 50% 1 task ungraded (audit); retry needed on 9 first attempts
A4-03 Cross-tab 12 same · 2 ours-better · 1 ours-worse · 4 mixed Cost $3.20 of $10 cap

B — First non-harnessed results, running Lima (ENV-02, lemans/miniswen on Colima)

ID Model Task Reward Outcome Steps Duration Note
B01 Gemma 12B 8-bit (pre-fix) ac-throttle-search 0 format_error 8 2397.6 s Bash casing bug ×3
B02 Gemma 12B (pre-fix) ar-announce-once 0 format_error 6 205.8 s same bug
B03 Gemma 12B (pre-fix) ar-archive-book-access 0 format_error 16 2588.4 s same bug
B04 Qwen 27B 4-bit ac-throttle-search 0 wall 70 3816.9 s never reached verifier
B05 Qwen 27B 4-bit as-purge-embedded-images 0 wall 80 3774.2 s —
B06 Qwen 27B 4-bit sup-log-to-terminal 0 wall 71 3742.4 s only run to produce a diff (1218 B)
B07 Qwen 27B 4-bit +RAG ac-throttle-search 0 wall 78 3754.6 s —
B08 Qwen 27B 4-bit +RAG as-purge-embedded-images 0 wall 80 3744.8 s —
B09 Qwen 27B 4-bit +RAG sup-log-to-terminal 0 wall 88 3784.9 s 61× repeated probe
B10 Gemma 12B (post-fix) sup-log-to-terminal — agent_error (exit 124) — 3660 s harness loss, unscored
B11 Qwen 27B 4-bit thinking sup-log-to-terminal 0 wall 231 10,980 s 180-min long run (T06)
B12 Gemma 12B +RAG ×3 3 tasks — exit 124 ×3 — ~3 h harness flush bug, all lost

B13 — Total: 0/11 scored trials, 0 reached the verifier; 11 killed by wall, 4 harness losses, 1 tool-casing bug set.

C — First pi results (ENV-03)

ID Run Task Arm Grade Calls Edit/Write Dup Time Note
C01 Exp 1 ac-throttle-search no stopping rule FAIL 87 0 52.9% 60 m killed 47× one grep
C02 Exp 1b ac-throttle-search +4-line stopping rule PASS 28 3 0% 21 m 16 s voluntary stop; first local pass
C03 Run A ac-throttle-search rule only PASS 28 3 0% ~21 m repeat of C02
C04 C (overnight) ac-throttle-search rule only PASS 26 4 0% — different implementation (re-derived)
C05 Bdod sup-legacy-conversions + definition-of-done FAIL 59 8 6.8% — global monkey-patch, broke own suite
C06 Bdead sup-legacy-conversions + dead-end rule FAIL 95 0 40.0% — 38× one probe
C07 Bphase sup-legacy-conversions two-phase INVALID 39 0 2.6% — survey cap → blank plan
C08 Mstop sup-log-to-terminal rule only FAIL 87 1 28.7% — edited only development.rb
C09 Mboth sup-log-to-terminal +DoD +dead-end FAIL 71 0 25.4% — edited db/schema.rb (forbidden)
C10 Bphase3 sup-legacy-conversions survey instrumented FAIL 49 0 24.5% — looped holding a correct plan
C11 Bphase4 sup-legacy-conversions same order, guard on FAIL 13 6 0% 6 m 13 s 6/6 files right scope, wrong remedy
C12 Bprobe sup-legacy-conversions +1.6 KB upgrade notes PASS 13 6 0% — diff byte-identical to oracle

D — Iterations & experiments (results)

ID Iteration / Experiment Headline result
D01 RAG A/B (ENV-02 bash-only) 0/6, all wall-clocked
D02 Cloud control (Qwen 27B full) 1/1 PASS (T08)
D03 RAG test (sup-legacy, targeted) 1/3 PASS (T09)
D04 Harness hypothesis / Opus control 1/1 PASS, no docs (T10)
D05 Matched pair (plain vs RAG) 0/2, byte-identical diffs (T11)
D06 Stopping rule (pi, paired) Exp 1 FAIL → 1b PASS; 4/4 total on ac-throttle (Table C)
D07 Overnight boundary (7 arms) 1/7 PASS; 0/6 on multi-locus tasks (Table C rows C04–C09)
D08 Two-phase survey→execute Bphase3 FAIL (phase B loops); Bphase4 FAIL (wrong remedy)
D09 Knowledge injection Bprobe 1/1 PASS, oracle-identical
D10 Replication queue (KB) 1 pass / 8 attempts; 8/10 blank plans; KBctl0 host-down
D11 Forced emission (FE) 3/4 graded passes; FE1 = first graded run from new repo
D12 Placebo + generality (S3) placebo 0/2, generality 2/2
D13 Step-4 ten tasks (E4) 2/10 passes, zero blank plans
D14 Held-out tasks (SPEC-2 q2) 1/8 first attempt (2/12 both rounds) — both ar-compact-positions
D15 Six known misses w/ sheet (q5) 1/6 — 0/3 with fact present, 1/3 without
D16 Plumbing (q7) 0/2; restore-aware addendum deferred
D17 Cloud calibration (q1) 10/20, $3.20, no lift/no harm vs leaderboard (Table E)
D18 Candidate screens LM MoE 1/5 · IQuest 0/1 · Gemma 0/5 · Gemma no-think 0/1
D19 Cutover run (avalon-speedway) 1/1 PASS — CUTlegacy (T40)

E — Our cloud results (all runs through our harness, ENV-03/05/06)

ID Config Task Result Calls Dup e/w Notes
E01 Qwen 27B full, openrouter ac-throttle-search PASS 40 — — 25 m, $0.26
E02 Qwen 27B full +RAG ac-throttle-search PASS 61 — — 37 m, guardrail cost
E03 Qwen 27B full +RAG sup-legacy-conversions PASS (1/3) 61 — — 44 m
E04 Opus 5, Claude Code sup-legacy-conversions PASS 27 — — 6.6 m, no docs
E05 flash (ENV-05) as-purge-embedded-images FAIL 139 2.9% 2 —
E06 flash ar-erase-account FAIL 84 6.0% 3 a1 blank → retry
E07 flash hw-scoped-broadcast FAIL 76 2.6% 2 —
E08 flash ar-bulk-access-grants FAIL 79 2.5% 1 —
E09 flash sup-log-to-terminal FAIL 68 7.4% 1 a1 blank → retry
E10 flash ac-deep-link-return FAIL 40 5.0% 2 —
E11 flash as-variant-processed-once FAIL 14 7.1% 0 a1 blank → retry
E12 flash ar-compact-positions PASS 74 4.1% 1 a1 blank → retry
E13 flash av-toc-cache-per-role FAIL 157 1.9% 1 —
E14 flash sec-audit-sweep UNGRADED 44×2 0% 0 no plan twice
E15 flash ar-announce-once PASS 64 3.1% 2 a1 blank → retry
E16 flash aj-enqueue-after-commit PASS 51 2.0% 1 a1 blank → retry
E17 flash sup-legacy-conversions PASS 146 4.1% 6 —
E18 flash ar-tenant-isolation FAIL 189 13.2% 12 —
E19 flash aj-resumable-cleanup PASS 35 8.6% 3 —
E20 flash tst-error-page-flake FAIL 68 5.9% 3 a1 blank; plumbing loss
E21 flash ar-archive-book-access PASS 58 3.4% 1 —
E22 flash ar-atomic-import PASS 82 4.9% 2 —
E23 flash sup-cache-library-digest PASS 44 4.5% 1 —
E24 flash ar-release-recap PASS 48 10.4% 1 —
E25 flash ac-throttle-search PASS 183 6.0% 3 —
E26 flash, avalon tree (ENV-06) sup-legacy-conversions (CUTlegacy) PASS 94 6.4% 7 C1 green first time

E27 — Calibration totals: 10 PASS · 10 FAIL · 1 ungraded (audit) · 20–25 m/run clean · $3.20 total.

L — Decision log (mirror of doco/model-ops/DECISION_LOG.md)

ID Date Decision Driver(s) — why Evidence Result / consequence
L01 2026-08-27 Adopt pi (earendil-works/pi-coding-agent) as the local-model executor instead of OpenClaw-native subagents OpenClaw subagents fed XML/text tool dialects (call:write{...}, <function=...>) that oMLX rejected/stripped; pi speaks standard OpenAI tools JSON → 100% native tool execution on oMLX/LM Studio doco/model-ops/PI_HARNESS_LOCAL_MODELS.md (VALIDATED) Qwen 3.8 27B 4-bit proven as champion local worker (0 tool errors)
L02 2026-08-28 Re-rank models by agentic execution, not static rubric Static winners (IQuest 95/100) failed agentically; harness mechanics, not code intelligence, were the bottleneck LOCAL_MODEL_BENCHMARK_RECONCILIATION.md §1–2 Qwen 27B = primary local executor; IQuest/Gemma demoted
L03 2026-09-06 Run the real lemans / rails-ai-evals harness (open-sourced that day) instead of our custom 4-task pilot lemans became open-source + shipped a Docker backend — the Daytona-account blocker disappeared; pilot was single-turn, no hidden tests, not leaderboard-comparable LEMANS_LOCAL_HARNESS_PLAN.md §1 First real, public-methodology-comparable local numbers (Table B: 0/11, all walls)
L04 2026-09-06 Wire lemans via the ollama provider, not openai openai_api_base silently never forwarded (miniswen's configuration_requirements omits it) → agent called real api.openai.com; ollama auto-forwards OLLAMA_API_BASE + zero-cost pricing LEMANS_LOCAL_HARNESS_PLAN.md §3.2 Clean provider path; no RUBYOPT hacks
L05 2026-09-06 Accept network.mode: public in the sandbox Docker backend doesn't support allowlist (Daytona-only) LEMANS_LOCAL_HARNESS_PLAN.md §3.3 Documented rigor reduction; local runs could reach the internet
L06 2026-09-06 Stop benchmark after 2 stages — enough lemans signal Gemma 0/3 (tool-casing), Qwen 0/2 (both walls); pattern established, move to RAG layer memory/2026-09-06.md 21:12; LOCAL_DEV_AGENT_THESIS.md Next phase = RAG layer build
L07 2026-09-07 Test the harness hypothesis: miniswen → pi (with edit tools) 8 miniswen runs, zero edit calls, bash-only; one run re-ran the same grep 61×; "failure = converting a known answer into an edit" JOURNEY.md §4; harness-hypothesis-session-kickoff.md Exp 1: FAIL even with edit tools (still 0 edits!)
L08 2026-09-07 Add the 4-line stopping rule to the pi prompt Exp 1 proved tools alone changed nothing; the model never stops investigating (47× one grep) 00_INDEX_AND_JOURNEY.md §11 First local PASS (Exp 1b, 21 m 16 s, voluntary stop); 3/3 on ac-throttle-search
L09 2026-09-07/08 Overnight queue with stopping rule + 4 named prompt additions Replicate §11; find what breaks the repetition loop (20–82% dup in failures) JOURNEY.md §12; evidence/RESULTS_TABLE.md 1/7 PASS; sup-legacy 0/4, sup-log 0/2 → stopping rule is single-locus only
L10 2026-09-08 Two-phase pipeline (survey → execute → verify) Survey with read-only tools looped far less (2.6–5% dup); separate planning from editing PHASED_RUN_DESIGN.md Phase A became the bottleneck: 8/10 blank plans
L11 2026-09-08 Forced emission — force a plan even on survey timeout Phase A never converges — 8 blank plans, 3 at 0% duplication; a survey that reads honestly must still emit (salvaged 7/7 plans, all complete) SCOREBOARD.md §2i FE1/FE2/FE3 PASS (3/4 graded); cutover FE1 = first graded run from new repo
L12 2026-09-08 Build loop guard + stall detector Duplication 6.8–52.9% in failures vs 0% in passes; one stall sat silent 33 min (three-quarters of a 45-min budget) SCOREBOARD.md §2h Guard tripped once (KBrep2__a2); 4/8 blanks were ≤3.1% dup — loops were the minority case
L13 2026-09-08 Inject 1.6 KB curated Rails upgrade notes into phase A Bphase4: correct scope, wrong remedy — model never found to_fs; a named missing fact, not retrieval SCOREBOARD.md §2g Bprobe PASS, byte-identical to oracle; 2 passes from 8 attempts (KBrep2 blanked)
L14 2026-09-09 Cloud calibration (deepseek-flash, 21 tasks, US$10 cap) Local results provisional; need a harness-lift number on a known model; leaderboard had 63 miniswen trials of the same model BACKLOG.md M9–M11; SCOREBOARD.md §2k 10/20 PASS, $3.20 — no lift, no harm; matches its own leaderboard profile
L15 2026-09-09 SPEC-2 experiment programme (held-out tasks, coverage check, plumbing) Answer the one question: can the local model + harness + blind sheet be trusted, at what pass rate, at what curation cost SCOREBOARD.md §2k Held-out 1/8; sheet 0/3 with fact present, 1/3 absent; 2 of 17 misses were knowledge
L16 2026-09-10 Pivot to cloud-first — stop active local investment Local 10/41 graded (1/8 unseen tasks) vs flash cloud 50% at $0.15/run; failures mostly process, and the harness that fixes them lifts any model 2026-09-10_EVIDENCE_LOCAL_VS_CLOUD.md; STATUS.md 09:50 Local hosts become free replication capacity; harness work continues cloud-first
L17 2026-09-10 Migrate to avalon-speedway (SPEC-3) Pivot confirmed; keep model-ops frozen as the private record REVIEW-3.md Migration reviewed PASS; CUTlegacy PASS from new tree (C1 green first time)