Project status: closed. The cluster is decommissioned. This page is the public record — the companion to the post-mortem blog post. Every number below comes from the raw forensic report and rotated server logs.
Three Macs, Thunderbolt, and a 51 GB model that ran at 1.42 tokens per second.
I built a distributed MLX inference cluster with exo across an M4 Mini and two M1 Max MacBook Pros — so models too big for one machine could run locally. The machines never thermally stalled. The software ate itself: 52,220 clock-skew rejections, a 16-hour hang, zero files written. Here's everything, including the parts that worked.
What the project was
A test of the claim that distributed local inference is a viable answer to cloud dependence.
Three Macs, 144 GB of RAM between them: an M4 Mac Mini as the hub and two M1 Max MacBook Pros (64 GB each) on TB4 bridges, running exo v0.3.70 for peer-to-peer model sharding. The goal: run a 51 GB bf16 model that fits no single machine, locally, with no API bills.
The first model that actually needed the cluster — Qwen3.8-27B-bf16, 51 GB — decoded at 1.42 tokens per second. The cluster's defining job was technically achieved and practically useless. Meanwhile every model that fit one machine got slower when sharded.
The collapse on day six — initially blamed on overheating, actually clock skew — was the decisive event. The independent verdict: "2 minutes vs. 3 hours and zero files isn't a tuning problem — it's a decisive result." The project pivoted to a software factory: cloud writes specs, local models execute well-specified tasks.
Node count = ⌈model size ÷ 64 GB⌉. Never "however many machines are on the desk." The cluster's only irreplaceable job was its worst outcome.
Findings
What the logs and benchmarks actually said — in the order they stopped being hypotheses.
Headless macOS MLX is impractical — TCC kills it
Metal GPU access and Local Network multicast need GUI consent, and `uv`'s changing Python binaries silently revoked grants. True headless exo needs a signed launcher no one ships. Rebuilt the nodes as workstations inside a day.
- Fix
- GUI sessions on all cluster nodes; M1 Mini stayed headless with established grants
- Filed
- reference to upstream #952
exo discovery had two bugs — I patched both
A `swap_remove` drained the announce list on any transient `EHOSTUNREACH` (upstream #2191), and a second bug seeded the list empty at boot so the node was deaf at birth. Added `resync_ifaces()` self-healing: periodic reconciliation plus an empty-list trigger.
- Status
- #2191 open upstream; superset of PR #2192
- Result
- 3-node convergence within 60s
Network bandwidth was never the bottleneck
The pipeline bubble: rank 0 at 91.7% CPU, rank 1 idle at 20.5%. Hidden-state transfers used 22 KB/s — 0.001% of the 20 Gbps available. Prefill parallelised at 141 tok/s; decode serialised at 1.42 tok/s — a 100:1 ratio baked into the architecture.
- Decode
- 1.42 tok/s (bf16, 2-node)
- Bandwidth used
- 0.001%
The crash was clock skew, not overheating
52,220 identical rejections in 16 minutes: MBP-16's clock ran 1.4s ahead of the hub's 500ms tolerance. My TB-only topology — no WiFi on any node, chosen for stability — had removed the NTP source. The machines never got hot. Fix: chronyd on the M4 + dnsmasq NAT for internet.
- Rejections
- 52,220 / 16 min
- Skew
- Fix
- chronyd + dnsmasq
Clustering made everything that fits one node slower
The only model that gained from 2 nodes was the 8B (+17% to 5.05 tok/s — still slow). Everything that fit one machine lost: the 27B-4bit dropped 68% when sharded. The 51GB bf16 that required clustering ran at 1.42 tok/s.
- 8B
- +17% · 5.05 tok/s
- 27B-4bit
- −68% · ~2.4 tok/s
- 27B-bf16
- 1.42 tok/s
The independent verdict was decisive
Billy Richards, working with Claude, read everything cold: "The test case indicts the topology, not the network" — a 26B at 8-bit fits one 64 GB M1 Max, so clustering pays hop tax for nothing — and "fan-out beats fan-in": split batch across N independent single-node instances so one node dying loses only its slice.
- Rule
- nodes = ⌈model ÷ 64 GB⌉
- For batch
- fan-out, not fan-in
Benchmarks
Measured on the cluster, 16–18 August. All runs in the raw report.
| Model | Size | Single node | 2-node pipeline | Relative |
|---|---|---|---|---|
| Qwen3-8B-8bit (36 layers) | 8.1 GB | 4.31 tok/s | 5.05 tok/s | +17% |
| Qwen3.8-27B-4bit (64 layers) | 15 GB | 7.4 tok/s | ~2.4 tok/s | −68% |
| Qwen3.8-27B-bf16 (64 layers) | 51 GB | doesn't fit | 1.42 tok/s | required |
| KAT-Coder-V2.5-Dev-oQ2e | 11.4 GB | ~60 tok/s | — | fastest coder |
| Gemma-4-e4b-it-8bit | 8.3 GB | 58.4 tok/s | — | generic baseline |
| IQuest-Coder-V1-14B (trust_remote_code patched) | 14.3 GB | 22.5 tok/s | — | usable |
Full catalog: 11 models, 108 GB, in the raw report. The lesson: MoE models with small active params (KAT-Coder, Gemma-4-e4b) run fast on a single M1 Max — architecture matters more than parameter count.
Timeline
Eleven days, including the nine that mattered.
Headless experiment ❌
TCC consent prompts made headless MLX impractical. Rebuilt as workstations.
Cluster converged ✅
Two discovery bugs patched (`resync_ifaces` self-heal). 3-node topology established.
First sharded inference ✅
NFS over TB fixed (229 MB/s, 28×). Qwen3-8B pipeline response: 2.21s.
TB-only, NFS removed
Models replicated locally (108 GB, 11 models). WiFi disabled on all nodes — the decision that later removed the NTP source.
trust_remote_code patch + first pipeline benchmarks
IQuest loads via local patch (PYTHONPATH override). Systematic sharding data: −68% for 27B-4bit.
The "overheating" crash 🔴
Cluster collapses mid-task. 16-hour hang. Zero files. Initial blame: thermal. Actual: 52,220 clock-skew rejections (MBP-16 1.4s ahead).
Clock sync + internet routing fixed ✅
chronyd on M4 (~1ms spread), dnsmasq NAT/DNS for TB-only MBPs.
External analysis
Billy Richards + Claude: topology indicted, not the network; fan-out beats fan-in; decisive result framing.
Software factory design + decommission ✅
Cluster retired. Pivot: cloud writes specs, local executes. Cost analysis: Claude Co Work doesn't save money.
The judgement
Where local inference wins, where it loses, and what the hardware is for.
Single-node MoE + batch
KAT-Coder ~60 tok/s, Gemma-4-e4b 58.4 tok/s. Fan-out batch across independent instances. Sovereign workloads that must not leave the machine.
Agentic engineering
Interactive, multi-turn, judgement-heavy work. Decisively faster and more capable than anything the cluster produced — reaffirmed by the later model-ops programme.
Software factory
Cloud (cheap flash) writes precise specs; local models execute well-specified tasks for zero ongoing token cost. Measured routing, not religion.
Hardware fate: M1 Max 4TB MBP — daily driver. M1 Max 1TB MBP — sold (storage was the binding constraint). M4 Mini — single-node inference host. Nothing idle; the "idle hardware" problem was a misreading.
Sources & files
The public record — a claim you can verify isn't marketing.
- POSTThe post-mortem blog post — the story in my own words
- REPORTThe full 938-line forensic report (benchmarks, log forensics, bug catalog, cost analysis) — publishing to a public repo with the benchmark harness
- BENCHRaw single-node and pipeline benchmark tables, model catalog (11 models, 108 GB)
- LOGSRotated exo server logs — 135,178 lines / 19.2 MB from the key file
- RELRelated: Model Ops Lab — what a single Mac's local models can do as Rails coding agents