Chapter 04 / The evidence
How do we know an agent got better?
Separate model quality, search, assistance and execution.
Start with the claim
“The model improved” and “the agent wins more” are different claims. Changing search, helpers, clocks or hardware can improve the system without changing a single weight.
Model
Held-out loss, raw move legality, WDL calibration.
Harness
Depth, coverage, cache hits, latency and failure recovery.
Agent
Wins, draws and losses under a frozen match protocol.
Freeze the comparison
Use the same checkpoint, opponent settings, openings, colors, clocks and failure accounting. Change one component when trying to isolate its contribution. Paired openings help control starting-position effects.
A useful ablation compares policy-only legal play, search without helpers, and the full assisted agent. The matches should share the protocol; separate historical cohorts are context, not a substitute.
Read the Sunfish result carefully
Recorded / 8 November 2026
49 wins + 1 draw + 0 losses = 49.5 points / 50 games = 99% score.
The win rate is 98%. The opponent is 1,000-node Sunfish. This was the CPU harness with a 10-second cap and full helpers.
That does not prove a reliable over-95% rate on unseen games, isolate the search contribution, or establish an Elo. Openings were repeatedly inspected, and the cohort is small.
Loss and ratings are different lenses
Lower loss can coexist with worse play. A small validation set may be noisy. A larger held-out panel improves the signal but does not replace game evaluation.
Lichess Glicko-2 ratings are separated by time control and depend on opponents and uncertainty. A temporal rise is not controlled evidence for one release. Keep dated snapshots separate from the live profile.
A short result-review checklist
- Which weights and harness version?
- Which opponent, budget, openings and time control?
- How many games, including failures and unfinished games?
- Was the result repeatedly inspected or tuned against?
- What changed, and what cannot be attributed?