LittlePlyModel + harness

Chapter 04 / The evidence

How do we know an agent got better?

Separate model quality, search, assistance and execution.

Start with the claim

“The model improved” and “the agent wins more” are different claims. Changing search, helpers, clocks or hardware can improve the system without changing a single weight.

Model

Held-out loss, raw move legality, WDL calibration.

Harness

Depth, coverage, cache hits, latency and failure recovery.

Agent

Wins, draws and losses under a frozen match protocol.

Freeze the comparison

Use the same checkpoint, opponent settings, openings, colors, clocks and failure accounting. Change one component when trying to isolate its contribution. Paired openings help control starting-position effects.

A useful ablation compares policy-only legal play, search without helpers, and the full assisted agent. The matches should share the protocol; separate historical cohorts are context, not a substitute.

Read the Sunfish result carefully

Recorded / 8 November 2026

49 wins + 1 draw + 0 losses = 49.5 points / 50 games = 99% score.

The win rate is 98%. The opponent is 1,000-node Sunfish. This was the CPU harness with a 10-second cap and full helpers.

That does not prove a reliable over-95% rate on unseen games, isolate the search contribution, or establish an Elo. Openings were repeatedly inspected, and the cohort is small.

Exact protocol, audits and all games →

Loss and ratings are different lenses

Lower loss can coexist with worse play. A small validation set may be noisy. A larger held-out panel improves the signal but does not replace game evaluation.

Lichess Glicko-2 ratings are separated by time control and depend on opponents and uncertainty. A temporal rise is not controlled evidence for one release. Keep dated snapshots separate from the live profile.

A short result-review checklist
  • Which weights and harness version?
  • Which opponent, budget, openings and time control?
  • How many games, including failures and unfinished games?
  • Was the result repeatedly inspected or tuned against?
  • What changed, and what cannot be attributed?