EXP-17 / evidence study
CM-13: the final snapshot
How did the later snapshot fare on matched openings?
Keep in view. Sunfish evidence, not general Elo or proof that duration alone caused improvement.
Matched 50 opening pairs
Gain +14.5 points; paired opening-cluster bootstrap 95% interval +1.5 to +27 points, 20,000 resamples. All 200 games audited.
| Cohort | Wins | Draws | Losses | Unfinished | Score / bounds |
|---|---|---|---|---|---|
| Final u279393 | 63 | 9 | 28 | 0 | 67.5% |
| Fresh eight-hour reference | 47 | 12 | 41 | 0 | 53% |
Protocol boundaries
Identical helper and timing for both evaluated snapshots.
| Interface | Assistance |
|---|---|
| Evaluated system | Stockfish 19 mate-only |
| Browser | Unassisted |
| Excluded from evaluation | Value search and 97% takeover |
Download the study’s recorded values. Exact source records, hashes and protocol details remain below. Selectors inspect recorded observations; no missing points are interpolated.
Full experiment record, provenance and original links
Study source files and SHA-256 provenance
Original narrative and source links are preserved in the full record below. A linked release inventory may document identity without supplying every study value.
- cm13-final/release.json
5591e07dd6d393a822b051b8ba323b283fdb3cc478f8c16cc2bca228358bf5a3
EXP-17 / completed / CM-13 / final snapshot
EXP-17: CM-13 final snapshot and matched 100-game evaluation
Update 279,393 replaces the earlier eight-hour snapshot; the earlier record remains intact.
Final snapshot versus eight-hour reference
| Matched Sunfish benchmark | Wins / draws / losses | Score |
|---|---|---|
| CM-13 final, 100 games | 63 / 9 / 28 | 67.5% |
| CM-13 eight-hour reference, 100 games | 47 / 12 / 41 | 53% |
Observed gain: 14.5 percentage points. Paired opening-cluster bootstrap 95% interval: +1.5 to +27 percentage points (50 paired openings, 20,000 resamples). Draws count as half a point. This supports improvement on this Sunfish benchmark; it is not general Elo or a browser-strength estimate.
Both cohorts used identical Stockfish 19 mate-only assistance and timing, with experimental outcome search off. Sunfish used 1,000 requested nodes; games used 300+2 clocks and a one-second minimum opportunity. The helper used two engines, two threads and 64 MiB per engine. The later 97% endgame takeover was not part of this evaluation. Browser play has neither this helper nor an endgame takeover.
All 200 games passed identity, replay and per-ply WDL audits, with no illegal moves, errors, forfeits, unfinished games or deadline misses. Opening/training exposure was not independently excluded. These are matched system results under a specific protocol, not an isolated causal test of training duration.
Training and input
Fresh random initialization with AdamW; clean game history, one token per complete UCI move (one ply), including promotions. The 86,966,016-parameter transformer has 12 layers, 12 heads and width 768, with a 512-token context and 1,975 vocabulary entries. No masking and no outcome labels in inputs. Move outputs and fixed-White WIN/DRAW/LOSS outputs are separately normalized from the shared transformer and tied output matrix.
This is the final snapshot of the same fresh CM-13 run, not a new CM number or a continuation from CM-10. The eight-hour experiment preserves the earlier comparison against CM-10; its 54% result and this newly matched 53% reference are different evaluations.
Exact identity and serving boundary
Source SHA-256 2db161c6c8396388565f414123ccbe413deec2d0a85faee530ea8b5913a16ae4; inference export SHA-256 745b457f7751d912b878fd5242243e058ad2da2fe3eae1b002fbc40d2ffd0cbc. Lichess release: 745b457f7751-35bdb54f6540. The browser uses an FP32 ONNX conversion of those weights, with its own hash and parity evidence.
Evaluation and identity metadata · Browser implementation and validation · Previous snapshot identity. Promotion does not claim that unassisted browser play reproduces the Stockfish-assisted benchmark.