# CM-15 outcome prediction: 100 Sunfish games

CM-15's WDL predictions contain useful information about eventual results, especially late in games, but they are not reliably calibrated across this cohort. This is evidence about predicting the played game's result, not proof that the model correctly evaluates objective chess strength.

## Scope and integrity

Frozen checkpoint SHA256 `fae796c2f7339650a1650066e7ff889c8f84df2acf577b122a6f9e98dc5030d5`, update 339215. The cohort finished 64 wins, 9 draws and 27 losses from CM-15's perspective, with the frozen live Stockfish mate/conversion helpers. All 100 games passed replay and legality audits.

The separate outcome query token 1973 was appended to clean chronological move history; no outcome label was supplied. Original output order is fixed White win/draw/White loss. All analysis and chart probabilities are **bot-relative**, swapping win and loss when CM-15 is Black. Full-vocabulary WDL mass and conditional WDL were retained separately. Conditional distributions sum to one; full-vocabulary WDL mass ranged from 0.999257 to 0.999998 (median 0.999929). Thus normalization is not concealing substantial mass outside WDL.

All 8,045 positions have complete ply coverage, including opening moves, both players' actual moves, and final positions. Isolated-prefix versus batched query maximum logit error was 0.0000179. The 100 terminal positions retain raw model predictions but are **excluded from every predictive metric and reliability bin**. There are 7,945 nonterminal rows. Terminal raw predictions appear only as final points in the per-game trajectory PDF.

## Predictive quality

Multiclass Brier is the sum of three squared probability errors, range 0–2; log loss uses natural logarithms, with a 1e-15 numerical floor. Lower is better. Each game contributes equally after averaging its positions, preventing long games from dominating.

| Window | Positions | Brier | Log loss | Mean probability of actual result | Argmax accuracy |
|---|---:|---:|---:|---:|---:|
| All nonterminal | 7,945 | 0.468 | 0.784 | 54.0% | 61.9% |
| Early third | 2,684 | 0.606 | 0.980 | 41.1% | 46.3% |
| Middle third | 2,650 | 0.488 | 0.821 | 53.0% | 62.6% |
| Late third | 2,611 | 0.305 | 0.543 | 68.3% | 77.2% |

Early means ply/final game plies <1/3; middle [1/3,2/3); late >=2/3, excluding terminal positions. These are retrospective progress thirds, not opening/middlegame/endgame board classifications. They use eventual game length for analysis only, never as a model input.

The leave-opening-pair-out frequency baseline estimates W/D/L from the other 98 games. Its Brier is 0.518 and log loss 0.874. It is cross-fitted, rather than estimated and evaluated on the same games. The model's all-position Brier advantage is 0.050, but its paired 95% interval spans no advantage: model minus baseline -0.050 [-0.120, +0.020]. Late-stage difference is -0.213 [-0.306, -0.119]; early-stage difference is +0.088 [+0.030, +0.144], favoring the frequency baseline early.

All-position model Brier 95% interval: [0.413, 0.530]; log-loss interval: [0.687, 0.895]. Intervals use 10,000 bootstrap resamples of the 50 opening pairs, retaining the two games together. This also preserves every game's correlated plies. Baseline predictions are held fixed during bootstrap; intervals describe this cross-fitted evaluation, not full uncertainty from retraining the frequency estimator in every resample.

Fixed-ply checks provide a second view: Brier at ply 0 = 0.578 (100 games), ply 16 = 0.619 (100), ply 40 = 0.488 (91), and ply 80 = 0.281 (36). Later fixed-ply samples exclude games already over and therefore have survival/composition bias. They do not independently establish a causal improvement over time.

## Reliability and confident mistakes

Ten fixed bins per W/D/L class cover [0,.1), ..., [.9,1]. `metrics.json` includes each bin's row count, distinct game count, mean predicted probability, observed frequency and clustered confidence interval. Reliability uses inverse nonterminal game-length weights; these are not independent-position sample sizes.

Examples:

* Win predictions in [.4,.5): mean 44.9%, observed win frequency 71.5% [62.3%,79.6%], across 2,382 rows and 100 games. The model underestimates this cohort's win tendency in this range.
* Win predictions in [.9,1]: mean 94.4%, observed 93.7% [83.2%,99.9%], across 660 rows but only 44 games. This is encouraging aggregate agreement, not a guarantee for an individual position.
* Draw predictions above .9 cover only 32 positions from four games. There are only nine drawn games in the whole cohort, so draw calibration is poorly constrained. A smooth reliability line should not be mistaken for strong evidence in sparse bins.

Selected extreme mismatches (bot-relative, selected retrospectively after ply 16):

| Game / ply | Prediction | Actual outcome |
|---|---|---|
| F100-097 / 57 | Win 98.55%, draw 0.71%, loss 0.74% | Loss |
| F100-075 / 53 | Win 1.99%, draw 1.40%, loss 96.61% | Win |
| F100-072 / 49 (Black bot) | Win 93.53%, draw 1.84%, loss 4.63% | Draw |

These are confident errors about eventual outcomes. They do not prove the original positions were objectively evaluated incorrectly: either player can subsequently blunder, and helpers can change the trajectory. No independent engine adjudication of these examples was performed.

## Interpretation and limits

The average trajectory separates eventual wins from losses increasingly late in games; late predictive scores also beat the cross-fitted frequency baseline. Early estimates are weaker and poorly adapted to this bot-versus-Sunfish result distribution. The overall score improvement over the simple baseline remains uncertain.

This is a single 100-game, 50-opening-pair cohort, with repeated development-opening exposure, nine draws, and Stockfish-assisted play on the bot side. Predictions were replayed after games with causal masks, so they did not alter decisions. The labels describe those particular continuations, not optimal-play outcomes, calibrated Lichess odds, or a causal benefit of outcome training. No comparison with previous-model WDL calibration was run. There is no evidence here to override the weaker Sunfish/head-to-head playing results or justify promotion.

Artifacts: `overview.png` / `overview.pdf`, `all-trajectories.pdf` (all 100 games), `metrics.json` (full statistics and reliability bins), raw `positions.jsonl`, PGNs, audits, checkpoint/config identities and the reproducible `calibrate.py` script. The raw JSONL uses fixed-White probabilities; charts and analysis use bot-relative probabilities.

Training remains on hold. No weights, live deployment, games or model inference were changed during this analysis.
