EXP-4 / completed / evaluation

EXP-4: A first 20-game evaluation

Measure how the CM-3 checkpoint behaves in a small, reproducible match with declared assistance.

Non-reasoning next-move model: complete game history from the standard start, three-token causal objective, no thought or search supervision. Follow data → tokenizer → training → harness → bot bridge.

Baseline

Sunfish classic at recorded commit e396948acbd80cdcbad3dccfb07e878fba9df8f4. Requested 1,000 nodes; reported work ranged from 1,000 to 11,422 nodes.

Data

All 20 games: ten hand-authored openings, both colors. No games selected by outcome; all 20 terminated by game rules.

Configuration

Exact recorded configuration

{
  "decoding": "full legal-move joint argmax; raw probabilities separately logged; tactical override skips model",
  "draw_policy": "immediate symmetric python-chess outcome(claim_draw=True)",
  "games": 20,
  "mate_max_plies": 3,
  "mate_nodes": 50000,
  "mate_seconds": 1,
  "opening_provenance": "Ten hand-authored common opening sequences selected before play; no database/engine/test-set selection",
  "ply_cap_including_opening": 256,
  "seed": 20261028,
  "sunfish_commit": "e396948acbd80cdcbad3dccfb07e878fba9df8f4",
  "sunfish_requested_nodes": 1000
}

Compute

656 total system decisions: 651 model queries and five mate-helper overrides that skipped the model. Total recorded game time 410.58 seconds.

Results

Recorded metrics — protocols differ by experiment
Metric	Value
Wins	3
Draws	0
Losses	17
Mate-helper moves	5
Raw greedy legal queries	591

Replay all 20 games →

What this does not establish

20 games and one opponent setting do not establish Elo.
Played moves use joint argmax over legal moves; raw model legality is a distinct diagnostic.
Five forced-mate helper overrides are assisted decisions, not model predictions.
Sunfish actual node count exceeded the requested budget in some searches.
No model-only ablation was played for these 20 positions.
All three wins contained helper overrides. No model-only ablation establishes that those overrides were necessary to win. Helper use is 5/656 total decisions; raw-greedy legality is 591/651 model queries.

Provenance

Browse the recorded source snapshot →

Checkpoint SHA-256 6133648b2926b38542ee3ee1251a1ba111d61da9a8ea69d1c5f0559b09933643

Download experiment record (JSON)

Evidence: summary.json

Original artifact SHA-256 fe9fc2d40a5c718fd6eeafeaaded44220057e5e662597c89c07132f05424d28c

Selected public fields; private paths and process/account details excluded.

{
  "by_colour": {
    "black": {
      "draws": 0,
      "losses": 9,
      "unfinished": 0,
      "wins": 1
    },
    "white": {
      "draws": 0,
      "losses": 8,
      "unfinished": 0,
      "wins": 2
    }
  },
  "capped_games": 0,
  "completed_games": 20,
  "failed_games": 0,
  "mean_raw_legal_probability": 0.8775225006586275,
  "model_queries": 651,
  "overrides": 5,
  "raw_greedy_legal": 591,
  "seconds_since_start": 420.33469025799997,
  "sunfish_reported_nodes_range": [
    1000,
    11422
  ],
  "sunfish_requested_nodes": 1000,
  "tactical_unknown": 0,
  "terminal_games": 20,
  "timeout_games": 0,
  "total_game_seconds": 410.580232404
}