EXP-5 / completed / training

EXP-5: Training on a broader corpus

Can a broader Lichess corpus improve next-move imitation within a fixed four-hour budget?

Non-reasoning next-move model: complete game history from the standard start, three-token causal objective, no thought or search supervision. Follow data → tokenizer → training → harness → bot bridge.

Baseline

Fresh random initialization; expanded-validation NLL 5.029582. This run was trained from scratch, not continued from the CM-3 checkpoint.

Data

Aprilâ€“September 2026 Lichess: 4,222,117 unique retained games. Train 4,137,488 games / 943,591,445 targets; validation 42,223 / 9,615,601; test 42,406 / 9,644,506 (not model-evaluated). Both recorded ratings >=2000; base+40Ã—increment >=480 seconds; completed standard-start legal games; no BOT label; exact game-ID/full-sequence deduplication. No left crop.

Configuration

Exact recorded configuration

{
  "dataset_manifest_sha256": "0c5097b4793fd54ad31dd2b2c4220741371aa59c8be1cbe1dfc6d480fc451c94",
  "integrity_receipt_sha256": "620cd6a23a9e05379b70fcf541a0f7c18699b0e1e7576ec16ed7cc492e9ab894",
  "heldout_audit_sha256": "d8905b66852bdd9c09748439091f388bf29124466666cbe20ffd2485cfffa4e3",
  "runtime_sha256": {
    "loader_codec_smoke.py": "e009bb8303f9a0740bc77f6096386b8956ed1bc92eedd94a11ab1def0a95688e",
    "prepare_scaleup_plan.py": "74c6980429c4598980d5acd797d54b40780f6ef6132f57910810938630eb7280",
    "scaleup_cpu_tests.py": "43392c3b5dad91e9e3d15390aa1dca4cfc063950a8246ac2bab9fbd6cf0ac133",
    "scaleup_job.py": "2b71cbf6332aeb4ff0e7eb1b61e2d4fe314f9de8998e4ca6c33493c2acccc889",
    "scaleup_job_tests.py": "a819edefd535e660aec9fdf971944bde2b40dd66975270a25578d8361fc20ba8",
    "scaleup_plan.py": "0b8bd4d8174ba56ffb8780fbb77ee33ed519b19779ad2fc5f892dc70d95c2012",
    "scaleup_plan_tests.py": "33fdc882e4960fe33796afe9f8d1f25b1bde64168ccafb069e3b7a79594d79cc",
    "training_adapter.py": "1a0271cad107870345ef839378692d0d4036de44910e3e187c52c4e4d8244637",
    "training_collective_tests.py": "0ba6dd9a8bc2aa190c0f150ff85800c65271253eddf918671d5b57fec92975ed",
    "training_core.py": "140022ab400e6ef5f56fe4c3a1df28834afdc7f90e39b011968418a93795b856",
    "training_deadline.py": "11685a7e1078ac7e5f2fab07daad20316efb20f9cef56373e9cc5400e8b7c4ca",
    "training_deadline_failure_tests.py": "8daff953ef7d4ed4b3407b3aea28e2fef76baa7fad18f1c259254b425015b673",
    "training_deadline_tests.py": "f5ffd9b5cc0f74deff8b368602718ea8e44c8a4fa242171521814025996cccd3",
    "training_owned_launcher_tests.py": "5533574138545b45a224a84fa9577e463661c204864dc4956e768d29892c47ba",
    "training_terminal_tests.py": "5135821307b0b24e082f5767dc16779008beb5ec71910d18f000ecfd5bcd73a2",
    "training_tests.py": "ebc212cd288f63e88ed5f8986d80652584bc796de3606130322041b341b6ee54",
    "train_chess.py": "d1ead13673f9b4e613415ca7d0904d68bda0248bbdbf06d3736c2c8138257799",
    "verified_training_data.py": "fa00eb60a7c4c2061706ef94016f52bdef6cc04cbb2492ce8178dc02568fc71b",
    "chessdata/convert.py": "12dd6ae517a0f857684e4f9273f4a27168d9080b5f20e9260c0355053ff391c1",
    "chessdata/limits.py": "4d5cbc5d171068aff56104eae6c952848ea9c04751159b75942d1b99c9507311",
    "chessdata/pgn.py": "6f81cc6318220626b7f2d23ea623013360fab13afb7f663455ef4b0d043e4951",
    "chessdata/reader.py": "b825cdbe517a4b65ac2891787740c970009535127623a728929e18e5d226cea5",
    "chessdata/rust_backend.py": "8edd2d2d887c6da24fcc951ac77090e13f6bcc0bb6f89bcf47a02c5e1fc3c7cb",
    "chessdata/tokenizer.py": "0db7085edec22885bd8c51aac4d6867afc04203ba913ef561be97213df019029",
    "chessdata/__init__.py": "f0156ddfe522b44c33d849bbbbbc07a44f52f39c762822a512e5e8c1e710cf40",
    "run-training.sh": "aaca0c9c2d2be452343cdeb9b91daa8af335ca7576f298b6c08969638975d6f0"
  },
  "micro": 4,
  "accum": 4,
  "bucket": 2048,
  "seed": 2026,
  "lr": 0.0003,
  "collective_timeout_seconds": 900,
  "legality_eval_games": 128,
  "minimum_device_free_bytes": 4294967296,
  "non_optimizer_reserve_seconds": 1200,
  "throughput_discount": 0.85,
  "planned_validation_checks": 6,
  "epoch_optimizer_updates": 129297,
  "train_prediction_targets": 943591445,
  "train_games": 4137488,
  "initialization": "from_scratch",
  "maximum_epochs": 1,
  "test_model_evaluation": false,
  "model": {
    "vocab_size": 128,
    "n_positions": 1024,
    "n_ctx": 1024,
    "n_embd": 768,
    "n_layer": 12,
    "n_head": 12,
    "resid_pdrop": 0,
    "embd_pdrop": 0,
    "attn_pdrop": 0,
    "use_cache": false,
    "bos_token_id": 69,
    "eos_token_id": 70,
    "pad_token_id": 71,
    "attn_implementation": "sdpa"
  },
  "parameter_count": 85940736
}

Compute

Two RTX 4090 GPUs. Whole job 3h 58m 04.401s; soft deadline stop. 75,558 updates, 2,417,856 unique games and 551,410,959 targets consumed: a partial first pass. Best checkpoint is earlier: step 72,198, 526,834,590 target exposures.

Results

Recorded metrics — protocols differ by experiment
Metric	Value
Best expanded-validation NLL	0.5739030
Final unique games consumed	2,417,856
Final prediction targets consumed	551,410,959

Validation learning curve (9 checkpoints)

Update	Validation NLL
8022	0.802028
16044	0.706241
24066	0.658581
32088	0.623070
40110	0.598451
48132	0.585372
56154	0.580607
64176	0.577153
72198	0.573903

What this does not establish

Best validation belongs to step 72,198, not the final step 75,558 weights.
Expanded-validation NLL is not directly comparable to the original smaller validation set; use the matched 7,000-game comparison.
No held-out test model evaluation, reasoning traces, Elo estimate or completed Elite continuation.
Foundation model-only weights and source are downloadable via the foundation guide. Six-month bulk data are not included; the older dataset download remains the historical September package.

Provenance

Browse the recorded source snapshot →

Checkpoint SHA-256 857bc93bfffb287dc67e2845a14c214a4691bc1a008dae1fa4d044e5fccc87d2

Download experiment record (JSON)

Evidence: Terminal verification

Original artifact SHA-256 7c1bfbf19d2e0ed7f14e59e8aaa892b1e064ab66610257802b30d1bf53023e4d

Selected public fields; private paths and process/account details excluded.

{
  "stopping_reason": "total_deadline_soft_stop",
  "step": 75558,
  "unique_games": 2417856,
  "prediction_target_exposures": 551410959,
  "checkpoints": {
    "best": {
      "sha256": "857bc93bfffb287dc67e2845a14c214a4691bc1a008dae1fa4d044e5fccc87d2",
      "step": 72198,
      "bytes": 1031498263,
      "targets_at_checkpoint": 526834590
    },
    "last": {
      "sha256": "d000224f697a891687a74a8cce871d60024af3c3e536ec220fd21567eb350dc9",
      "step": 75558,
      "bytes": 1031498263,
      "targets_at_checkpoint": 551410959
    }
  }
}