LittlePlyModel + harness

Act Two / paused indefinitely / updated 13 November 2026

Let the model ask for what it needs.

We trained a model to interact with a chess VM, measured where it works and where it fails, and outlined a possible reinforcement-learning continuation. That continuation has not been run.

Act Two is indefinitely paused. No GRPO training has been run and no reinforcement-learning improvement is claimed. The Act One CM-14 agent remains the live Lichess player.

The change in one minute

In Act One, the harness asks the model for predictions and decides what to do next. In Act Two, the model emits moves and simple tool-call tokens. A virtual machine (VM) checks them and returns observations to the model. The board stays authoritative in the VM; the harness still manages the clock and the connection to the game.

ACT TWO / EXPERIMENT

The model uses a small tool interface.

Ask for the board, propose a move, learn from feedback.

The checkpoint we reached

PVM-01 recovery training started from the original CM-14 weights, followed by an eight-hour supervised recovery run and an additional eight-hour continuation on the same data. The evaluated final checkpoint is step 471,623. These were supervised runs, not self-play training. The earlier random-initialization experiment is not the origin of these final results.

The VM returns observations and attributed rejections. There is no legality mask in the following two evaluations. After three rejected move proposals on a turn, the harness inserts a seeded random legal move into history. That safeguard prevents getting stuck; it is not a model-generated move.

Against uniform random legal play

84 wins / 16 draws / 0 losses, out of 100 games. All reached chess endings: 84 checkmates and 16 stalemates. No scored VM failures or incomplete games.

Random fallback: 512 / 3,818 PVM turns (13.4%). This measures the model-plus-fallback system, not fully autonomous legal proposals.

Same-checkpoint self-play

32 White wins / 36 Black wins / 32 draws, out of 100 games. Independent seeds and private VM histories on each side; no teacher and no learning during games.

All reached chess endings: 68 checkmates, 24 stalemates, four fivefold repetitions and four insufficient-material draws. Zero scored incompletes.

Self-play reliability, not just the result

  • Random fallback: 1,103 / 11,093 turns (9.94%); White 520 / 5,569 (9.34%), Black 583 / 5,524 (10.55%).
  • 5,341 rejected proposals; autonomous recovery on 1,555 of 2,658 turns that encountered a rejection.
  • 1,250 board queries and 472 legal-move queries across both sides.
  • The final replay and move-attribution audit passed. Two infrastructure-interrupted self-play attempts were preserved separately and retried with the same seeds; the random-opponent run also preserved one interrupted attempt.

A near-even White/Black split is not evidence of improved strength or proof that chess is balanced. Both sides use the same checkpoint. The useful finding is that the system can produce complete games with varied outcomes, while fallback dependence remains substantial.

These are offline VM-harness results, not results for the full live CM-14 search harness. Weak-engine and teacher-intervention explorations did not establish a training improvement. Teacher-assisted victories would measure the assisted system, not learned behavior.

A turn, seen from both sides

The output becomes the next input.

A model proposes. The VM checks. Its response joins the context the model reads next.

Illustrative opening sequence, not a training result or a recorded game. The recovery path is shown deliberately; a legal first proposal skips it.

Standard starting position♜♞♝♛♚♝♞♜♟♟♟♟♟♟♟♟♙♙♙♙♙♙♙♙♖♘♗♕♔♗♘♖
Before White to move
One real move.

Queries and rejected proposals do not move a piece.

e2e4
After the accepted move e2e4♜♞♝♛♚♝♞♜♟♟♟♟♟♟♟♟♙♙♙♙♙♙♙♙♖♘♗♕♔♗♘♖
After Black to move
  1. 01

    Ask for the board.

    MODEL → VM · OUTPUTBOARD_QUERY

    “What is the position?”

    VM → MODEL · NEXT INPUTBoard snapshot

    Pieces + side to move + rule state.

    The VM supplies the snapshot. The model does not have to invent the board.

  2. 02

    Keep the failed attempts.

    MODEL → VM · OUTPUT
    e2e5e2e6e2e7
    VM → MODEL · NEXT INPUT
    [e2e5, REJECT][e2e6, REJECT][e2e7, REJECT]

    Three separate exchanges. Each VM response contains two tokens: the attempted move, then REJECT. Append that returned pair once; do not append the proposal separately. The board stays unchanged.

  3. 03

    Ask for help, rather than guess again.

    MODEL → VM · OUTPUTLEGAL_QUERY

    A learned request after three rejections, not a forced evaluator step.

    VM → MODEL · NEXT INPUTLEGAL_QUERY + complete list

    From this unchanged starting position. Complete list below.

    The VM lists legal options, not the best move. The model still chooses.

  4. 04

    Choose a move that can be played.

    MODEL → VM · OUTPUTe2e4

    The model selects the two-square pawn move.

    VM → MODEL · NEXT INPUT[e2e4]

    Move accepted. The VM advances the board; Black is now to move.

    The accepted move appears once in history. The opponent’s real reply, for example e7e5, is appended next.

One history, updated only from VM responses.

The proposal is a request, not a second history entry. The harness appends the VM's returned tokens once, then builds the next model input from that updated history.

  1. Context before the attemptH = [earlier events, BOARD_QUERY, snapshot]
  2. Model proposese2e5Send to the VM. Do not append separately.
  3. VM returns two tokens[e2e5, REJECT]No board change.
  4. Context for the next predictionH + [e2e5, REJECT]One attempted-move token, one rejection token.

After all three rejected attempts: H + [e2e5, REJECT, e2e6, REJECT, e2e7, REJECT]. White is still to move in the starting position. Query responses and accepted moves extend the same history; only accepted moves advance the board.

Inspect the input / output sequence

Readable labels for the protocol, not a literal tokenizer dump. BOARD_QUERY stands for the board-query command; snapshots and legal lists are VM-supplied observations. “Move accepted” above explains the state change, not an extra ACCEPT token.

Model output
BOARD_QUERY
VM input
[BOARD_QUERY, snapshot] appended once. Human-readable FEN equivalent:rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
Model → VM, three times
[e2e5, REJECT] [e2e6, REJECT] [e2e7, REJECT]
Model output
LEGAL_QUERY
VM input · complete legal list
[LEGAL_QUERY, LIST_BEGIN, all 20 legal moves, LIST_END] appended once.
Complete list contents
Model proposal
e2e4
VM response appended once
[e2e4] · legal, committed once
Next game input
e7e5 · illustrative opponent reply, not generated by this model

What must be learned: asking for the board, choosing when to request legal options, and recovering from feedback. This illustration does not establish that PVM-01 already does these reliably.

The interaction to learn

  1. Ask. A board-query token requests a fixed-width, FEN-like snapshot from the VM.
  2. Play. A move token is checked. A legal move advances the board and appears once in history. Otherwise REJECT leaves the board unchanged and the model retries.
  3. Refresh. A new snapshot replaces dependence on old history before the context becomes incomplete. The opponent's real move is appended after a committed move.
Token structure and context limits

The earlier design used a 1,024-token context; the evaluated recovery and continuation checkpoint used 512 tokens with a full snapshot plus recent moves and feedback. The supervised data included refresh and partial-history examples; evaluation must measure actual autonomous refresh rather than assume it works. The model must learn to request refresh, rather than relying on an evaluator to force it. Snapshot tokens are VM observations, not text the model must invent.

Start/end markers delimit games. Move, query and outcome tokens share one next-token prediction interface. The outer harness can insert a win/draw/loss query in an isolated diagnostic context; the agent does not call it during normal play. There is no separate WDL head.

A mixed dataset, not a scripted lesson order

Human games and engine-analyzed positions teach next moves. Derived interactions teach board queries, partial-history play, refresh and recovery from illegal proposals. Outcome examples teach the harness-only diagnostic query.

Examples are prepared and checked offline, then randomly mixed with controlled proportions. The optimizer reads cached sequences; it does not call the VM during training. Related examples share family-level exposure limits and held-out boundaries so repetition is not mistaken for new data.

What alternative engine lines contribute now

They provide analyzed positions and next-move supervision. Multiple alternatives are not automatically negative examples, and no search trace is trained in this round. Using those alternatives to learn exploration is a later experiment.

What would count as progress?

  • Legal first proposals, and recovery after REJECT.
  • Autonomous board queries before the full snapshot falls out of context.
  • Sustained play with partial history and an accurate current board.
  • Useful diagnostic outcome estimates, compared with the outcome distribution.
  • Completed games and measured playing strength, not just lower training loss.

Each measure needs attempt counts, baselines and a distinction between model behavior and forced safeguards.

Later: learning to search through tools

The longer-term idea is to let the model generate short what-if lines, with legality checked on isolated VM boards, before committing a real move. A six-ply target is a starting point, not an achieved capability. Board queries and diagnostic WDL calls would stay outside hypothetical continuations.

What-if traces remain in context in the proposed design to preserve useful history and compatible cache prefixes. A host-injected DEADLINE would request a final move, with a separate hard clock limit enforced by the harness. These are unrun design questions, not demonstrated capabilities. The evaluated recovery recipe excludes what-if search and resignation.

Proposed only / not run

A possible next act: GRPO from self-play

The idea was to continue from the supervised PVM checkpoint using Group Relative Policy Optimization (GRPO), without adding an imitation-learning loss. The environment supplies rule-based outcomes; no learned reward judge is required. This is a roadmap for a future experiment, not a scheduled training run.

  1. Collect comparable groups. Freeze a rollout-policy snapshot. Sample several continuations from the same starting state, with the same opponent condition and resource budget. Keep private VM histories and record sampled-action probabilities, policy version, seeds and every intervention.
  2. Score and compare within each group. Score win/draw/loss from the acting side's perspective, and report protocol failures and fallback separately. Compute relative advantages from the group's rewards. If every member gets the same reward, that group supplies no outcome-based relative learning signal; a global 50% win rate alone does not ensure informative groups.
  3. Train only on the policy's own actions. Use a clipped policy-ratio objective with a reference-policy constraint as a starting recipe. VM observations, opponent moves and forced random fallbacks stay in history but are excluded from the policy loss. Credit assignment still needs care: a win rescued by the harness is not evidence that an illegal proposal was useful.
  4. Introduce stronger opponents gradually. Begin with self-play against frozen/current and older checkpoints. A tentative mix was about 75% self-play and 25% engine-opponent games, at the game level. Introduce C Sunfish at several measured strength/node settings, then Stockfish at varied strength levels as performance permits. The mix and promotion thresholds were not validated.
  5. Keep independent evaluation. Use held-out openings, paired colors and seeds, fixed engine budgets and older checkpoint opponents. Track win/draw/loss, incomplete games, raw legality, recovery, queries, fallback rate, WDL calibration and compute. Separate assisted-system performance from model-only behavior and guard against exploiting a narrow opponent.

Engine losses may expose weaknesses that self-play misses and reduce the risk of co-adaptation, but an all-loss group can still have zero outcome variance. Opponent diversity is a hypothesis to test, not a guarantee against training collapse. Requested engine nodes can overshoot, so actual work must be recorded.

Teacher intervention and imitation were explored as alternatives, but the preferred next experiment was pure GRPO. No teacher moves should be silently relabeled as samples from PVM. No GRPO gains, Elo improvement or successful training curriculum have been demonstrated here.