ARC Observatory
Experiment / Direct-grid distillation

Learning to reason.
Learning to solve.

Follow Qwen3.5-9B as it learns from Qwen’s hinted reasoning and verified grid answers. One fresh adapter, measured across 100 epochs.

Qwen3.5 · 9BLoRA rank 128Fresh SFT + formatting lossGrid weight 5×Batch 128
Connecting

Fetching the latest snapshot…

Epoch progress
—
Waiting for metrics
Total SFT loss
—
Reasoning + 5× grid + formatting
5× grid-answer NLL
—
Scaled numerator · teacher-forced
Raw grid NLL × 5 · applied training weight
Training tasks visited
—
Unique tasks, not tasks solved
1,099 train · 20 held out

Inside an answer

Compare the baseline, selected checkpoint and correct grids. Track each task across evaluations.

ACTUAL ROLLOUTS
Loading evaluation progress…
Completed responses will appear here as evaluation runs.

Total SFT loss

Actual optimized objective · one curve

Why can grid loss be tiny? During SFT, the model sees the teacher’s reasoning and the correct preceding grid tokens. Predictable brackets, commas and repeated colours lower this average. Independently generated answers below measure actual solving.

5× grid-answer loss

Grid NLL × 5 · dedicated vertical scale · optimizer steps

APPLIED WEIGHT: 5×
● 5× grid NLL · batch

This plots 5 × raw grid NLL directly. The total SFT objective separately normalizes its weighted token sums; that additive contribution remains in the loss breakdown below.

Reasoning loss contribution

Normalized contribution to the SFT objective · one curve

Formatting loss contribution

½ syntax NLL + ½ closing-boundary NLL · one curve

Checkpoint results

Every 2 epochs · matched prompts and seeds · direct grid checks

GENERATED ANSWERS

What changed from baseline?

The same 20 training tasks, before and after learning