Executive Technical Summary: By combining the Muon Matrix Optimizer, B32 Systems Microbatching, and Timing-Cheap Split-QKV Attention Projections, our composite training stack reaches the official GPT-2 validation loss threshold ($\text{val\_disjoint} \le 3.3821$) in an average of 3.544 hours across 3 independent random seeds. This delivers an 8.3% training time reduction compared to the sealed AdamW baseline (3.866 hours) on a single NVIDIA A100 GPU.
Replacing scalar coordinate scaling with 2D matrix momentum and 5-step Newton–Schulz polynomial updates.
Standard AdamW scales parameter coordinates independently. Muon treats 2D weight tensors as matrix operators. It computes SGD momentum matrix $G$, then updates weights using its nearest semi-orthogonal matrix $O = \text{Ortho}(G) = U V^T$:
Hidden 2D weight matrices (linear projections & attention layers) are optimized with Muon. 1D parameters (embedding tables, layer norms, and biases) remain on AdamW.
Instead of expensive SVD calls, Muon uses a 5-step Newton–Schulz polynomial iteration ($\phi(x) = 3.4445x - 4.7750x^3 + 2.0315x^5$) executed via native bfloat16 matrix multiplications.
Click the steps to see how 5 Newton–Schulz iterations flatten singular value variance into an orthogonal matrix update.
AdamW rescales parameters scalar-by-scalar using coordinate variance $\sqrt{v_t}$. In deep Transformer weight matrices, singular values span several orders of magnitude, causing slow gradient convergence along ill-conditioned directions.
Shampoo computes preconditioner matrix roots $(G G^T)^{-1/4}$, costing $O(d^3)$ operations and suffering from severe numerical instability in bfloat16 precision.
Muon replaces matrix square roots with 5 native matrix multiplication polynomial steps ($\phi(X) = a X + b X^3 + c X^5$), achieving exact semi-orthogonality ($U V^T$) at maximum GPU Tensor Core throughput.
Maximizing A100 Tensor Core throughput while keeping total tokens per update fixed at 524,288 tokens.
Changing microbatch size to 32 (with 16 gradient accumulation steps) is a hardware throughput optimization. It does not alter the loss function or optimization gradient math.
To guarantee a fair benchmark clock, the total token count per optimizer step is kept strictly constant:
Larger microbatches improve SM occupancy on NVIDIA A100 GPUs, halving gradient accumulation kernel launches and saving ~6% core step execution time.
Separating Query, Key, and Value matrices under Muon for independent orthogonalization.
Fused QKV ($W_{QKV} \in \mathbb{R}^{d \times 3d}$) forces Newton–Schulz to orthogonalize Query, Key, and Value projections as one block. Separating them into $W_Q, W_K, W_V \in \mathbb{R}^{d \times d}$ allows independent optimization.
Dual-polar split QKV improved validation loss (+0.015 CE), but introduced a +2.6% compute overhead per step. It was rejected as a NO-GO in the Failure Ledger.
Our candidate uses Timing-Cheap Split-QKV (muon_split_qkv=1, match_joint=0), preserving the validation gain with zero step-time tax.
Disjoint validation loss vs. training execution time in hours on 1x NVIDIA A100 GPU.
Scrub through training steps to verify cumulative execution time and percentage time saved.
Identify the first logged step where disjoint validation loss drops to ≤ 3.3821.
Divide cumulative training time (train_time_ms) by 3,600,000.
Compute relative time saved against the sealed AdamW baseline clock (3.866 hours).
Comparing candidate seeds, matched stock controls, and the sealed AdamW baseline.
| Run Identifier | Stack Configuration | First Cross Step | Validation Loss | Train Time (Hours) | Time Reduction (ΔT%) | Status |
|---|---|---|---|---|---|---|
| Candidate · Seed 1337 | Muon + B32 + Split-QKV | Step 4736 | 3.3741 | 3.553 h | +8.1% | Passed Gate |
| Candidate · Seed 2029 | Muon + B32 + Split-QKV | Step 4736 | 3.3726 | 3.567 h | +7.7% | Passed Gate |
| Candidate · Seed 4242 | Muon + B32 + Split-QKV | Step 4736 | 3.3733 | 3.511 h | +9.2% | Passed Gate |
| Baseline (F1_seal) | Sealed AdamW Baseline | Step 4736 | 3.3815 | 3.866 h | 0.0% (Ref) | Baseline Denominator |
| Stock Control · Seed 1337 | Muon + Joint QKV | No Cross | 3.3848 | N/A | N/A | Did Not Cross |
| Stock Control · Seed 2029 | Muon + Joint QKV | No Cross | 3.3840 | N/A | N/A | Did Not Cross |
Documenting failed hyperparameter attempts and discarded architectural variants.
Recomputing polar maps for dual-polar QKV introduced a +2.6% execution time overhead, negating optimizer gains.
Adding naive canonical block transformations added a severe ~30% per-step compute penalty.
Attempting a short 256-step warmdown ended at validation loss ~3.405, failing to reach the 3.3821 target line.
Five screening candidates were rejected due to early loss plateauing before reaching full sealed runs.
Definitions of core deep learning optimization and benchmark terminology.
Preview raw datasets online or click direct download links for code, logs, and technical docs.