Map of this document
Mission & environment constraints
The environment is the BottleCapAI NoCap-Test hiring challenge: train GPT-2 124M on FineWeb to the baseline’s final validation loss, faster.
Reach val loss ≤ 3.3821 on the official disjoint val set, in less training wall-clock than the stock recipe on the same GPU class. Clock pauses during validation. One GPU only.
Rules: re-benchmark baseline on your hardware. F1_seal (rpp, torch 2.13) crossed 3.3821 at step 4736 = 232.0 min (3.866 h), final 3.3798, step_avg ~2939 ms. That is the only speedup denominator. Historical F1 was 229.1 min (provisional).
| Axis | Allowed / required by rules | Forbidden / trap |
|---|---|---|
| Data | Any filter/reorder of the given FineWeb tokens | New datasets outside the kit’s FineWeb stream |
| Model / loss | Architecture, loss, schedule changes (with valid prob. model at eval) | Copying Modded-NanoGPT / leaderboard primary ideas |
| Eval | Any seq length that scores the full val set; sliding reported alongside official | Changing val_tokens on real runs (it IS the metric) |
| Hardware | Any GPU; report relative speedup vs self-baseline | Treating absolute hours as comparable to the 4090 board |
| Clock (rules) | Training wall-clock only; harness pauses timer during validation; one GPU | Scoring validation time, multi-GPU, or absolute 4090 hours as our claim |
| Fairness (rules) | Any generic tuning we take, baseline gets too; re-benchmark stock recipe on our hardware | Unfair comparisons / stack advantages only on the challenger |
| Software (rules) | Official pin was PyTorch 2.8.0-dev (cu128); compile may be off if unstable — both runs | Silent env drift between baseline and challenger |
| Hyperparams | Schedule changes justified by underfit regime / matched steps | Pure hyperparameter search as the “idea” |
(1) Official repo baseline — 5.401 h on RTX 4090, defines target loss 3.3821.
(2) F1 / F1_seal — stock recipe on our A100-40; only a sealed F1 (same torch pin + GPU class as challengers) is a valid speedup denominator. Historical F1 (notebook-era) is provisional.
(3) m0_anchor / proxy control — 800-step reference for cheap A/B only; never appears in the official % claim.
Fair-clock forensic protocol
Competition rules require a honest relative speedup on our hardware. That is necessary but not sufficient for forensic-grade claims: unpinned images, compile warmup, and GPU-class noise can move wall-clock by several percent — enough to fake a “win” or hide a real one. This section states both the official rules and the extra methodology we enforce.
A. What the competition already requires
| Rule (NoCap-Test / BottleCapAI) | How we implement it |
|---|---|
| Reach val ≤ 3.3821 on official disjoint val, faster than baseline | Primary DV: first step where disjoint val ≤ 3.3821 |
| Re-benchmark stock baseline on your GPU; report relative % | F1_seal on A100-40 · torch 2.13 · \(T\)=3.866 h @4736 · DONE |
| Training clock only — timer pauses during validation | Use harness train_time_ms, not wall calendar time that includes val |
| One GPU | Modal A100-40GB only; never bare A100 (80GB trap) |
| Fairness: if you tune X for your run, baseline gets X too | Same compile flag, same precision, same pin on denominator and challenger |
| Official software pin (repo era): PyTorch 2.8.0-dev cu128; compile optional if broken | We diverged on Modal CLI (see below) — so we re-seal, we do not hide it |
| Negative results welcome; no copying Modded-NanoGPT primary speedups as the “idea” | Archive miss configs; kill gates pre-registered |
B. What we add for forensic accuracy (beyond the README)
CLI images used unpinned pip install torch. F2b / Wave-4 / indeea resolved 2.13.0+cu130; notebook F2/W3 were 2.8.0+cu129; historical F1 torch was never logged. Code now pins TORCH_PIN = "torch==2.13.0". Smoke writes /vol/env_fingerprint.txt (profile, torch, CUDA, GPU name, pin).
torch.compile first-iteration overhead and early step instability are included in \(T\) (train_time to first cross). We never start the official clock after a secret warmup that baseline did not get. Steady-state step_avg is reported separately with \(N_{warm}=200\).
\(T\) = train_time_ms at first disjoint val ≤ 3.3821.
\(N^*\) = that step index (sample efficiency).
step_avg = mean over \([200, N^*]\).
Speedup = \((T_{F1\_seal} - T_M) / T_{F1\_seal}\). Never claim from sliding/EMA alone.
Measured stack drift ~6% step-time (802 vs 753 ms) is the empirical floor until sealed CV is known. Single-seed \|X\| < ~6% → do not publish as win. 6–10%: sealed + 2 seeds preferred. >10% or ≥200 steps earlier: 1 seed OK for narrative + fingerprint. Old “<3%” rule was too tight (would call infra a win).
C. Torch audit (why indeea shows 2.13)
| Run | Path | torch recorded | Forensic note |
|---|---|---|---|
| F1 baseline (historical) | notebook | not in config | Provisional only — superseded by F1_seal |
| F1_seal (rpp) | CLI stock 4768 | 2.13.0+cu130 | SEALED \(T\)=3.866 h · val@4736=3.3815 · final 3.3798 |
| F2 / Wave-3 | notebook | 2.8.0+cu129 | Within-family comparisons only |
| F2b / Wave-4 | CLI modal_nocap | 2.13.0+cu130 | Same unpinned resolve day/family |
| indeea smoke / proxies | CLI same app | 2.13.0+cu130 | Not a cheat kit — same stack as F1_seal / F2b/W4; ordinal only |
| Going forward | all CLI profiles | torch==2.13.0 pin | Any pin change = new seal required |
D. Claim classes (what is allowed to publish)
| Claim | Needs sealed env? | Status after audit |
|---|---|---|
| Val ≤ 3.3821 hit/miss (quality) | No (log fingerprint anyway) | F2 miss · F2b hit SOUND |
| Wall-clock % vs historical F1 | Yes | F2b −2.9% RETRACTED (speed) — below ~6% infra floor |
| Within-app A/B (Wave-4 wd120 vs wd720) | Same pin family | SOUND within CLI 2.13 |
| Wave-3 parallel ~1.6% | Same notebook 2.8 family | WITHIN-FAMILY ONLY |
| indeea proxy ranking | Same pin on indeea | ORDINAL ONLY — not a board % |
| Published official speedup | F1_seal DONE + claim still both on rainbowpuffpuff | CLAIM NEXT |
E. Seal procedure (mandatory before any speed claim)
- Code pin: TORCH_PIN = "torch==2.13.0" in modal_nocap.py + indeea_proxy.py (done; log full 2.13.0+cu130 string).
- Smoke on clock account rainbowpuffpuff → fingerprint torch + A100-40.
- F1_seal on rainbowpuffpuff: full 4768 stock F1 — DONE · cross @4736 · \(T\)=3.866 h · final 3.3798 · torch 2.13.0+cu130.
- Research proxies on indeea = ordinal ranking only.
- Claim final also on rainbowpuffpuff vs that F1_seal — no published cross-account cardinal %.
- If claim is 6–10%, prefer 2nd seed; if <~6% single-seed, do not publish as win.
F. Meta AI forensic review (2026-07-11)
External skeptical methods review (Meta AI browser skill). Thread e5e853d3-e8e8-4afa-91f3-e8f525078f98. We adopt the fixes; we do not adopt factual errors.
| Standard | Verdict |
|---|---|
| Competition-legal plan | APPROVE — F1_seal, relative %, train_time, one GPU |
| Forensic / publish-grade after fixes | APPROVE-WITH-FIXES |
| Meta raw tone | “REJECT as written” on pure forensic grounds — accepted as pressure; path is fixes, not stop |
• Published % only on rainbowpuffpuff (F1_seal + claim)
• Noise floor ~6% single-seed (not 3%)
• Proxies = ordinal only; winner’s-curse awareness
• F2b −2.9% retracted for speed (quality hit stands)
• Same-team disclosure; publish failed proxies
• Muon needs attribution if shipped (Modded-NanoGPT adjacency)
• First-passage \(T\) is high-variance; single seed = case study
• “torch 2.13 does not exist” — false; we run 2.13.0+cu130 on Modal
• “~$2 / full run, 20 runs on $60” — false; F1 ≈ 3.8 h × ~$2.10 ≈ $8–10
• Mandatory 3× F1_seal before claim — budget-killing; one seal + optional method seed is plan
• Dual accounts are still good ops; they are not a free statistical lunch
No silent torch upgrades between denominator and challenger.
No published cross-account cardinal % — claim final on clock; indeea is ordinal only.
No single-seed win claim if \|X\| < ~6% until sealed CV says otherwise.
No “indeea won because 2.13” narrative — same CLI stack as F2b/W4.
No claiming from sliding/EMA; official metric is disjoint val only.
Include compile warmup in \(T\); report steady step_avg separately.
Writeup: one team, two workspaces; archive negatives; attribute Muon if used.
Source of truth: private docs/ops/FAIR_CLOCK.md + smoke env_fingerprint.txt.
What the agent is (and is not)
We call it an “agent” because it is a goal-directed, multi-phase system with memory, tools, and a control loop — not because it chatters. Humans set kill gates and recipes; Modal executes the DAG; the volume is durable memory; the kit is the tool surface.
log.csv (train/val/sliding/EMA), config.json, triage_stats.json, billing probes, volume listings. Every keep/kill is trajectory-based, not a single snapshot.
Data shards, features, scores, triage bins, priors, run dirs, LAUNCH markers, runs_logs.zip. Survives container death. Concurrent-writer safe (v2).
Train, score, triage, prior-build, summarize — each a Modal function with hard timeouts, scaledown windows, and idempotent skip rules.
Remote driver: phase DAG, spawn/fan-out, join. modal run --detach decouples local client; panic stop is one CLI command.
Playbook ideas behind flags (defaults = stock). Negative results are first-class outputs. Time-scale hparams re-derived at proxy length.
No tool-calling LLM in the loop. Humans write gates and interpret curves. The “agent” is the experiment runtime + method, not a conversational planner.
Layered architecture
GitHub layout — rainbowpuffpuff/nocap
Private working repo until email submission (synced 2026-07-11, commit family 17b9e95+). Mirrors brain (docs + literature), body (kit), hands (notebooks + indeea scripts), and memory (results logs — no multi‑GB weights in git).
Not in git (local / Modal only): multi‑GB *.pt, FineWeb bins, score npz, Modal tokens. Restore via volume_backup / CLI get-put.
Companion machine workspace: software_i_built/modal/ — full archives, Cloudflare deploy source (cf-pages-nocap-agent/), live pulls (pull_wave4_*, nocap_archive/).
Kit modules — tool surface
Design contract of the trainer: with all new flags at defaults, semantics match the stock baseline (model, loss, schedule, timing protocol). Every experiment is opt-in.
| Module | Role | Key I/O |
|---|---|---|
| train_gpt2_t4.py | Instrumented GPT-2 trainer (d12 / d2 smoke). RoPE, RMSNorm, optional SwiGLU, Canon, parallel block, MTP, loss shaping, prior, EMA, z-loss, sliding val. | bins → runs/<name>/ |
| get_data.py | Fetch FineWeb pre-tokenized chunks from HF into volume/local. | → data/*.bin |
| triage_features.py | Stage-A doc features (len, unique, top, rep8, digit, punct). | → features/*.npz |
| score_docs.py | Stage-B: per-doc mean CE under a trained ckpt; per-shard skip = resumable. | ckpt + bins → scores/*.npz |
| triage_select.py | Junk rules + optional score band + 10% random rescue + optional curriculum order. | → triage_*/fineweb_triage_*.bin |
| build_unigram_prior.py | Corpus unigram log-prior for fixed logit bias. | → unigram_prior.pt / prior_full.pt |
| eval_sliding.py | Boundary-aware sliding-window val (report alongside official disjoint). | ckpt → Δ vs disjoint |
| summarize_runs.py | Table over all run dirs; report zips logs without final.pt. | → console + zip |
| modal_nocap.py | Full experiment DAG as Modal App; phases idempotent; detach-safe orchestrator. | volume as bus |
Trainer flag groups (opt-in ideas)
--logit_prior · --ema_beta · --ema_start_frac · --warmdown_iters · --input_bin (triage_*) · --val_stride · --precision bf16 · --compile 1
--mtp_* · --loss_shape / --ls_* · --mlp swiglu · --canon · --parallel_block · --zloss
Data pipeline architecture
FineWeb10B pre-tokenized GPT-2 BPE: up to 50×100M tokens. Competition budget cap 5B; baseline consumes 2.5B. Our F2 path uses a curated subset scored and banded from 28 chunks.
| Artifact (scale verify, 2026-07-09) | Value |
|---|---|
| docs_total / docs_kept | 4,045,481 → 2,927,881 |
| tokens_total / tokens_kept | 2.796B → 1.922B (kept_frac ≈ 0.687) |
| Score band (mean-loss) | [4.1887, 5.1831] = pct 10–80 of 4,045,472 docs |
| scores/*.npz | 28 shards (expect 28) |
| triage_full bins | 20 shards · ≈3.58 GiB |
| prior_full.pt | ~198 KiB · unigram CE 7.6690 nats |
| Dominant rule_hit | score_band 1,213,639 · unique 33,465 · len 9,532 |
Scoring is offline like the baseline’s own pre-tokenization cost narrative: GPU prep is honest and disclosed in IDEA.md. Training then sees a higher-signal stream (~1.22 epochs of 1.92B curated tokens cycled toward ~2.5B consumed). Proxy isolation: w2_databand alone was −0.051 nats @400 vs raw stream.
Modal orchestrator — control loop
App name: nocap-wave2. Image: debian slim + torch==2.13.0 (pinned) + numpy/huggingface_hub/tiktoken. Volume: nocap version=2. GPU string: A100-40GB (never bare A100). Fingerprint on every smoke.
Phase catalog
Invocation shape
# local entrypoint only spawns the remote orchestrator — safe to close the terminal modal run --detach kit/modal_nocap.py --phase all modal run --detach kit/modal_nocap.py --phase scale --chunks 28 --band 0.10,0.80 modal run --detach kit/modal_nocap.py --phase f2 # panic / observe modal app stop nocap-wave2 modal app list modal volume get nocap runs_logs.zip ./runs_logs_modal.zip
Confirmed recipe brain
Every component proxy-validated at matched steps / hardware. Stack additivity ~70–100% at proxy scale.
| Lever | Mechanism | Proxy effect | Status |
|---|---|---|---|
| Perplexity-band curation | Keep docs in mean-loss pct 10–80 + junk rules + 10% anti-drift rescue | −0.051 @400 | SHIP |
| Unigram logit prior | Fixed corpus log-bias on logits; start ~7.7 not 10.8 | −0.047 @400 | SHIP |
| Shortened warmdown | Early LR decay wasteful while still underfit | −0.013 @800 | SHIP |
| Sliding-window eval | Boundary-aware val; reported with official disjoint | Δ 0.019@800 → 0.060 full (F1) | SHIP (dual-report) |
| EMA weights | β scaled to horizon; dual-eval at end | lag artifact at proxy | MAINTAIN free |
| Stack (data+prior+WD) | Composition for F2 | −0.056 raw / −0.074 w/ eval @800 | F2 RECIPE |
Killed with evidence (submission material)
+11% step time and worse loss @800. Consistent with small-model NLL cautions. Configs + curves archived.
~0 cost, never ahead (+0.007 @800). Val includes every token — down-weighting is a mid-run tax.
Big early lead evaporates by step 800; ends ≈ baseline (4.78).
−0.078 nats (largest training-side effect) but +30% step time naive → net loss. #1 FUTURE lead if fused ≤2% tax.
~1.6% faster, loss-neutral-or-better (−0.005). Out of F2 for composition risk only.
Baseline sits at exactly 20 tok/param (Chinchilla isoline). Resizing unstable sign, ≤2%.
Method — how the agent decides
- Pre-registered kill gates before each arm (RUNBOOK + playbook).
- Matched steps on identical GPU type when step times equal; matched wall-clock when they don’t.
- Paired within-run measurements where possible (immune to run noise).
- Re-derive every time-scale hyperparameter at proxy length (EMA-β lesson: 0.999 full-run ≠ 0.99 proxy).
- Gates on trajectories, not snapshots (sliding Δ grows through training).
- Negatives as first-class outputs — README invites them; we archive configs + CSVs.
- Effects < 0.01 CE need two seeds before keep/kill.
- Pin environment — torch==2.13.0 seal; historical drift 802 vs 753 ms/step (~6%). See fair-clock protocol.
- Compile + GPU warm on clock — \(T\) includes compile first-iter; no secret warmup; steady step_avg after \(N_{warm}=200\).
- Fingerprint every run — torch, CUDA, GPU name, pin, compile, seed, batch×accum×seq → env_fingerprint.txt.
- Noise floor ~6% — measured stack drift; single-seed \|X\| < ~6% is not a published win (Meta review 2026-07-11).
- Proxies ordinal only — 800-step ranks methods; published % = full-horizon \(T\) on clock account.
- Claim on clock — F1_seal + final both on rainbowpuffpuff; no cross-account cardinal %.
model d12 · batch 8 · grad_accum 16 · seq 1024 → 131,072 tok/step (¼ of full 524,288) · LR 4.5e-4 · warmup 128 · warmdown 160 (or 40 with EMA) · val every 100 · bf16 · compile 1 · zloss 1e-4 (F1 drops zloss).
Full finals: 4768 steps · batch 16 · accum 32 · LR 0.0018 · warmup 256 · warmdown 1024 stock / 256 stack · val every 128.
Cost safety, idempotency, serverless semantics
Serverless: no long-lived GPU. timeouts 1h (train) / 8h (finals). scaledown_window=10s. Worst hung arm ~$2.10; absolute multi-hang cap ~$15.
Safe re-run of any phase. score_docs skips per-shard .npz. prep skips prior/features/triage if present.
LAUNCH + log.csv freshness <15 min → fail loud instead of racing twin trainers (learned after a silent double score ~$2 tax).
Volume v1 vs v2: scripts request version=2; creating with default v1 breaks concurrent writers — recreate with --version=2 (or omit version pin when profiles already differ).
Cloudflare 403 on kit download: default Python-urllib UA blocked inside Modal; send curl UA from ensure_kit().
Image build free, cold start billed: smoke in-container 14s ≈ $0.008 GPU; actual bill ~$0.009 with pull.
Preemption + duplicate guard (Wave-4, 2026-07-10): Modal preempted w4_wd120 ~step 1589; restart saw fresh log.csv (<15 min) and raised RuntimeError: another instance appears live — orchestrator died, w4_wd720 never started. Guard is correct for true doubles; needs a preemption-aware path (resume / ignore LAUNCH after worker preemption).
Cross-account data: volumes never share. F2b on acct3 required CLI volume get/put of triage_full/ + prior_full.pt (+ scores/features insurance) from acct1 — pure transfer, no GPU.
Live state — 2026-07-11
| Milestone | Result | Status |
|---|---|---|
| Wave-1 (Colab T4) | fp16 parity, prior, sliding Δ growth, warmdown waste found | DONE |
| Smoke (Modal) | A100 lifecycle on profiles incl. indeea · ~$0.01 each; rainbowpuffpuff profile ready (not smoked yet) | DONE / READY |
| Wave-2 (Modal) | m0 4.7788 · stack 4.7232 · databand −0.051 · MTP/shape kill | DONE |
| Wave-3 (Modal, acct 3) | Canon −0.078 but +30% time · parallel banked · SwiGLU kill | DONE |
| Scale (acct 1) | 28 scores · prior_full · triage_full 1.92B tokens · band [4.19, 5.18] | DONE |
| F1 (acct 2) | Crosses 3.3821 @ step 4736 = 229.1 min · final 3.3795 · step_avg 2903 ms | DONE |
| F2 record (acct 1) | Final 3.4050 (never crossed 3.3821) · 235.0 min · step_avg 2957 ms · WD 256 too short | MISS |
| F2b recovery (acct 3) | Stack + WD 1024 · final 3.377309 · first cross @4736 = 235.6 min · step0 val 7.88 · ~3.95 h | HIT TARGET |
| Official speedup (retracted) | (13746s − 14139s) / 13746s = −2.9% · same step · torch confound + \|X\| < ~6% infra floor — retracted as speed evidence; quality hit still stands. Meta review 2026-07-11. | RETRACTED (SPEED) |
| Meta forensic review | Dual-account idea OK; cross-account cardinal % weak; noise floor → ~6%; proxies ordinal; APPROVE-WITH-FIXES · see §01b·F | ADOPTED FIXES |
| Wave-4 diagnostic (acct 2) | COMPLETE · app stopped · w4_wd120 final val 3.982 · w4_wd720 final val 3.993 · both ~34 min · step_avg ~845–850 ms · pull_wave4_20260711/ | DONE |
| rainbowpuffpuff (clock) | Role = F1_seal + published claim only · A100-40 + torch==2.13.0 · volume nocap · seal complete · claim next | READY FOR CLAIM |
| indeea / indeeaindi (R&D) | Full board + ablations ALL DONE · best muon_par800 4.218 · muon 4.236 · seed42 4.235 · par1600@1600 3.941 · claim: Muon+parallel | DONE |
| rainbowpuffpuff F1_seal | COMPLETE · 4768/4768 · final.pt · first cross @step 4736 val 3.381538 · \(T\)=13,918,668 ms = 232.0 min = 3.866 h · final val 3.379813 · step_avg ~2939 ms · torch 2.13.0+cu130 · app ap-bWq1… stopped | SEALED |
| Literature program | 72 objects (36 core + 36 tangential) · experiment cards · gap analysis G1–G15 | ARCHIVED |
Stock AdamW · raw FineWeb · zloss 0 · no prior/EMA · WD 1024 · A100-40 · torch 2.13.0+cu130 · seed 1337 · app ap-bWq1… stopped · final.pt on volume.
First cross ≤ 3.3821: step 4736 · val 3.381538 · \(T_{F1\_seal}\)=13,918,667.5 ms = 231.978 min = 3.866 h · step_avg 2938.91 ms.
Final: val 3.379813 @4768 · train_time 14,011,911 ms (3.892 h) · step_avg 2938.74 ms.
Same cross step as historical F1 (4736); seal wall is ~1.3% slower (232 vs 229 min) under pinned torch — use seal \(T\) only for published %. Logs: results/f1_seal/.
Recipe: triage_full + prior_full + EMA β=0.999 + warmdown 1024. Hit val target (3.377 ≤ 3.3821). Wall −2.9% vs historical F1 is retracted as speed evidence (stack confound + below ~6% infra noise floor). Quality hit stands. Official speed number only vs F1_seal \(T\)=3.866 h on rainbowpuffpuff.
wd120 final 3.982 beats wd720 final 3.993 by ~0.01 CE. Curves nearly identical early; wd720 was briefly better at steps 1800–2100 (−0.04 at 2100), then short anneal wins the endpoint. Effect size is small — anneal length is not a free +5% lever at this proxy. Does not overturn F2 (WD256 @ full 4768 miss): full-horizon short anneal still failed; proxy 2400 ≠ finals.
App ap-LuwPnb6EpePg92rZ6zEMk4 stopped. Logs: pull_wave4_20260711/.
1. Stay on A100-40 through end of campaign (no GPU-class switch; VRAM underuse accepted).
2. Win plan approved — executing. Full roadmap: §13b.
3. Phase A (rpp): F1_seal SEALED — \(T_{F1\_seal}\)=3.866 h @4736. Phase B: indeea all finished — best muon+par 4.218. Next: full claim Muon+parallel on rpp.
4. No official speedup claim until \(T_{cross} < T_{F1\_seal}\) (= 3.866 h) on rpp with |X| ≥ ~6% (or 2 seeds).
Literature & hypotheses (why these races)
Closed-loop agent + 72 literature objects in academic_lit/ (core BIB.md + tangential BIB_TANGENTIAL.md). Time-to-target decomposes as:
T_cross ≈ N_steps(data, opt, schedule, arch) × t_step(arch, systems) + T_fixed(compile, val)
What the 2026-focused literature implies
| Cluster | Key papers | Hypothesis we test |
|---|---|---|
| O · Optimizers | Muon scalable (2502.16982) · Newton–Muon (2604.01472) −6% steps / −4% wall on GPT-2 speedrun class · SOAP (2409.11321) | B-Muon: matrix-aware updates cut N_steps — promoted on indeea (−0.50 CE @800). Full dossier: §13c |
| S · Schedules | Anytime pretraining (2602.03702) · ScheduleFree+ (2605.19095) · EMA weights (2411.18704) · Wave-4: wd120 3.982 vs wd720 3.993 @2400 | B-Anytime: F2 short-WD miss = full-horizon issue; at proxy 2400, short vs long anneal Δ≈0.01 — prefer mild WD + averaging, not extreme short-only bets |
| D · Data | Prior-filter (2509.18577) · ADAPT online reweight (2605.05227) · proxy reliability (2512.24503) · FineWeb-Edu | B-PriorFilter / B-Mix / B-Curriculum: static PPL band hit quality not steps; try prior-only filter, mixes, order |
| A · Arch / aux | TOP (2508.19228) · Future summaries (2510.14751) · Parallel tracks (2602.07306) | MTP killed; parallel banked (~1.6%); aux2.0 only with strict step-tax gate |
| T · Tangential gaps | µP transfer · packing · mixing laws · FP8 · critical batch · softcap | G1–G15 fills: free t_step / better proxies / 5B budget use |
Keep: band curation, unigram prior, dual val, parallel, kill-gate discipline.
Kill: MTP, loss shaping, SwiGLU-at-finals, naive Canon, short-WD-only final, EMA-as-early-cross claim.
Park: fused Canon until step tax ≤3%.
Ranked experiments (indeea) — literature-backed
| # | Bet | Why now | Script | Est. $ |
|---|---|---|---|---|
| 0 | Val data seed | Need fineweb_val for official metric | 01_seed_val_data.sh | ~0 |
| 1 | AdamW control @800 curated | Denominator for all proxies on torch 2.13 | 10_proxy_control.sh | ~1–1.5 |
| 2 | Parallel on/off | W3 bank; free t_step (A4 lit) | 15_proxy_parallel.sh | ~2–3 |
| 3 | Schedule WD sweep | Anytime lit + Wave-4 anneal data | 12_proxy_schedule.sh | ~2–4 |
| 4 | Muon (after kit port) | Newton-Muon on GPT-2 class | 11_proxy_muon.sh | ~1–2 |
| 5 | Prior-filter / curriculum / mix | D3 / D7 / mixing laws | 13, 14 + cards | ~2–5 |
Full cards: academic_lit/CARDS/ · synthesis: academic_lit/SYNTHESIS.md · related work draft: RELATED_WORK.md
Dual accounts — clock vs research
Two active ~$30 Modal workspaces split by forensic role, not by GPU class. Both stay on A100-40GB + pin torch==2.13.0 until campaign end (VRAM ~10/40 GiB underused; we accept that for a stable clock).
Workspace rainbowpuffpuff. Clean denominator and official finals only. No proxy clutter on this volume.
Run here: volume create → smoke + fingerprint → raw FineWeb + val → F1_seal (stock 4768) → claim final on this same account when indeea nominates → 2nd seed if claim is 6–10%; never publish single-seed <~6% as win.
Do not run here: Muon/schedule A/B spam, half-finished proxies, Wave-4-style anneal sweeps.
Budget sketch (~$30): F1_seal ~$9–10 · claim final ~$9–10 · buffer/seed ~$10.
Isolated from Wave-4 / F2b. Curated seed already on volume. All risky method search lands here.
Run here: val seed → AdamW control @800 → parallel on/off → mild WD only → Muon when kit ready → at most one data-order bet. Rank vs control, not vs historical F1.
Do not run here: published % vs F1; second full F1_seal (that’s rainbowpuffpuff’s job).
Budget sketch (~$30): proxies ~$12–18 · reserve ~$10–15 for a compose final or hand recipe to rainbowpuffpuff.
What runs where (checklist)
| Work | Account | Est. $ | Status |
|---|---|---|---|
| Re-smoke + env_fingerprint.txt (pin 2.13) | rainbowpuffpuff | ~0.01 | DONE |
| Raw FineWeb + val for stock F1 | rainbowpuffpuff | ~0–1 | DONE |
| F1_seal stock 4768 (sole speed denominator) | rainbowpuffpuff | ~8–10 | DONE · \(T\)=3.866 h |
| Full claim final (Muon+par+stack) | rainbowpuffpuff | ~9–10 | NEXT |
| Val shard + readiness | indeea | ~0 | DONE |
| Proxy control · ix_control | indeea | ~0.4 | DONE · 4.731 |
| Parallel on · ix_parallel_on | indeea | ~0.4 | DONE · 4.585 · BANK |
| Muon · ix_muon | indeea | ~0.4 | DONE · 4.236 · PROMOTE |
| Compose / schedule / prior_band | indeea | ~2 | DONE compose≈par · sched park · prior 4.768 kill |
| Full compose / claim @4768 | rainbowpuffpuff | ~8–10 | ONLY IF PROXY WINS |
Retired / archive accounts (do not burn for new science)
| Profile | Past role | Now |
|---|---|---|
| acalincarol | Scale · F2 record | Archive / data source |
| stufflaters | F1 historical · Wave-4 | Done — no more anneal GPU |
| deeferentleeg | Wave-3 · F2b | Archive (quality hit, provisional wall) |
rainbowpuffpuff — clock commands (when approved)
# profile is local name = workspace rainbowpuffpuff MODAL_PROFILE=rainbowpuffpuff modal volume create nocap --version=2 MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase smoke # confirm /vol/env_fingerprint.txt → torch 2.13.0 + A100-40 MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase data --chunks 28 MODAL_PROFILE=rainbowpuffpuff modal run --detach modal_nocap.py --phase f1 # → runs/f1_baseline (or f1_seal name) = sole T denominator
indeea — research commands (when approved)
| Item | Status |
|---|---|
| Volume nocap v2 | CREATED |
| Curated seed (triage_full 20 · scores 28 · features 28 · prior) | SEEDED |
| Smoke A100-SXM4-40GB · torch 2.13.0+cu130 | OK |
| Readiness f2b_ok | TRUE (raw train optional) |
| Scripts indeea_scripts/*.sh + indeea_proxy.py | DRY-RUN READY |
# always dry-run first; --execute only when approved cd software_i_built/modal/indeea_scripts ./00_check.sh ./01_seed_val_data.sh --execute ./10_proxy_control.sh --execute ./15_proxy_parallel.sh --execute # mild schedule only — not extreme short-WD finals ARM=wd160 ./12_proxy_schedule.sh --execute MODAL_PROFILE=indeea modal run indeea_proxy.py --arm control --iters 800 --input curated
Published speedup: both \(T_{F1\_seal}\) and \(T_M\) on rainbowpuffpuff only (same pin, A100-40, compile warm in \(T\)). No cross-account cardinal %.
indeea ranks methods vs its own proxy control — ordinal only; a “win” is a nomination, never a board %.
Same team: writeup discloses two workspaces, one methodology (not two independent competitors).
Noise: single-seed \|X\| < ~6% is not a publishable win until sealed CV known.
indeea torch 2.13 = same CLI stack as F2b — not a cheat kit. Protocol: §01b · docs/ops/FAIR_CLOCK.md.
Legacy anchor #indeea redirects conceptually to §13 Dual accounts.
Win plan — best Modal path (approved · executing)
Goal: disjoint val ≤ 3.3821 with \(T < T_{F1\_seal}\) by a margin that survives ~6% noise, A100-40 + torch==2.13, claim on rainbowpuffpuff, story that is not “paste Modded-NanoGPT.” F2b proved quality stack alone does not win the clock — we need \(N^*\) and/or \(t_{step}\) levers under fair-clock rules.
Phase A (clock) COMPLETE: F1_seal on rpp · app ap-bWq1ZYnXFOfFe98SbgxhRs stopped · final.pt · first cross @step 4736 val 3.381538 · \(T_{F1\_seal}\)=13,918,668 ms = 232.0 min = 3.866 h · final 3.379813 · step_avg ~2939 ms · torch 2.13.0+cu130 · A100-40. Preempt attempt archived separately (not the seal).
Phase B (indeea) ALL FINISHED. Best @800: muon_par 4.218 · muon 4.236 · seed-robust. Phase D next: full claim Muon + parallel + curated/prior on rpp vs sealed \(T\).
Private doc: docs/ops/WIN_PLAN.md · fair clock: §01b.
Tier 0 — clock (mandatory)
| # | Work | Account | Est. $ | Status |
|---|---|---|---|---|
| 0a | Volume + smoke + fingerprint | rainbowpuffpuff | ~0.01 | DONE |
| 0b | Raw FineWeb + val (data --chunks 28) | rainbowpuffpuff | ~0–1 | DONE |
| 0c | F1_seal stock 4768 | rainbowpuffpuff | ~8–10 | DONE · \(T\)=3.866 h @4736 |
| B* | indeea full board + Muon ablations | indeea | ~$6–8 | ALL DONE |
Tier 1 — highest EV for a real win (implement then run)
| Rank | Bet | Code? | Why it can win | Where |
|---|---|---|---|---|
| 1 | Muon / Newton-Muon + AdamW embed/head | Kit port required | Only big untried \(N^*\) lever; GPT-2-class lit | indeea proxy → rpp claim |
| 2 | Compose: F2b stack + parallel (+ Muon if #1) | Parallel flags exist | Free \(t_{step}\) + known quality hit | rpp claim |
| 3 | Horizon-free / anytime + registered averaging | Schedule code | Explains F2/F2b; earlier first-cross | indeea → rpp |
| 4 | Prior-filter or raw/curated mix | Light CPU + proxy | Original data story; may cut \(N^*\) | indeea → rpp |
| 5 | Prior anneal (strong early → 0) | Small flag | Uses existing prior asset | indeea → rpp |
Tier 2 — ordinal screens (indeea)
| # | Action | Account | Est. $ | Status |
|---|---|---|---|---|
| 1 | muon_par800 | indeea | 4.218 | BEST |
| 2 | muon / seed42 / lr01 | indeea | 4.23–4.24 | seed-robust · lr0.01≈0.02 |
| 3 | muon_noprior / raw | indeea | ~4.26 | prior+curated help a little |
| 4 | parallel / compose | indeea | 4.58 | BANK systems |
| 5 | control | indeea | 4.731 | denom |
| — | muon_par1600 | indeea | 3.941@1600 | longer horizon OK |
| — | lr05 / prior / sched | indeea | 4.41–4.81 | kill/park |
Best @800: muon+parallel 4.218 · muon alone 4.236 · seed42 4.235 (not a fluke) · lr 0.05 hurts (4.409).
@1600: muon+parallel 3.941 — keeps falling (proxy→full transfer hopeful).
Claim recipe: Muon (lr≈0.02) + parallel + curated + prior on rpp vs F1_seal. Attribute Keller Muon.
Kill schedule/data fast if Δ < 0.01 CE; put remaining $ into Muon + one full claim. Proxies never publish board %.
Phased execution
# PHASE A — CLOCK (rpp) ~$10 MODAL_PROFILE=rainbowpuffpuff modal volume create nocap --version=2 MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase smoke MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase data --chunks 28 MODAL_PROFILE=rainbowpuffpuff modal run --detach modal_nocap.py --phase f1 # PHASE B — SCREEN (indeea) while coding Muon ~$8–12 cd indeea_scripts ./00_check.sh ./01_seed_val_data.sh --execute ./10_proxy_control.sh --execute ./15_proxy_parallel.sh --execute # + implement Muon in train_gpt2_t4.py # PHASE C — MUON DECISION (indeea) ~$3–5 after kit ./11_proxy_muon.sh --execute # unblocked only after port # PHASE D — CLAIM (rpp) ~$9–18 # if Muon promotes: Muon + parallel + stack # else: parallel + stack + best data/schedule nominee # if |X| in 6–10%: 2nd seed; if <~6%: do not publish as win
Skip (do not burn credits)
- Extreme short-WD finals · re-score whole FineWeb for another static PPL band
- Historical F1 as denominator · cross-account published %
- Single-seed <~6% “win” · torch/GPU shopping · blind Modded-NanoGPT dump
- 3× F1_seal “for science” without a claim plan
Seal clock → implement Muon → proxy on indeea → if promote, full claim = Muon + parallel + proven stack on rpp vs F1_seal. Fallback if Muon fails: anytime + prior-filter/mix + parallel compose.
Deep dive: §13c Muon (theory · kit · full indeea board · claim recipe · attribution).
Muon — theory, kit, indeea board, claim
This is the campaign’s main optimizer bet. After data+prior+EMA+long warmdown (F2b) hit quality but not earlier first-cross, we needed a lever that cuts steps-to-target \(N^*\), not just step time. Muon is that lever at proxy scale. Full claim still waits on sealed \(T_{F1}\) on rainbowpuffpuff.
Hybrid Muon on 2D hidden weights + AdamW on embed/head reaches val@800 ≈ 4.23–4.24 vs AdamW control 4.731 (≈ −0.50 CE), is seed-stable, prefers lr ≈ 0.01–0.02, stacks slightly with parallel (4.218), and at 1600 steps still falls to 3.941. Not a board % — nomination for a full 4768 claim.
What Muon is
Muon treats each 2D weight matrix as a matrix, not a bag of scalars. It runs Nesterov-style momentum on the gradient, then approximates a polar / zero-power map via Newton–Schulz iteration so the update is closer to an orthogonal direction in spectral geometry. Scale factor \(\sqrt{\max(1,\,m/n)}\) restores a sensible magnitude after NS.
AdamW is excellent and remains our embed / lm_head / 1D path. Hidden Linear weights are low-rank-ish operators; matrix-aware updates often reduce the number of steps to a CE target on GPT-2-class runs — exactly the \(N^*\) gap F2b left open.
Standard practice from Keller Jordan / community Muon: do not Muon the embedding or lm_head; keep AdamW there with its own LR group.
Muon LRs are not AdamW LRs. Our board: 0.01 ≈ 0.02; 0.05 hurts (4.409). Default claim: --muon_lr 0.02.
Kit uses the standard 5-step NS coeffs (3.4445, −4.7750, 2.0315) in bf16, same family as public Muon implementations.
Why Muon for NoCap (campaign logic)
| Fact from earlier waves | Implication |
|---|---|
| F2 miss: short WD @ full horizon never crossed 3.3821 | Anneal length is fragile; not our primary \(N^*\) bet |
| F2b hit quality (3.377) but first-cross @ same step 4736 as F1 | Data+prior+EMA+long WD ≠ earlier passage time |
| Wave-4 @2400: WD120 vs WD720 Δ≈0.01 CE | Anneal is a small dial at proxy — park as main strategy |
| Parallel ~1.6% faster step @ historical proxy | Bank \(t_{step}\); cannot alone open a ≥6% win |
| Literature O1/O2: Muon / Newton-Muon cut steps on GPT-2 speedrun class | Highest-EV untried lever → kit port → indeea board |
Pre-registered card: private docs/academic_lit/CARDS/B-Muon.md — promote if proxy dominates AdamW early; claim only with sealed fair-clock \(T\).
Literature we cite (not a paste of Modded-NanoGPT)
| ID | Paper / note | What we take | What we do not take |
|---|---|---|---|
| O1 | 2502.16982 · Muon is Scalable for LLM Training | Matrix Muon on hidden weights; hybrid with AdamW; scale facts | Whole speedrun repo as submission body |
| O2 | 2604.01472 · Newton–Muon | Second-order motivation; reported −6% steps / −4% wall on GPT-2-class | Unverified port before hybrid Muon ships |
| O3 | 2602.21545 · Better Muon + norm | Optional later refinement if hybrid plateaus | Extra knobs on first claim |
| Blog | Keller Jordan · kellerjordan.github.io/posts/muon/ | Algorithm clarity, NS intuition, hybrid grouping | Claiming their wall-clock as ours |
Competition writeup must attribute Muon (Keller Jordan et al.) and note Modded-NanoGPT adjacency if we ship this optimizer. We motivate via our F2b failure mode + sealed ablations — not “we downloaded a leaderboard script.” Meta forensic review: same rule (adopted).
Kit implementation (what we actually run)
Embedded single-device hybrid (no extra pip): zeropower_via_newtonschulz5, muon_update, SingleDeviceMuonWithAuxAdam, build_muon_adam_optimizer. Shipped via volume kit_override/ so Modal picks it up without republishing the whole tarball.
--optimizer {adamw,muon} · --muon_lr (default: follow --learning_rate; proxies often 0.02). LR schedule scales each param-group relative to its base (spectral vs AdamW). GradScaler path stays AdamW-safe; Muon path is bf16-friendly.
| Param group | Optimizer | Typical defaults in kit |
|---|---|---|
| Hidden 2D weights (use_muon=True) | Muon | lr 0.02 · momentum 0.95 · NS 5 · WD 0 |
| embed / lm_head / 1D biases norms | AdamW | lr ~3e-4 class · β=(0.9, 0.95) · eps 1e-10 |
# indeea proxy shape (ordinal) MODAL_PROFILE=indeea ./11_proxy_muon.sh --execute # or direct flags once kit override is live: # --optimizer muon --muon_lr 0.02 --iters 800 --input curated (+ prior) # claim shape (after F1_seal) on rainbowpuffpuff only # Muon + parallel + curated + prior · full 4768 · report train_time to first disjoint ≤3.3821
indeea protocol (how the board was run)
- Hardware / pin: A100-40 · torch==2.13.0 (same CLI stack as F2b / Wave-4 — not a cheat kit).
- Proxy length: 800 steps primary; one 1600-step horizon check on best compose.
- Base recipe: d12 · batch 8 · grad_accum 16 · seq 1024 · bf16 · compile 1 · val every 100.
- Denominator: ix_control AdamW + curated + prior → val@800 4.731 (step0 val ~7.88 from prior).
- Metric for ranking: disjoint val at matched steps. Never published as % vs historical F1.
- Kill gates (card): worse than AdamW by >0.02 CE; or step tax >10% without ≥0.03 better val; or NaN/explosion.
Full ordinal board (2026-07-11) — COMPLETE
| Rank | Run | Val@800 | step_avg | Call |
|---|---|---|---|---|
| 1 | ix_muon_par800 | 4.218 | 937 | BEST COMPOSE |
| 2 | ix_muon | 4.236 | 902 | PROMOTE strong alone |
| 3 | ix_muon_seed42 | 4.235 | 906 | SEED-ROBUST ≈ seed 1337 |
| 4 | ix_muon_lr01 | 4.231 | 947 | lr 0.01 ≈ 0.02 |
| 5 | ix_muon_noprior | 4.256 | 944 | prior helps ~0.02 |
| 6 | ix_muon_raw | 4.258 | 972 | curated helps ~0.02 |
| 7 | ix_parallel_on | 4.585 | 866 | BANK systems |
| 8 | ix_compose800 | 4.583 | 875 | ≈ parallel |
| 9 | ix_control | 4.731 | 878 | AdamW denom |
| 10 | ix_sched_wd160 | 4.734 | 929 | PARK |
| 11 | ix_prior_band | 4.768 | 925 | KILL |
| 12 | ix_sched_wd400 | 4.814 | 881 | PARK |
| — | ix_muon_lr05 | 4.409 | 902 | LR TOO HIGH |
| Longer horizon | Val@1600 | step_avg | Notes |
|---|---|---|---|
| ix_muon_par1600 | 3.941 | 835 | Still falling · claim-shaped · not a seal |
What the board taught us
~0.50 CE vs control at 800 steps — larger than parallel, schedule, or data ablations combined. Seed 1337 vs 42 is a coin flip (4.236 vs 4.235). lr 0.01–0.02 is flat; do not “turn it up.”
muon+par 4.218 vs muon 4.236 (~0.02 CE). Parallel alone 4.585 is a systems bank (often better step_avg on non-Muon arms). Ship both on claim.
Drop prior → 4.256; raw → 4.258. Keep F2b-style curated+prior for continuity and ~0.02 CE — they do not replace Muon.
prior_band kill (4.768). Schedule WD160/400 park. Muon lr 0.05 kill. Another short-WD final is still low EV after F2/W4.
Optimizer: hybrid Muon · --muon_lr 0.02 (or 0.01).
Systems: parallel block on (banked \(t_{step}\)).
Data / head-start: triage curated FineWeb + unigram prior (F2b lineage).
Where: full 4768 on rainbowpuffpuff only, after F1_seal lands.
Metric: \(T = \) train_time_ms at first disjoint val ≤ 3.3821. Speedup = \((T_{F1\_seal}-T_M)/T_{F1\_seal}\).
Publish bar: single-seed \|X\| ≥ ~6% (infra noise floor), else 2nd seed or no win claim. Fingerprint torch + GPU on both runs.
Fair-clock constraints (Muon-specific)
| Rule | Why |
|---|---|
| indeea board = ordinal only | Different workspace; proxies rank methods, they do not print board % |
| Both \(T_{F1\_seal}\) and \(T_M\) on rainbowpuffpuff | No cross-account cardinal speedup |
| Same torch==2.13.0 pin + A100-40 | Otherwise stack confound (see F2b −2.9% retraction) |
| Disjoint val only for first-cross | Sliding/EMA are diagnostic, not claim metrics |
| Attribute Muon in writeup | Forensic / scientific honesty; Modded-NanoGPT adjacency |
| Proxy→full transfer unproven until claim finishes | 3.941@1600 is hopeful, not a seal |
Status vs next action
| Step | Status | Notes |
|---|---|---|
| Card B-Muon + lit map | DONE | docs/academic_lit/CARDS/B-Muon.md |
| Kit port + volume override | DONE | muon_opt.py · --optimizer muon |
| indeea full board + ablations | DONE | Best compose 4.218 · logs under results/indeea/ |
| F1_seal on rpp | SEALED | \(T\)=3.866 h @4736 · final 3.3798 · results/f1_seal/ |
| Full claim 4768 on rpp | NEXT | Muon + parallel + curated/prior · beat 3.866 h by ≥~6% |
| Newton-Muon / Better-Muon refinements | LATER | Only if hybrid claim is close but short of ~6% |
Muon is no longer a literature wishlist item — it is a proxy-validated promote with seed and LR ablations, a clear kill list around it, and a single claim recipe. F1_seal is done (\(T\)=3.866 h). Remaining risk is proxy→full transfer and fair-clock variance on the claim run — not “does Muon beat AdamW at 800 steps?”
Future architecture (post-finals leads)
After F2b (−2.9% wall), Wave-4 anneal data, and the indeea Muon board: prioritize shipping the Muon claim under fair-clock, then horizon-free schedules if still short — not another short-WD final.
| # | Lead | Why it matters |
|---|---|---|
| 1 | Muon / Newton-Muon | Promoted on indeea (−0.50 CE @800 vs control). Next: full claim vs F1_seal. See §13c. |
| 2 | Anytime / ScheduleFree + averaging | Fixes horizon mismatch that killed F2 WD256. |
| 3 | Parallel attn+FFN (+ fuse) | Already ~1.6% faster @proxy; free basis points. |
| 4 | Fused Canon | −0.078 nats if step tax ≤3%; else park. |
| 5 | Prior-filter / mix / curriculum | Quality stack didn’t cut steps; try cheaper filters + order. |
| 6 | Packing / batch / FP8 | Tangential t_step systems (G3/G5/G6). |
How to run the agent
# once per workspace pip install modal modal setup modal volume create nocap --version=2 # cheap proof (~$0.01) modal run kit/modal_nocap.py --phase smoke # proxy DAG (~$4–6, 60–80 min wall) modal run --detach kit/modal_nocap.py --phase all # full curated dataset for F2 modal run --detach kit/modal_nocap.py --phase scale --chunks 28 # finals (per-account profiles — volumes do NOT share) MODAL_PROFILE=stufflaters modal run --detach kit/modal_nocap.py --phase f1 MODAL_PROFILE=acalincarol modal run --detach kit/modal_nocap.py --phase f2 MODAL_PROFILE=deeferentleeg modal run --detach kit/modal_nocap.py --phase f2b MODAL_PROFILE=stufflaters modal run --detach kit/modal_nocap.py --phase wave4 MODAL_PROFILE=indeea modal run kit/modal_nocap.py --phase smoke # indeea research proxies (dry-run unless --execute) cd indeea_scripts && ./10_proxy_control.sh # add --execute when approved # readiness / transfer (CPU only — no GPU) MODAL_PROFILE=<name> modal run kit/modal_nocap.py --phase readiness MODAL_PROFILE=acalincarol modal volume get nocap triage_full/ ./triage_full/ MODAL_PROFILE=indeea modal volume put nocap ./triage_full triage_full -f # fetch evidence modal volume get nocap runs_logs.zip . python kit/summarize_runs.py runs
01–04 campaign docs → 05 this map → academic_lit/ · indeea_scripts/ · volume_backup/MANIFEST.md · LESSONS.md
modal_nocap.py · academic_lit (72 papers) · indeea_scripts · volume_backup · pull_20260711 · nocap_archive
One-line summary
A serverless FineWeb experiment agent with F1_seal DONE (\(T\)=3.866 h @4736 on rpp, torch 2.13), historical F1 3.82 h provisional, F2 miss, F2b quality hit (3.377) with −2.9% wall retracted as speed evidence (infra floor ~6%), Wave-4 done, indeea Muon board promote (4.218), dual accounts, Meta forensic review → APPROVE-WITH-FIXES. Next: Muon+parallel claim vs sealed \(T\).