Map of this document
Mission & environment constraints
The environment is the BottleCapAI NoCap-Test hiring challenge: train GPT-2 124M on FineWeb to the baseline’s final validation loss, faster.
Reach val loss ≤ 3.3821 on the official disjoint val set, in less training wall-clock than the stock recipe on the same GPU class. Clock pauses during validation. One GPU only.
Rules: re-benchmark baseline on your hardware. F1 on A100-40 crossed 3.3821 at step 4736 = 229.1 min (3.82 h), final 3.3795, step_avg 2903 ms. That is the speedup denominator.
| Axis | Allowed / required by rules | Forbidden / trap |
|---|---|---|
| Data | Any filter/reorder of the given FineWeb tokens | New datasets outside the kit’s FineWeb stream |
| Model / loss | Architecture, loss, schedule changes (with valid prob. model at eval) | Copying Modded-NanoGPT / leaderboard primary ideas |
| Eval | Any seq length that scores the full val set; sliding reported alongside official | Changing val_tokens on real runs (it IS the metric) |
| Hardware | Any GPU; report relative speedup vs self-baseline | Treating absolute hours as comparable to the 4090 board |
| Clock (rules) | Training wall-clock only; harness pauses timer during validation; one GPU | Scoring validation time, multi-GPU, or absolute 4090 hours as our claim |
| Fairness (rules) | Any generic tuning we take, baseline gets too; re-benchmark stock recipe on our hardware | Unfair comparisons / stack advantages only on the challenger |
| Software (rules) | Official pin was PyTorch 2.8.0-dev (cu128); compile may be off if unstable — both runs | Silent env drift between baseline and challenger |
| Hyperparams | Schedule changes justified by underfit regime / matched steps | Pure hyperparameter search as the “idea” |
(1) Official repo baseline — 5.401 h on RTX 4090, defines target loss 3.3821.
(2) F1 / F1_seal — stock recipe on our A100-40; only a sealed F1 (same torch pin + GPU class as challengers) is a valid speedup denominator. Historical F1 (notebook-era) is provisional.
(3) m0_anchor / proxy control — 800-step reference for cheap A/B only; never appears in the official % claim.
Fair-clock forensic protocol
Competition rules require a honest relative speedup on our hardware. That is necessary but not sufficient for forensic-grade claims: unpinned images, compile warmup, and GPU-class noise can move wall-clock by several percent — enough to fake a “win” or hide a real one. This section states both the official rules and the extra methodology we enforce.
A. What the competition already requires
| Rule (NoCap-Test / BottleCapAI) | How we implement it |
|---|---|
| Reach val ≤ 3.3821 on official disjoint val, faster than baseline | Primary DV: first step where disjoint val ≤ 3.3821 |
| Re-benchmark stock baseline on your GPU; report relative % | F1 (historical) and upcoming F1_seal on A100-40 |
| Training clock only — timer pauses during validation | Use harness train_time_ms, not wall calendar time that includes val |
| One GPU | Modal A100-40GB only; never bare A100 (80GB trap) |
| Fairness: if you tune X for your run, baseline gets X too | Same compile flag, same precision, same pin on denominator and challenger |
| Official software pin (repo era): PyTorch 2.8.0-dev cu128; compile optional if broken | We diverged on Modal CLI (see below) — so we re-seal, we do not hide it |
| Negative results welcome; no copying Modded-NanoGPT primary speedups as the “idea” | Archive miss configs; kill gates pre-registered |
B. What we add for forensic accuracy (beyond the README)
CLI images used unpinned pip install torch. F2b / Wave-4 / indeea resolved 2.13.0+cu130; notebook F2/W3 were 2.8.0+cu129; historical F1 torch was never logged. Code now pins TORCH_PIN = "torch==2.13.0". Smoke writes /vol/env_fingerprint.txt (profile, torch, CUDA, GPU name, pin).
torch.compile first-iteration overhead and early step instability are included in \(T\) (train_time to first cross). We never start the official clock after a secret warmup that baseline did not get. Steady-state step_avg is reported separately with \(N_{warm}=200\).
\(T\) = train_time_ms at first disjoint val ≤ 3.3821.
\(N^*\) = that step index (sample efficiency).
step_avg = mean over \([200, N^*]\).
Speedup = \((T_{F1\_seal} - T_M) / T_{F1\_seal}\). Never claim from sliding/EMA alone.
Measured stack drift ~6% step-time (802 vs 753 ms) is the empirical floor until sealed CV is known. Single-seed \|X\| < ~6% → do not publish as win. 6–10%: sealed + 2 seeds preferred. >10% or ≥200 steps earlier: 1 seed OK for narrative + fingerprint. Old “<3%” rule was too tight (would call infra a win).
C. Torch audit (why indeea shows 2.13)
| Run | Path | torch recorded | Forensic note |
|---|---|---|---|
| F1 baseline | notebook | not in config | Invalid speed denominator for CLI 2.13 until F1_seal |
| F2 / Wave-3 | notebook | 2.8.0+cu129 | Within-family comparisons only |
| F2b / Wave-4 | CLI modal_nocap | 2.13.0+cu130 | Same unpinned resolve day/family |
| indeea smoke | CLI same app | 2.13.0+cu130 | Not a cheat kit — same stack as F2b/W4; still needs F1_seal for % vs F1 |
| Going forward | all CLI profiles | torch==2.13.0 pin | Any pin change = new seal required |
D. Claim classes (what is allowed to publish)
| Claim | Needs sealed env? | Status after audit |
|---|---|---|
| Val ≤ 3.3821 hit/miss (quality) | No (log fingerprint anyway) | F2 miss · F2b hit SOUND |
| Wall-clock % vs historical F1 | Yes | F2b −2.9% RETRACTED (speed) — below ~6% infra floor |
| Within-app A/B (Wave-4 wd120 vs wd720) | Same pin family | SOUND within CLI 2.13 |
| Wave-3 parallel ~1.6% | Same notebook 2.8 family | WITHIN-FAMILY ONLY |
| indeea proxy ranking | Same pin on indeea | ORDINAL ONLY — not a board % |
| Published official speedup | F1_seal + claim both on rainbowpuffpuff | NOT YET |
E. Seal procedure (mandatory before any speed claim)
- Code pin: TORCH_PIN = "torch==2.13.0" in modal_nocap.py + indeea_proxy.py (done; log full 2.13.0+cu130 string).
- Smoke on clock account rainbowpuffpuff → fingerprint torch + A100-40.
- F1_seal on rainbowpuffpuff: full 4768 stock F1 — not run until go-ahead (~$8–10).
- Research proxies on indeea = ordinal ranking only.
- Claim final also on rainbowpuffpuff vs that F1_seal — no published cross-account cardinal %.
- If claim is 6–10%, prefer 2nd seed; if <~6% single-seed, do not publish as win.
F. Meta AI forensic review (2026-07-11)
External skeptical methods review (Meta AI browser skill). Thread e5e853d3-e8e8-4afa-91f3-e8f525078f98. We adopt the fixes; we do not adopt factual errors.
| Standard | Verdict |
|---|---|
| Competition-legal plan | APPROVE — F1_seal, relative %, train_time, one GPU |
| Forensic / publish-grade after fixes | APPROVE-WITH-FIXES |
| Meta raw tone | “REJECT as written” on pure forensic grounds — accepted as pressure; path is fixes, not stop |
• Published % only on rainbowpuffpuff (F1_seal + claim)
• Noise floor ~6% single-seed (not 3%)
• Proxies = ordinal only; winner’s-curse awareness
• F2b −2.9% retracted for speed (quality hit stands)
• Same-team disclosure; publish failed proxies
• Muon needs attribution if shipped (Modded-NanoGPT adjacency)
• First-passage \(T\) is high-variance; single seed = case study
• “torch 2.13 does not exist” — false; we run 2.13.0+cu130 on Modal
• “~$2 / full run, 20 runs on $60” — false; F1 ≈ 3.8 h × ~$2.10 ≈ $8–10
• Mandatory 3× F1_seal before claim — budget-killing; one seal + optional method seed is plan
• Dual accounts are still good ops; they are not a free statistical lunch
No silent torch upgrades between denominator and challenger.
No published cross-account cardinal % — claim final on clock; indeea is ordinal only.
No single-seed win claim if \|X\| < ~6% until sealed CV says otherwise.
No “indeea won because 2.13” narrative — same CLI stack as F2b/W4.
No claiming from sliding/EMA; official metric is disjoint val only.
Include compile warmup in \(T\); report steady step_avg separately.
Writeup: one team, two workspaces; archive negatives; attribute Muon if used.
Source of truth: private docs/ops/FAIR_CLOCK.md + smoke env_fingerprint.txt.
What the agent is (and is not)
We call it an “agent” because it is a goal-directed, multi-phase system with memory, tools, and a control loop — not because it chatters. Humans set kill gates and recipes; Modal executes the DAG; the volume is durable memory; the kit is the tool surface.
log.csv (train/val/sliding/EMA), config.json, triage_stats.json, billing probes, volume listings. Every keep/kill is trajectory-based, not a single snapshot.
Data shards, features, scores, triage bins, priors, run dirs, LAUNCH markers, runs_logs.zip. Survives container death. Concurrent-writer safe (v2).
Train, score, triage, prior-build, summarize — each a Modal function with hard timeouts, scaledown windows, and idempotent skip rules.
Remote driver: phase DAG, spawn/fan-out, join. modal run --detach decouples local client; panic stop is one CLI command.
Playbook ideas behind flags (defaults = stock). Negative results are first-class outputs. Time-scale hparams re-derived at proxy length.
No tool-calling LLM in the loop. Humans write gates and interpret curves. The “agent” is the experiment runtime + method, not a conversational planner.
Layered architecture
GitHub layout — rainbowpuffpuff/nocap
Private working repo until email submission (synced 2026-07-11, commit family 17b9e95+). Mirrors brain (docs + literature), body (kit), hands (notebooks + indeea scripts), and memory (results logs — no multi‑GB weights in git).
Not in git (local / Modal only): multi‑GB *.pt, FineWeb bins, score npz, Modal tokens. Restore via volume_backup / CLI get-put.
Companion machine workspace: software_i_built/modal/ — full archives, Cloudflare deploy source (cf-pages-nocap-agent/), live pulls (pull_wave4_*, nocap_archive/).
Kit modules — tool surface
Design contract of the trainer: with all new flags at defaults, semantics match the stock baseline (model, loss, schedule, timing protocol). Every experiment is opt-in.
| Module | Role | Key I/O |
|---|---|---|
| train_gpt2_t4.py | Instrumented GPT-2 trainer (d12 / d2 smoke). RoPE, RMSNorm, optional SwiGLU, Canon, parallel block, MTP, loss shaping, prior, EMA, z-loss, sliding val. | bins → runs/<name>/ |
| get_data.py | Fetch FineWeb pre-tokenized chunks from HF into volume/local. | → data/*.bin |
| triage_features.py | Stage-A doc features (len, unique, top, rep8, digit, punct). | → features/*.npz |
| score_docs.py | Stage-B: per-doc mean CE under a trained ckpt; per-shard skip = resumable. | ckpt + bins → scores/*.npz |
| triage_select.py | Junk rules + optional score band + 10% random rescue + optional curriculum order. | → triage_*/fineweb_triage_*.bin |
| build_unigram_prior.py | Corpus unigram log-prior for fixed logit bias. | → unigram_prior.pt / prior_full.pt |
| eval_sliding.py | Boundary-aware sliding-window val (report alongside official disjoint). | ckpt → Δ vs disjoint |
| summarize_runs.py | Table over all run dirs; report zips logs without final.pt. | → console + zip |
| modal_nocap.py | Full experiment DAG as Modal App; phases idempotent; detach-safe orchestrator. | volume as bus |
Trainer flag groups (opt-in ideas)
--logit_prior · --ema_beta · --ema_start_frac · --warmdown_iters · --input_bin (triage_*) · --val_stride · --precision bf16 · --compile 1
--mtp_* · --loss_shape / --ls_* · --mlp swiglu · --canon · --parallel_block · --zloss
Data pipeline architecture
FineWeb10B pre-tokenized GPT-2 BPE: up to 50×100M tokens. Competition budget cap 5B; baseline consumes 2.5B. Our F2 path uses a curated subset scored and banded from 28 chunks.
| Artifact (scale verify, 2026-07-09) | Value |
|---|---|
| docs_total / docs_kept | 4,045,481 → 2,927,881 |
| tokens_total / tokens_kept | 2.796B → 1.922B (kept_frac ≈ 0.687) |
| Score band (mean-loss) | [4.1887, 5.1831] = pct 10–80 of 4,045,472 docs |
| scores/*.npz | 28 shards (expect 28) |
| triage_full bins | 20 shards · ≈3.58 GiB |
| prior_full.pt | ~198 KiB · unigram CE 7.6690 nats |
| Dominant rule_hit | score_band 1,213,639 · unique 33,465 · len 9,532 |
Scoring is offline like the baseline’s own pre-tokenization cost narrative: GPU prep is honest and disclosed in IDEA.md. Training then sees a higher-signal stream (~1.22 epochs of 1.92B curated tokens cycled toward ~2.5B consumed). Proxy isolation: w2_databand alone was −0.051 nats @400 vs raw stream.
Modal orchestrator — control loop
App name: nocap-wave2. Image: debian slim + torch==2.13.0 (pinned) + numpy/huggingface_hub/tiktoken. Volume: nocap version=2. GPU string: A100-40GB (never bare A100). Fingerprint on every smoke.
Phase catalog
Invocation shape
# local entrypoint only spawns the remote orchestrator — safe to close the terminal modal run --detach kit/modal_nocap.py --phase all modal run --detach kit/modal_nocap.py --phase scale --chunks 28 --band 0.10,0.80 modal run --detach kit/modal_nocap.py --phase f2 # panic / observe modal app stop nocap-wave2 modal app list modal volume get nocap runs_logs.zip ./runs_logs_modal.zip
Confirmed recipe brain
Every component proxy-validated at matched steps / hardware. Stack additivity ~70–100% at proxy scale.
| Lever | Mechanism | Proxy effect | Status |
|---|---|---|---|
| Perplexity-band curation | Keep docs in mean-loss pct 10–80 + junk rules + 10% anti-drift rescue | −0.051 @400 | SHIP |
| Unigram logit prior | Fixed corpus log-bias on logits; start ~7.7 not 10.8 | −0.047 @400 | SHIP |
| Shortened warmdown | Early LR decay wasteful while still underfit | −0.013 @800 | SHIP |
| Sliding-window eval | Boundary-aware val; reported with official disjoint | Δ 0.019@800 → 0.060 full (F1) | SHIP (dual-report) |
| EMA weights | β scaled to horizon; dual-eval at end | lag artifact at proxy | MAINTAIN free |
| Stack (data+prior+WD) | Composition for F2 | −0.056 raw / −0.074 w/ eval @800 | F2 RECIPE |
Killed with evidence (submission material)
+11% step time and worse loss @800. Consistent with small-model NLL cautions. Configs + curves archived.
~0 cost, never ahead (+0.007 @800). Val includes every token — down-weighting is a mid-run tax.
Big early lead evaporates by step 800; ends ≈ baseline (4.78).
−0.078 nats (largest training-side effect) but +30% step time naive → net loss. #1 FUTURE lead if fused ≤2% tax.
~1.6% faster, loss-neutral-or-better (−0.005). Out of F2 for composition risk only.
Baseline sits at exactly 20 tok/param (Chinchilla isoline). Resizing unstable sign, ≤2%.
Method — how the agent decides
- Pre-registered kill gates before each arm (RUNBOOK + playbook).
- Matched steps on identical GPU type when step times equal; matched wall-clock when they don’t.
- Paired within-run measurements where possible (immune to run noise).
- Re-derive every time-scale hyperparameter at proxy length (EMA-β lesson: 0.999 full-run ≠ 0.99 proxy).
- Gates on trajectories, not snapshots (sliding Δ grows through training).
- Negatives as first-class outputs — README invites them; we archive configs + CSVs.
- Effects < 0.01 CE need two seeds before keep/kill.
- Pin environment — torch==2.13.0 seal; historical drift 802 vs 753 ms/step (~6%). See fair-clock protocol.
- Compile + GPU warm on clock — \(T\) includes compile first-iter; no secret warmup; steady step_avg after \(N_{warm}=200\).
- Fingerprint every run — torch, CUDA, GPU name, pin, compile, seed, batch×accum×seq → env_fingerprint.txt.
- Noise floor ~6% — measured stack drift; single-seed \|X\| < ~6% is not a published win (Meta review 2026-07-11).
- Proxies ordinal only — 800-step ranks methods; published % = full-horizon \(T\) on clock account.
- Claim on clock — F1_seal + final both on rainbowpuffpuff; no cross-account cardinal %.
model d12 · batch 8 · grad_accum 16 · seq 1024 → 131,072 tok/step (¼ of full 524,288) · LR 4.5e-4 · warmup 128 · warmdown 160 (or 40 with EMA) · val every 100 · bf16 · compile 1 · zloss 1e-4 (F1 drops zloss).
Full finals: 4768 steps · batch 16 · accum 32 · LR 0.0018 · warmup 256 · warmdown 1024 stock / 256 stack · val every 128.
Cost safety, idempotency, serverless semantics
Serverless: no long-lived GPU. timeouts 1h (train) / 8h (finals). scaledown_window=10s. Worst hung arm ~$2.10; absolute multi-hang cap ~$15.
Safe re-run of any phase. score_docs skips per-shard .npz. prep skips prior/features/triage if present.
LAUNCH + log.csv freshness <15 min → fail loud instead of racing twin trainers (learned after a silent double score ~$2 tax).
Volume v1 vs v2: scripts request version=2; creating with default v1 breaks concurrent writers — recreate with --version=2 (or omit version pin when profiles already differ).
Cloudflare 403 on kit download: default Python-urllib UA blocked inside Modal; send curl UA from ensure_kit().
Image build free, cold start billed: smoke in-container 14s ≈ $0.008 GPU; actual bill ~$0.009 with pull.
Preemption + duplicate guard (Wave-4, 2026-07-10): Modal preempted w4_wd120 ~step 1589; restart saw fresh log.csv (<15 min) and raised RuntimeError: another instance appears live — orchestrator died, w4_wd720 never started. Guard is correct for true doubles; needs a preemption-aware path (resume / ignore LAUNCH after worker preemption).
Cross-account data: volumes never share. F2b on acct3 required CLI volume get/put of triage_full/ + prior_full.pt (+ scores/features insurance) from acct1 — pure transfer, no GPU.
Live state — 2026-07-11
| Milestone | Result | Status |
|---|---|---|
| Wave-1 (Colab T4) | fp16 parity, prior, sliding Δ growth, warmdown waste found | DONE |
| Smoke (Modal) | A100 lifecycle on profiles incl. indeea · ~$0.01 each; rainbowpuffpuff profile ready (not smoked yet) | DONE / READY |
| Wave-2 (Modal) | m0 4.7788 · stack 4.7232 · databand −0.051 · MTP/shape kill | DONE |
| Wave-3 (Modal, acct 3) | Canon −0.078 but +30% time · parallel banked · SwiGLU kill | DONE |
| Scale (acct 1) | 28 scores · prior_full · triage_full 1.92B tokens · band [4.19, 5.18] | DONE |
| F1 (acct 2) | Crosses 3.3821 @ step 4736 = 229.1 min · final 3.3795 · step_avg 2903 ms | DONE |
| F2 record (acct 1) | Final 3.4050 (never crossed 3.3821) · 235.0 min · step_avg 2957 ms · WD 256 too short | MISS |
| F2b recovery (acct 3) | Stack + WD 1024 · final 3.377309 · first cross @4736 = 235.6 min · step0 val 7.88 · ~3.95 h | HIT TARGET |
| Official speedup (retracted) | (13746s − 14139s) / 13746s = −2.9% · same step · torch confound + \|X\| < ~6% infra floor — retracted as speed evidence; quality hit still stands. Meta review 2026-07-11. | RETRACTED (SPEED) |
| Meta forensic review | Dual-account idea OK; cross-account cardinal % weak; noise floor → ~6%; proxies ordinal; APPROVE-WITH-FIXES · see §01b·F | ADOPTED FIXES |
| Wave-4 diagnostic (acct 2) | COMPLETE · app stopped · w4_wd120 final val 3.982 · w4_wd720 final val 3.993 · both ~34 min · step_avg ~845–850 ms · pull_wave4_20260711/ | DONE |
| rainbowpuffpuff (clock) | New workspace · role = F1_seal + published claim only · same A100-40 + torch==2.13.0 · volume not seeded yet | SETUP |
| indeea / indeeaindi (R&D) | control val@800 4.731 · parallel_on 4.585 (better CE + slightly faster steps) · next: Muon proxy | PARALLEL DONE |
| rainbowpuffpuff F1_seal | 1st attempt preempted ~step 840 (evidence archived) · resilience (ckpt/resume/vol commit) · relaunch RUNNING ap-bWq1ZYnXFOfFe98SbgxhRs | RUNNING |
| Literature program | 72 objects (36 core + 36 tangential) · experiment cards · gap analysis G1–G15 | ARCHIVED |
Recipe: triage_full + prior_full + EMA β=0.999 + warmdown 1024. Hit val target (3.377 ≤ 3.3821). Wall −2.9% vs historical F1 is retracted as speed evidence (stack confound + below ~6% infra noise floor). Quality hit stands. Next speed number only vs F1_seal on rainbowpuffpuff.
wd120 final 3.982 beats wd720 final 3.993 by ~0.01 CE. Curves nearly identical early; wd720 was briefly better at steps 1800–2100 (−0.04 at 2100), then short anneal wins the endpoint. Effect size is small — anneal length is not a free +5% lever at this proxy. Does not overturn F2 (WD256 @ full 4768 miss): full-horizon short anneal still failed; proxy 2400 ≠ finals.
App ap-LuwPnb6EpePg92rZ6zEMk4 stopped. Logs: pull_wave4_20260711/.
1. Stay on A100-40 through end of campaign (no GPU-class switch; VRAM underuse accepted).
2. Win plan approved — executing. Full roadmap: §13b.
3. Phase A (rpp): F1_seal relaunched after preemption (resilient ckpt). Phase B: control 4.731 · parallel_on 4.585 DONE; next Muon.
4. No official speedup claim until \(T_{cross} < T_{F1\_seal}\) on rpp with |X| ≥ ~6% (or 2 seeds).
Literature & hypotheses (why these races)
Closed-loop agent + 72 literature objects in academic_lit/ (core BIB.md + tangential BIB_TANGENTIAL.md). Time-to-target decomposes as:
T_cross ≈ N_steps(data, opt, schedule, arch) × t_step(arch, systems) + T_fixed(compile, val)
What the 2026-focused literature implies
| Cluster | Key papers | Hypothesis we test |
|---|---|---|
| O · Optimizers | Muon scalable (2502.16982) · Newton–Muon (2604.01472) −6% steps / −4% wall on GPT-2 speedrun class · SOAP (2409.11321) | B-Muon: matrix-aware updates cut N_steps enough that warmup/GPU noise don’t matter — largest untried lever for us |
| S · Schedules | Anytime pretraining (2602.03702) · ScheduleFree+ (2605.19095) · EMA weights (2411.18704) · Wave-4: wd120 3.982 vs wd720 3.993 @2400 | B-Anytime: F2 short-WD miss = full-horizon issue; at proxy 2400, short vs long anneal Δ≈0.01 — prefer mild WD + averaging, not extreme short-only bets |
| D · Data | Prior-filter (2509.18577) · ADAPT online reweight (2605.05227) · proxy reliability (2512.24503) · FineWeb-Edu | B-PriorFilter / B-Mix / B-Curriculum: static PPL band hit quality not steps; try prior-only filter, mixes, order |
| A · Arch / aux | TOP (2508.19228) · Future summaries (2510.14751) · Parallel tracks (2602.07306) | MTP killed; parallel banked (~1.6%); aux2.0 only with strict step-tax gate |
| T · Tangential gaps | µP transfer · packing · mixing laws · FP8 · critical batch · softcap | G1–G15 fills: free t_step / better proxies / 5B budget use |
Keep: band curation, unigram prior, dual val, parallel, kill-gate discipline.
Kill: MTP, loss shaping, SwiGLU-at-finals, naive Canon, short-WD-only final, EMA-as-early-cross claim.
Park: fused Canon until step tax ≤3%.
Ranked experiments (indeea) — literature-backed
| # | Bet | Why now | Script | Est. $ |
|---|---|---|---|---|
| 0 | Val data seed | Need fineweb_val for official metric | 01_seed_val_data.sh | ~0 |
| 1 | AdamW control @800 curated | Denominator for all proxies on torch 2.13 | 10_proxy_control.sh | ~1–1.5 |
| 2 | Parallel on/off | W3 bank; free t_step (A4 lit) | 15_proxy_parallel.sh | ~2–3 |
| 3 | Schedule WD sweep | Anytime lit + Wave-4 anneal data | 12_proxy_schedule.sh | ~2–4 |
| 4 | Muon (after kit port) | Newton-Muon on GPT-2 class | 11_proxy_muon.sh | ~1–2 |
| 5 | Prior-filter / curriculum / mix | D3 / D7 / mixing laws | 13, 14 + cards | ~2–5 |
Full cards: academic_lit/CARDS/ · synthesis: academic_lit/SYNTHESIS.md · related work draft: RELATED_WORK.md
Dual accounts — clock vs research
Two active ~$30 Modal workspaces split by forensic role, not by GPU class. Both stay on A100-40GB + pin torch==2.13.0 until campaign end (VRAM ~10/40 GiB underused; we accept that for a stable clock).
Workspace rainbowpuffpuff. Clean denominator and official finals only. No proxy clutter on this volume.
Run here: volume create → smoke + fingerprint → raw FineWeb + val → F1_seal (stock 4768) → claim final on this same account when indeea nominates → 2nd seed if claim is 6–10%; never publish single-seed <~6% as win.
Do not run here: Muon/schedule A/B spam, half-finished proxies, Wave-4-style anneal sweeps.
Budget sketch (~$30): F1_seal ~$9–10 · claim final ~$9–10 · buffer/seed ~$10.
Isolated from Wave-4 / F2b. Curated seed already on volume. All risky method search lands here.
Run here: val seed → AdamW control @800 → parallel on/off → mild WD only → Muon when kit ready → at most one data-order bet. Rank vs control, not vs historical F1.
Do not run here: published % vs F1; second full F1_seal (that’s rainbowpuffpuff’s job).
Budget sketch (~$30): proxies ~$12–18 · reserve ~$10–15 for a compose final or hand recipe to rainbowpuffpuff.
What runs where (checklist)
| Work | Account | Est. $ | Status |
|---|---|---|---|
| Re-smoke + env_fingerprint.txt (pin 2.13) | rainbowpuffpuff | ~0.01 | DONE |
| Raw FineWeb + val for stock F1 | rainbowpuffpuff | ~0–1 | DONE |
| F1_seal stock 4768 (sole speed denominator) | rainbowpuffpuff | ~8–10 | RUNNING |
| Hold for full claim final | rainbowpuffpuff | ~9–10 | RESERVED |
| Val shard + readiness | indeea | ~0 | DONE |
| Proxy control AdamW @800 curated · ix_control | indeea | ~0.4 | DONE · val 4.731 |
| Parallel on · ix_parallel_on | indeea | ~0.4 | DONE · val 4.585 vs control 4.731 · step_avg ~866 ms |
| Mild schedule (wd160/wd400 — not extreme short) | indeea | ~2–3 | OPTIONAL · low EV post-W4 |
| Muon proxy | indeea | ~1–2 | NEXT |
| Full compose / claim @4768 | rainbowpuffpuff | ~8–10 | ONLY IF PROXY WINS |
Retired / archive accounts (do not burn for new science)
| Profile | Past role | Now |
|---|---|---|
| acalincarol | Scale · F2 record | Archive / data source |
| stufflaters | F1 historical · Wave-4 | Done — no more anneal GPU |
| deeferentleeg | Wave-3 · F2b | Archive (quality hit, provisional wall) |
rainbowpuffpuff — clock commands (when approved)
# profile is local name = workspace rainbowpuffpuff MODAL_PROFILE=rainbowpuffpuff modal volume create nocap --version=2 MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase smoke # confirm /vol/env_fingerprint.txt → torch 2.13.0 + A100-40 MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase data --chunks 28 MODAL_PROFILE=rainbowpuffpuff modal run --detach modal_nocap.py --phase f1 # → runs/f1_baseline (or f1_seal name) = sole T denominator
indeea — research commands (when approved)
| Item | Status |
|---|---|
| Volume nocap v2 | CREATED |
| Curated seed (triage_full 20 · scores 28 · features 28 · prior) | SEEDED |
| Smoke A100-SXM4-40GB · torch 2.13.0+cu130 | OK |
| Readiness f2b_ok | TRUE (raw train optional) |
| Scripts indeea_scripts/*.sh + indeea_proxy.py | DRY-RUN READY |
# always dry-run first; --execute only when approved cd software_i_built/modal/indeea_scripts ./00_check.sh ./01_seed_val_data.sh --execute ./10_proxy_control.sh --execute ./15_proxy_parallel.sh --execute # mild schedule only — not extreme short-WD finals ARM=wd160 ./12_proxy_schedule.sh --execute MODAL_PROFILE=indeea modal run indeea_proxy.py --arm control --iters 800 --input curated
Published speedup: both \(T_{F1\_seal}\) and \(T_M\) on rainbowpuffpuff only (same pin, A100-40, compile warm in \(T\)). No cross-account cardinal %.
indeea ranks methods vs its own proxy control — ordinal only; a “win” is a nomination, never a board %.
Same team: writeup discloses two workspaces, one methodology (not two independent competitors).
Noise: single-seed \|X\| < ~6% is not a publishable win until sealed CV known.
indeea torch 2.13 = same CLI stack as F2b — not a cheat kit. Protocol: §01b · docs/ops/FAIR_CLOCK.md.
Legacy anchor #indeea redirects conceptually to §13 Dual accounts.
Win plan — best Modal path (approved · executing)
Goal: disjoint val ≤ 3.3821 with \(T < T_{F1\_seal}\) by a margin that survives ~6% noise, A100-40 + torch==2.13, claim on rainbowpuffpuff, story that is not “paste Modded-NanoGPT.” F2b proved quality stack alone does not win the clock — we need \(N^*\) and/or \(t_{step}\) levers under fair-clock rules.
Phase A (clock) on rpp: 1st F1_seal preempted ~step 840 (evidence in pull_…/rpp_f1_baseline_partial_preempt/) · resilience patch (ckpt every 256 + resume + mid-run vol commit) · F1 relaunch RUNNING ap-bWq1ZYnXFOfFe98SbgxhRs.
Phase B (indeea): control 4.731 · parallel_on 4.585 (Δ −0.15 CE, slightly faster steps) · bank parallel. Next: Muon proxy vs control; optional mild WD / prior-filter.
Private doc: docs/ops/WIN_PLAN.md · fair clock: §01b.
Tier 0 — clock (mandatory)
| # | Work | Account | Est. $ | Status |
|---|---|---|---|---|
| 0a | Volume + smoke + fingerprint | rainbowpuffpuff | ~0.01 | DONE |
| 0b | Raw FineWeb + val (data --chunks 28) | rainbowpuffpuff | ~0–1 | DONE |
| 0c | F1_seal stock 4768 (relaunch after preempt) | rainbowpuffpuff | ~8–10 | RUNNING ap-bWq1… |
| B0 | indeea control @800 | indeea | ~0.4 | DONE · 4.731 |
| B1 | indeea parallel_on @800 | indeea | ~0.4 | DONE · 4.585 |
Tier 1 — highest EV for a real win (implement then run)
| Rank | Bet | Code? | Why it can win | Where |
|---|---|---|---|---|
| 1 | Muon / Newton-Muon + AdamW embed/head | Kit port required | Only big untried \(N^*\) lever; GPT-2-class lit | indeea proxy → rpp claim |
| 2 | Compose: F2b stack + parallel (+ Muon if #1) | Parallel flags exist | Free \(t_{step}\) + known quality hit | rpp claim |
| 3 | Horizon-free / anytime + registered averaging | Schedule code | Explains F2/F2b; earlier first-cross | indeea → rpp |
| 4 | Prior-filter or raw/curated mix | Light CPU + proxy | Original data story; may cut \(N^*\) | indeea → rpp |
| 5 | Prior anneal (strong early → 0) | Small flag | Uses existing prior asset | indeea → rpp |
Tier 2 — ordinal screens (indeea)
| # | Action | Account | Est. $ | Status |
|---|---|---|---|---|
| 2a | AdamW control @800 · ix_control | indeea | ~0.4 | DONE val 4.731 · step_avg ~878 ms |
| 2b | Parallel on @800 · ix_parallel_on | indeea | ~0.4 | DONE val 4.585 · step_avg ~866 ms · bank |
| 2c | Mild WD only (wd160 vs wd400) | indeea | ~2–3 | OPTIONAL · low EV |
| 2d | Muon proxy vs control | indeea | ~1–2 | NEXT |
| 2e | Prior-filter / curriculum / compose | indeea | ~2–5 | AFTER MUON |
ix_control val@800 4.731 (step0 7.884 prior) · step_avg ~878 ms · ranking denominator.
ix_parallel_on val@800 4.585 (−0.146 vs control) · step_avg ~866 ms · wall ~693 s. Parallel is a clear keep for compose/claim.
Next: Muon @800 vs control (kit ready). Full claim waits on F1_seal on rpp.
Kill schedule/data fast if Δ < 0.01 CE; put remaining $ into Muon + one full claim. Proxies never publish board %.
Phased execution
# PHASE A — CLOCK (rpp) ~$10 MODAL_PROFILE=rainbowpuffpuff modal volume create nocap --version=2 MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase smoke MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase data --chunks 28 MODAL_PROFILE=rainbowpuffpuff modal run --detach modal_nocap.py --phase f1 # PHASE B — SCREEN (indeea) while coding Muon ~$8–12 cd indeea_scripts ./00_check.sh ./01_seed_val_data.sh --execute ./10_proxy_control.sh --execute ./15_proxy_parallel.sh --execute # + implement Muon in train_gpt2_t4.py # PHASE C — MUON DECISION (indeea) ~$3–5 after kit ./11_proxy_muon.sh --execute # unblocked only after port # PHASE D — CLAIM (rpp) ~$9–18 # if Muon promotes: Muon + parallel + stack # else: parallel + stack + best data/schedule nominee # if |X| in 6–10%: 2nd seed; if <~6%: do not publish as win
Skip (do not burn credits)
- Extreme short-WD finals · re-score whole FineWeb for another static PPL band
- Historical F1 as denominator · cross-account published %
- Single-seed <~6% “win” · torch/GPU shopping · blind Modded-NanoGPT dump
- 3× F1_seal “for science” without a claim plan
Seal clock → implement Muon → proxy on indeea → if promote, full claim = Muon + parallel + proven stack on rpp vs F1_seal. Fallback if Muon fails: anytime + prior-filter/mix + parallel compose.
Future architecture (post-finals leads)
After F2b (−2.9% wall) and Wave-4 anneal data: prioritize optimizer geometry and horizon-free schedules (2026 lit), then free step wins (parallel), then data mixes — not another short-WD final.
| # | Lead | Why it matters |
|---|---|---|
| 1 | Muon / Newton-Muon | Largest untried N_steps lever; GPT-2-class evidence in 2026 papers. |
| 2 | Anytime / ScheduleFree + averaging | Fixes horizon mismatch that killed F2 WD256. |
| 3 | Parallel attn+FFN (+ fuse) | Already ~1.6% faster @proxy; free basis points. |
| 4 | Fused Canon | −0.078 nats if step tax ≤3%; else park. |
| 5 | Prior-filter / mix / curriculum | Quality stack didn’t cut steps; try cheaper filters + order. |
| 6 | Packing / batch / FP8 | Tangential t_step systems (G3/G5/G6). |
How to run the agent
# once per workspace pip install modal modal setup modal volume create nocap --version=2 # cheap proof (~$0.01) modal run kit/modal_nocap.py --phase smoke # proxy DAG (~$4–6, 60–80 min wall) modal run --detach kit/modal_nocap.py --phase all # full curated dataset for F2 modal run --detach kit/modal_nocap.py --phase scale --chunks 28 # finals (per-account profiles — volumes do NOT share) MODAL_PROFILE=stufflaters modal run --detach kit/modal_nocap.py --phase f1 MODAL_PROFILE=acalincarol modal run --detach kit/modal_nocap.py --phase f2 MODAL_PROFILE=deeferentleeg modal run --detach kit/modal_nocap.py --phase f2b MODAL_PROFILE=stufflaters modal run --detach kit/modal_nocap.py --phase wave4 MODAL_PROFILE=indeea modal run kit/modal_nocap.py --phase smoke # indeea research proxies (dry-run unless --execute) cd indeea_scripts && ./10_proxy_control.sh # add --execute when approved # readiness / transfer (CPU only — no GPU) MODAL_PROFILE=<name> modal run kit/modal_nocap.py --phase readiness MODAL_PROFILE=acalincarol modal volume get nocap triage_full/ ./triage_full/ MODAL_PROFILE=indeea modal volume put nocap ./triage_full triage_full -f # fetch evidence modal volume get nocap runs_logs.zip . python kit/summarize_runs.py runs
01–04 campaign docs → 05 this map → academic_lit/ · indeea_scripts/ · volume_backup/MANIFEST.md · LESSONS.md
modal_nocap.py · academic_lit (72 papers) · indeea_scripts · volume_backup · pull_20260711 · nocap_archive
One-line summary
A serverless FineWeb experiment agent with a historical F1 clock (3.82 h, invalid for speed), F2 miss, F2b quality hit (3.377) with −2.9% wall retracted as speed evidence (infra floor ~6%), Wave-4 done, torch 2.13.0 on A100-40, dual accounts (rainbowpuffpuff clock + claim; indeea ordinal proxies), Meta forensic review → APPROVE-WITH-FIXES, and 72 papers aiming at Muon/anytime/parallel under sealed fair-clock conditions.