# NoCap campaign — comprehensive report

**Evidence state:** 2026-07-16 (seals) · volume inventory 2026-07-17  
**Purpose:** submission packaging — methods, timeline, failures, seals, components  
**Status:** multi-seed under-baseline vs sealed F1 — **not** an official NoCap win. Matched cand-vs-stock ΔT% = N/A. Public % vs F1_seal only.

**Sources:** `NUMBERS_CANON.md`, `nocap-repo/results/claim_stack_v1/MULTI_SEED_CLAIM.md` (+ seed summaries / `PREREG.md`), `FAILURE_LEDGER.md`, `CAMPAIGN_TIMELINE.md`, `CAMPAIGN_RUNS_OVERVIEW.md`, `VOLUME_INVENTORY.local.md`, `CHALLENGE_TERMS.md`, `FAIR_CLOCK.md`, `RECIPE_CARD.md`.

---

## Verdict

On three independent seeds (1337, 2029, 4242), the claim-stack candidate (Muon B32 + timing-cheap split-QKV) first-crosses `val_disjoint ≤ 3.3821` about **8% faster** than the sealed stock AdamW F1 baseline (**3.866 h**; mean ΔT% ≈ **+8.3%**), with terminal CE ≈ 3.37. Where measured, the joint-QKV stock control misses the quality bar.

Local seals are campaign evidence. They are **not** organizer acceptance.

---

## Challenge terms

Official source: [BottleCapAI/NoCap-Test](https://github.com/BottleCapAI/NoCap-Test) (`master` README).

| Official rule | Summary |
|---|---|
| Optional goal | val loss **≤ 3.3821** on FineWeb **faster** than stock baseline |
| Non-4090 | Benchmark baseline on your setup; report speedup vs **that** baseline |
| Fair comparison | Same treatment for baseline and candidate |
| Documentation | `IDEA.md` (motivation, method, results). Negative results welcome |
| Submission | `git bundle` → hey@bottlecapai.com (speedup % in subject; negative OK). Beat baseline → add `RESULTS.md` |
| Status | First competition finished Nov 2025; self-paced submissions still welcome |

They want algorithmic ideas, not LR-only tuning or copied Modded-NanoGPT tricks.

### Mapping

| Official | This campaign |
|---|---|
| Quality ≤ 3.3821 | First logged `val_disjoint ≤ 3.3821` (horizon 4768) |
| Speed vs baseline | First-passage train time \(T\) from `train_time_ms` at first-cross (compile warmup **in**) |
| Non-4090 | Denom = sealed stock AdamW **F1_seal** on pinned stack — not the 4090 leaderboard row |
| Fair comparison | One sealed denom; no matched speedup vs joint stock where stock missed quality |
| Bundle | Campaign packaging for organizers — not an official win by itself. |

---

## Fair clock

Published speed metric = **first-passage train time**:

1. Train on sealed recipe / pin.
2. Log official `val_disjoint` (claim-stack: every 128 steps).
3. First step with `val_disjoint ≤ 3.3821`.
4. Read `train_time_ms` at that row → \(T = \mathrm{train\_time\_ms} / 3.6\times10^6\).
5. Compare to public denominator:

\[
\Delta T\% = 100 \times \frac{3.866\,\mathrm{h} - T_M}{3.866\,\mathrm{h}}
\]

Positive ΔT% = fewer hours to first hit the quality bar.

### Pin (required for cardinal %)

| Knob | Sealed value |
|---|---|
| GPU | **NVIDIA A100-SXM4-40GB** exact |
| Torch | **2.13.0** (resolved **2.13.0+cu130**) |
| Quality bar | `val_disjoint ≤ 3.3821` |
| Clock field | `train_time` / `train_time_ms` |

### Not the metric

| Not the metric | Why |
|---|---|
| Terminal val alone | Quality without first-passage \(T\) is not a speed claim |
| step_avg alone | Forensic only |
| Sliding / EMA val | Not the official disjoint endpoint |
| Proxy @800 ΔT% | Ordinal only — never published cardinal % |
| Historical notebook F1 (~229.1 min) | Torch unlogged; provisional |
| Matched ΔT% vs joint-QKV stock | Stock never crossed on sealed pairs → **N/A** |

### Noise floor

Measured stack drift ~**6%** step-time is the empirical prior for single-seed noise. \|ΔT%\| < ~6% (single seed): do not publish as win. Claim-stack sealed candidates all ≥6%.

---

## Recipe (claim-stack v1)

### Hardware / software

| Knob | Value |
|---|---|
| GPU | A100-SXM4-40GB exact (80GB rejected by guard; seed 4242 relaunched) |
| Torch | 2.13.0+cu130 |
| Horizon | **4768** updates |
| Tokens / update | **524,288** (B32 × accum16 × seq 1024) |

### Shared recipe (both claim-stack arms)

| Knob | Value |
|---|---|
| Batch | microbatch **32** × accumulation **16** |
| Muon groups | lr **0.02** · momentum **0.95** · WD **0.1** |
| Adam groups (embed/head) | lr **0.0018** · WD **0.1** |
| Newton–Schulz steps | **5** |
| Warmup / warmdown | **476** / **1192** |
| z-loss | **0** |
| EMA | **off** |
| Data | raw FineWeb shards + official val |
| Prior | frozen `prior_full.pt` |
| Parallel block | on |
| single_process_fastpath | on |
| val every / save every | **128** / **256** |
| Trainer | `train_gpt2_split_qkv_cheap_v1.py` (kit_override; SHA freeze in `PREREG.md`) |

### Candidate vs stock control

| | Stock control | Candidate |
|---|---|---|
| QKV | **joint** (`muon_split_qkv=0`) | **timing-cheap split** (`muon_split_qkv=1`) |
| match_joint | 1 | **0** |
| Everything else | identical | identical |

Stock control = joint-QKV **Muon** stack, not AdamW F1.  
**F1_seal** / **F1_Claim-W2** are separate AdamW denominators for under-baseline ΔT%.

**Attribution:** Muon → Keller Jordan. Timing-cheap split-QKV → this campaign’s banked mechanism.

---

## Sealed denominators

| Label | Workspace | Recipe | First-cross \(T\) |
|---|---|---|---:|
| **F1_seal** | Clock-W | stock AdamW · raw FineWeb · zloss 0 · no prior/EMA | **3.866 h** @4736 |
| **F1_Claim-W2** | Claim-W2 | same-class AdamW re-seal | **3.858 h** @4768 (~+0.2% vs F1_seal) |

Exact F1_seal ms @ cross: **13,918,667.5** → CSV hours ≈ 3.866297; publish **3.866**.  
Public headline ΔT% uses **F1_seal**. Seed 2029 may also report vs F1_Claim-W2 (+7.5%).

Do **not** publish % vs historical notebook F1 (~229.1 min).

---

## Claim-stack seals (3/3)

| Seed | Workspace | Cross | val @ cross | \(T\) (pub) | ΔT% vs F1_seal | Terminal CE | Stock |
|---:|---|---:|---:|---:|---:|---:|---|
| **1337** | Claim-W1 | 4736 | 3.3741 | **3.553 h** | **+8.1%** | 3.3710 | final 3.3848 · **no cross** |
| **2029** | Claim-W2 | 4736 | 3.3726 | **3.567 h** | **+7.7%** | 3.3695 | final 3.3840 · **no cross** |
| **4242** | Claim-W1 | 4736 | 3.3733 | **3.511 h** | **+9.2%** | 3.3702 | **not run** |

**Aggregates:** mean cand \(T\) ≈ **3.544 h** · mean ΔT% ≈ **+8.3%**. All three candidates first-cross at step **4736**.

### Explicit matched-pair statement

| Seed | Candidate first-cross | Stock first-cross | Matched ΔT% vs stock |
|---:|---|---|---|
| 1337 | yes @4736 · 3.553 h | **NONE** | **N/A** |
| 2029 | yes @4736 · 3.567 h | **NONE** | **N/A** |
| 4242 | yes @4736 · 3.511 h | not run | **N/A** |

Published under-baseline percentages compare **candidate \(T\)** to **F1_seal**, **not** to joint stock.

### Pass checklist (claim-stack era)

1. Candidate first-crosses ≤ 3.3821 — **3/3**
2. Candidate \(T\) vs F1_seal ≥ 6% — **3/3** (+8.1%, +7.7%, +9.2%)
3. Terminal CE cand ≤ stock — where stock ran
4. Soft: stock may miss — observed on 1337 and 2029
5. Same-workspace F1 on Claim-W2 for seed-2029 ratios

### Ops notes (non-metric)

- Fire-and-forget `spawn()` after nested `.remote()` cancels killed earlier work.
- Seed 4242: first spawn rejected (A100-SXM4-**80GB**). Seal is relaunched **40GB**.

---

## Timeline (causal order)

### 1. Early waves → failed finals

Absolute quality bar: disjoint `val ≤ 3.3821` on GPT-2 ~124M / FineWeb, one A100.

| Run | Outcome |
|---|---|
| **F2** (WD 256, curated/prior/EMA) | Terminal CE ≈ **3.405**; **no** first-cross |
| **F2b** (WD 1024, same family) | Quality **HIT** (cross @4736, terminal ≈ 3.377); historical wall ≈ **−2.9%** vs notebook F1 **retracted** as speed evidence |

### 2. Fair-clock reform → sealed F1 denominator

Pin: A100-SXM4-40GB · torch 2.13.0+cu130. Metric \(T\) = train_time at first logged disjoint val ≤ 3.3821.  
**F1_seal** on Clock-W: **3.866 h**. Public speed denominator.

### 3. Failed full claims (pre–component bank)

| Run | Quality | Speed vs F1_seal |
|---|---|---|
| **claim_muon_par** (Clock-W) | Miss (final **3.386130**, never crossed) | N/A |
| **claim_muon_wsd** | Hit (~3.376 terminal) | \(T\) ≈ **3.899 h** · ~**0.8% slower** |

Board-v3 800-step screens: **no promote**.

### 4. Component bank (research workspaces)

- **B32×accum16**: ~**6%** faster core step-time vs B16 (tokens/step fixed at 524,288).
- **Timing-cheap split-QKV** (block NS5, `match_joint=0`): multi-seed MECHANISM-REPEAT @800 · mean terminal gain ≈ **+0.015 CE** · time ratios ≤ ~1.01. Persistence sealed @2400 on Research-W4 seed **5151** · **+0.0147 CE** · no late reverse.
- Through 2026-07-14 evening: under-baseline complete recipes still **0**. Bank existed; full claim did not.

### 5. Tier A hparam screens — all NO-GO

| Lever | Verdict |
|---|---|
| Muon mom 0.90 / 0.98 | NO-GO (reverse) |
| Adam WD 0.05 / 0.2 | NO-GO / flat |
| Warmdown 200 → 160 | NO-GO (shorter hurts) |

Defaults kept: mom **0.95**, Adam WD **0.1**, warmdown **200** (proxy; full claim uses wu476/wd1192).

### 6. Claim-stack seals (3/3 candidates under F1)

Recipe: Muon B32 + timing-cheap split-QKV candidate vs joint-QKV stock control. Full 4768. Exact A100-40 + torch 2.13. Result: **3/3** under F1_seal (candidate vs F1, **not** matched stock speedup).

### Causal spine (one line)

Failed finals → fair clock + F1_seal → Muon compose misses → bank systems + cheap split → Tier A keeps defaults → claim-stack **3/3** under sealed F1.

---

## Negative results (failure ledger)

Hard non-claim (claim-stack era): **no matched candidate-vs-stock first-cross ΔT%**.

| What was tested | Outcome | Why not a claim |
|---|---|---|
| **F2** short warmdown | Terminal ≈ 3.405; no first-cross | Quality miss |
| **F2b** long warmdown | Quality HIT; historical wall ≈ −2.9% | Speed framing retracted (noise / stack confound) |
| Historical notebook F1 (~229.1 min) | Absolute cross exists; torch unlogged | Not the sealed denominator |
| **claim_muon_par** | Final val 3.386130; never crossed | Quality miss → speed N/A |
| **claim_muon_wsd** | Quality HIT; \(T\) ≈ 3.899 h | ~0.8% slower than F1_seal |
| Board-v3 800-step screens | Control wins/ties | No promote; proxies ≠ full claim |
| FineWeb cleaners / curation @800 | No arm promoted | Closed levers |
| Adam LR **0.003** alone | Mid-horizon lead failed @2400 | NO-GO |
| Dual-polar split-QKV | ~+0.015 CE; time ratio ≈ 1.026 | CE edge eaten by step tax |
| muon_lr **0.025** | Reverse mid-run | NO-GO |
| Fixed interleave shard order | ~0 edge | NO-GO |
| Muon NS6 / warmdown 300 | ~0 @400 screens | NO-GO |
| Late EMA as train CE | Mid ema_val mirage | NO-GO for train metric |
| **Tier A** mom / Adam WD / WD160 | **5/5 NO-GO**; zero promote | Defaults stay |
| Joint stock seed 1337 | Terminal 3.3848; first-cross NONE | Control quality miss |
| Joint stock seed 2029 | Terminal ≈ 3.3840; first-cross NONE | Same |
| Seed 4242 stock arm | Not run | No matched pair on this seed |
| Component bank alone through 2026-07-14 | Real CE/time component evidence | **0** under-baseline full recipes until claim-stack |
| “Officially won NoCap” | Organizer acceptance not in local seals | Not an official NoCap win |

---

## What was tried (family level · all workspaces)

Scale: **13** workspaces · each has a `nocap` volume · **153** `runs/` entries (incl. partials). Codes: Clock-W, Claim-W1/W2, Research-W1…W7, Archive-W1…W3.

| Family | Workspaces | What it was | Outcome (short) |
|---|---|---|---|
| **claim_stack** | Claim-W1, Claim-W2 | Full-horizon Muon B32 + timing-cheap split vs joint stock | **3/3 cand HIT** vs F1; stock MISS on 1337/2029 |
| **f1** | Clock-W, Claim-W2, Archive-W2 | Stock AdamW denominator | Clock-W + Claim-W2 **SEALED**; Archive historical only |
| **claim_muon** | Clock-W, Research-W2 | Early compose / WSD full claims | `claim_muon_par` **MISS**; `claim_muon_wsd` complete on volume (local ΔT% seal gap) |
| **ix / px / b3** | Research-W1, Research-W2 | Proxy ordinals @800–1600 | Promoted Muon+parallel; WSD best @1600; no cardinal % |
| **systems sysv1** | Research-W6, Research-W7 | Dual-workspace step-time | B32 **~6%**; NS3 / z-loss **kill** |
| **theory tv\*** | Research-W3…W7 | Mechanism + hparam screens | Cheap-split **MECH-REPEAT** + W4 **p2400 SEAL**; Tier A **all NO-GO**; AdamLR2400 **NO-GO**; dual-polar **NO-GO** |
| **curation cv1** | Research-W7 | FineWeb cleaner variants | Closed / not claim path |
| **crossfit** | Research-W6 (+ root on W5/W7) | Anchor even/odd | Research evidence |
| **waves w2/w3/w4** | Archive-W1…W3 | Historical finals / anneal | Complete archive |
| **f2 / f2b / m\*** | Archive-W1, Archive-W3 | Finals records | Historical |

### Workspace roles (sealed narrative)

| Code | Role |
|---|---|
| **Clock-W** | Public F1_seal; also `claim_muon_par` |
| **Claim-W1** | Seeds 1337 (matched pair) + 4242 (candidate-only) |
| **Claim-W2** | Seed 2029 (matched pair) + F1_Claim-W2 |
| **Research-W1** | Proxy / ordinal (`ix_*`, `b3_*`) |
| **Research-W2** | Horizon screens (`px_*`) + `claim_muon_wsd` |
| **Research-W3** | Theory / Tier A / order / EMA / cheap-split s4 |
| **Research-W4** | Cheap-split persistence seal (seed 5151 @2400) |
| **Research-W5** | Early cheap-split + Adam LR @2400 |
| **Research-W6** | Systems-v1 + cheap-split; p2400 joint **incomplete** (not a seal) |
| **Research-W7** | Systems peer + curation |
| **Archive-W1…W3** | Waves / f2 / f2b / historical f1 — not claim-stack seals |

Volumes do not share across workspaces.

---

## Components banked into the claim recipe

| Component | Evidence | In claim-stack? |
|---|---|---|
| Keller Jordan Muon | Proxy boards; full claims; claim-stack | yes |
| B32 × accum16 systems path | Research-W6/W7 systems-v1 (~6% step-time) | yes |
| Timing-cheap split-QKV (`match_joint=0`) | MECH-REPEAT @800 multi-seed; persistence @2400 Research-W4 | yes (candidate arm) |
| Frozen unigram prior | Banked; used on claim stack | yes |
| Parallel block + single_process_fastpath | Banked | yes |
| WSD schedule wu476/wd1192 | WSD lineage; Tier A rejected WD160 | yes |
| z-loss 0 | Systems kill on z-loss 1e-4 | yes |
| NS 5 | NS6 NO-GO | yes |
| Raw FineWeb (no curation) | Curation screens closed | yes |
| EMA off | Late EMA train CE NO-GO | yes |
| Joint QKV (stock control) | Matched pairs 1337/2029 | control only |

---

## Explicit non-claims

1. **Not an official NoCap win.** Local seals ≠ organizer acceptance.
2. **No matched candidate-vs-stock first-cross speedup.** Joint stock missed quality on 1337/2029; 4242 had no stock arm.
3. **No % vs historical notebook F1.**
4. **No board % from 800-step proxies.**
5. **Candidate recipe ≠ joint stock.** Headline compares candidate to **AdamW F1_seal**.
6. Do **not** say “we officially won” or inflate means beyond sealed rounding.

---

## Compute

Three tiers — different clocks. Workspaces: Clock-W / Claim-W* / Research-W* / Archive-W*. Local artifacts only · do not deploy. No GPU launches.

| Tier | Hours | What it measures | Source |
|---|---|---|---|
| **1 · Local CSVs (pulled)** | **72.03 h** | Deduped local `train_time_ms` (90 unique runs; 101 CSVs scanned, 11 byte-identical copies dropped) | [`GPU_HOURS_LOCAL.md`](GPU_HOURS_LOCAL.md) |
| **2 · Volume `train_time`** | **110.18 h** | Volume `runs/*/log.csv` max `train_time_ms` (151 run dirs · 99.3% log coverage) | [`GPU_HOURS_VOLUMES.local.md`](GPU_HOURS_VOLUMES.local.md) |
| **3 · Modal billed A100** | **147.91 h** all apps · **128.28 h** NoCap-named apps | A100-40GB container time from billing $ ÷ list rate (13 codes · 2026-07-08→16) | [`GPU_HOURS_BILLED.local.md`](GPU_HOURS_BILLED.local.md) |

**Ratios (same campaign window).** Billed all-apps vs volume: **1.34×** (147.91 / 110.18). Billed NoCap-apps vs volume: **1.16×** (128.28 / 110.18). Volume vs local: local is a pulled subset.

**Why the tiers diverge.** Local < volume: not all volume runs were pulled. Billed > volume: Modal bills container wall clock (setup, CUDA init, failed/killed partials, notebook-kernel GPU); `train_time_ms` is trainer-internal only. Archive-W* notebooks (~19.63 h) drive much of the all-apps vs NoCap gap.

**Billed method.** `modal billing report --show-resources --json` per workspace code; keep `resource == "A100-40GB"`; `gpu_hours = cost_usd / (0.000583 $/s × 3600)`. NoCap split = app `description` contains `nocap`. Report $ is pre-credit; hours assume base-region preemptible list price.

### Storage (volume bytes)

Across 13 `nocap` volumes: **~302 GiB** under `runs/` (almost all `.pt` checkpoints; text logs ~2 MiB), **~56 GiB** sum of `data/` (**≈5.6 GiB** unique FineWeb pack, replicated on many workspaces), **~420 GiB** grand total. Source: [`VOLUME_BYTES.local.md`](VOLUME_BYTES.local.md) (local · do not deploy).

### Local family rollup (from pulled CSVs)

| Field | Value |
|---|---|
| Method | max `train_time_ms` per CSV ÷ 3.6e6; SHA-256 dedupe |
| `f1` family | 19.47 h (5 runs) |
| `claim_stack` | 14.66 h (4 runs) |
| All `theory_*` | 20.78 h (26 runs) |
| Waves (`wave2`–`wave4` + `colab_wave`) | 6.91 h (16 runs) |
| `systems_v1` | 2.95 h (20 runs) |
| `curation` | 3.68 h (6 runs) |
| Horizon / other screens | 3.58 h (13 runs) |

**Caveats.** Tiers 1–2: `train_time_ms` is wall-clock inside the training loop as logged — excludes idle GPU, container spin-up, failed spawns without CSV; partial runs at measured max only. Tier 3: not a native GPU-hour field; derived from resource cost; region/non-preempt multipliers would overstate hours; local 72.03 h is not apples-to-apples with full billed window. No GPU reservation or deploy in the production of this report.

---

## Submission packaging notes

| Artifact | Role |
|---|---|
| This report | Methods + timeline + failures + seals + components |
| `NUMBERS_CANON.md` | Public number resolution order |
| `MULTI_SEED_CLAIM.md` | 3/3 table source of truth |
| `FAILURE_LEDGER.md` | Negative results |
| `FAIR_CLOCK.md` / `CHALLENGE_TERMS.md` | Metric + official mapping |
| `RECIPE_CARD.md` | Exact recipe |
| Official ask | `IDEA.md` + optional `RESULTS.md` + git bundle to hey@bottlecapai.com |

Bundle / email: **pending**. Next packaging step is writeup + organizer submission — not more science GPUs unless an operator asks.

---

## Related local paths

| Path | Content |
|---|---|
| `nocap-repo/results/claim_stack_v1/` | MULTI_SEED_CLAIM, PREREG, seed summaries, CSVs |
| `nocap-repo/results/f1_seal/` | F1_seal SUMMARY + log |
| `VOLUME_INVENTORY.local.md` | Per-workspace run inventory (local only · do not deploy) |
| `WORKSPACE_MAP.local.md` | Code ↔ CLI profile (private · never public) |