Map of this document
Mission & environment constraints
The environment is the BottleCapAI NoCap-Test hiring challenge: train GPT-2 124M on FineWeb to the baseline’s final validation loss, faster.
Reach val loss ≤ 3.3821 on the official disjoint val set, in less training wall-clock than the stock recipe on the same GPU class. Clock pauses during validation. One GPU only.
Rules: re-benchmark baseline on your hardware. F1 on A100-40 crossed 3.3821 at step 4736 = 229.1 min (3.82 h), final 3.3795, step_avg 2903 ms. That is the speedup denominator.
| Axis | Allowed | Forbidden / trap |
|---|---|---|
| Data | Any filter/reorder of the given FineWeb tokens | New datasets outside the kit’s FineWeb stream |
| Model / loss | Architecture, loss, schedule changes (with valid prob. model at eval) | Copying Modded-NanoGPT / leaderboard primary ideas |
| Eval | Any seq length that scores the full val set; sliding reported alongside official | Changing val_tokens on real runs (it IS the metric) |
| Hardware | Any GPU; report relative speedup vs self-baseline | Treating absolute hours as comparable to the 4090 board |
| Hyperparams | Schedule changes justified by underfit regime / matched steps | Pure hyperparameter search as the “idea” |
(1) Official repo baseline — 5.401 h on RTX 4090, defines target loss 3.3821.
(2) F1 — stock recipe on our A100-40; speedup denominator.
(3) m0_anchor — 800-step proxy reference for cheap A/B only; never appears in the official claim.
What the agent is (and is not)
We call it an “agent” because it is a goal-directed, multi-phase system with memory, tools, and a control loop — not because it chatters. Humans set kill gates and recipes; Modal executes the DAG; the volume is durable memory; the kit is the tool surface.
log.csv (train/val/sliding/EMA), config.json, triage_stats.json, billing probes, volume listings. Every keep/kill is trajectory-based, not a single snapshot.
Data shards, features, scores, triage bins, priors, run dirs, LAUNCH markers, runs_logs.zip. Survives container death. Concurrent-writer safe (v2).
Train, score, triage, prior-build, summarize — each a Modal function with hard timeouts, scaledown windows, and idempotent skip rules.
Remote driver: phase DAG, spawn/fan-out, join. modal run --detach decouples local client; panic stop is one CLI command.
Playbook ideas behind flags (defaults = stock). Negative results are first-class outputs. Time-scale hparams re-derived at proxy length.
No tool-calling LLM in the loop. Humans write gates and interpret curves. The “agent” is the experiment runtime + method, not a conversational planner.
Layered architecture
GitHub layout — rainbowpuffpuff/nocap
Private working repo until email submission. Mirrors the agent’s brain (docs), body (kit), hands (notebooks), and memory dumps (results).
Companion workspace (this machine): Modal CLI scripts, smoke writeup, scale-verify bundle, local archive of runs/logs under software_i_built/modal/.
Kit modules — tool surface
Design contract of the trainer: with all new flags at defaults, semantics match the stock baseline (model, loss, schedule, timing protocol). Every experiment is opt-in.
| Module | Role | Key I/O |
|---|---|---|
| train_gpt2_t4.py | Instrumented GPT-2 trainer (d12 / d2 smoke). RoPE, RMSNorm, optional SwiGLU, Canon, parallel block, MTP, loss shaping, prior, EMA, z-loss, sliding val. | bins → runs/<name>/ |
| get_data.py | Fetch FineWeb pre-tokenized chunks from HF into volume/local. | → data/*.bin |
| triage_features.py | Stage-A doc features (len, unique, top, rep8, digit, punct). | → features/*.npz |
| score_docs.py | Stage-B: per-doc mean CE under a trained ckpt; per-shard skip = resumable. | ckpt + bins → scores/*.npz |
| triage_select.py | Junk rules + optional score band + 10% random rescue + optional curriculum order. | → triage_*/fineweb_triage_*.bin |
| build_unigram_prior.py | Corpus unigram log-prior for fixed logit bias. | → unigram_prior.pt / prior_full.pt |
| eval_sliding.py | Boundary-aware sliding-window val (report alongside official disjoint). | ckpt → Δ vs disjoint |
| summarize_runs.py | Table over all run dirs; report zips logs without final.pt. | → console + zip |
| modal_nocap.py | Full experiment DAG as Modal App; phases idempotent; detach-safe orchestrator. | volume as bus |
Trainer flag groups (opt-in ideas)
--logit_prior · --ema_beta · --ema_start_frac · --warmdown_iters · --input_bin (triage_*) · --val_stride · --precision bf16 · --compile 1
--mtp_* · --loss_shape / --ls_* · --mlp swiglu · --canon · --parallel_block · --zloss
Data pipeline architecture
FineWeb10B pre-tokenized GPT-2 BPE: up to 50×100M tokens. Competition budget cap 5B; baseline consumes 2.5B. Our F2 path uses a curated subset scored and banded from 28 chunks.
| Artifact (scale verify, 2026-07-09) | Value |
|---|---|
| docs_total / docs_kept | 4,045,481 → 2,927,881 |
| tokens_total / tokens_kept | 2.796B → 1.922B (kept_frac ≈ 0.687) |
| Score band (mean-loss) | [4.1887, 5.1831] = pct 10–80 of 4,045,472 docs |
| scores/*.npz | 28 shards (expect 28) |
| triage_full bins | 20 shards · ≈3.58 GiB |
| prior_full.pt | ~198 KiB · unigram CE 7.6690 nats |
| Dominant rule_hit | score_band 1,213,639 · unique 33,465 · len 9,532 |
Scoring is offline like the baseline’s own pre-tokenization cost narrative: GPU prep is honest and disclosed in IDEA.md. Training then sees a higher-signal stream (~1.22 epochs of 1.92B curated tokens cycled toward ~2.5B consumed). Proxy isolation: w2_databand alone was −0.051 nats @400 vs raw stream.
Modal orchestrator — control loop
App name: nocap-wave2. Image: debian slim + torch/numpy/huggingface_hub/tiktoken. Volume: nocap version=2. GPU string: A100-40GB (never bare A100).
Phase catalog
Invocation shape
# local entrypoint only spawns the remote orchestrator — safe to close the terminal modal run --detach kit/modal_nocap.py --phase all modal run --detach kit/modal_nocap.py --phase scale --chunks 28 --band 0.10,0.80 modal run --detach kit/modal_nocap.py --phase f2 # panic / observe modal app stop nocap-wave2 modal app list modal volume get nocap runs_logs.zip ./runs_logs_modal.zip
Confirmed recipe brain
Every component proxy-validated at matched steps / hardware. Stack additivity ~70–100% at proxy scale.
| Lever | Mechanism | Proxy effect | Status |
|---|---|---|---|
| Perplexity-band curation | Keep docs in mean-loss pct 10–80 + junk rules + 10% anti-drift rescue | −0.051 @400 | SHIP |
| Unigram logit prior | Fixed corpus log-bias on logits; start ~7.7 not 10.8 | −0.047 @400 | SHIP |
| Shortened warmdown | Early LR decay wasteful while still underfit | −0.013 @800 | SHIP |
| Sliding-window eval | Boundary-aware val; reported with official disjoint | Δ 0.019@800 → 0.060 full (F1) | SHIP (dual-report) |
| EMA weights | β scaled to horizon; dual-eval at end | lag artifact at proxy | MAINTAIN free |
| Stack (data+prior+WD) | Composition for F2 | −0.056 raw / −0.074 w/ eval @800 | F2 RECIPE |
Killed with evidence (submission material)
+11% step time and worse loss @800. Consistent with small-model NLL cautions. Configs + curves archived.
~0 cost, never ahead (+0.007 @800). Val includes every token — down-weighting is a mid-run tax.
Big early lead evaporates by step 800; ends ≈ baseline (4.78).
−0.078 nats (largest training-side effect) but +30% step time naive → net loss. #1 FUTURE lead if fused ≤2% tax.
~1.6% faster, loss-neutral-or-better (−0.005). Out of F2 for composition risk only.
Baseline sits at exactly 20 tok/param (Chinchilla isoline). Resizing unstable sign, ≤2%.
Method — how the agent decides
- Pre-registered kill gates before each arm (RUNBOOK + playbook).
- Matched steps on identical GPU type when step times equal; matched wall-clock when they don’t.
- Paired within-run measurements where possible (immune to run noise).
- Re-derive every time-scale hyperparameter at proxy length (EMA-β lesson: 0.999 full-run ≠ 0.99 proxy).
- Gates on trajectories, not snapshots (sliding Δ grows through training).
- Negatives as first-class outputs — README invites them; we archive configs + CSVs.
- Effects < 0.01 CE need two seeds before keep/kill.
- Pin environment across accounts (torch version drift caused 802 vs 753 ms/step).
model d12 · batch 8 · grad_accum 16 · seq 1024 → 131,072 tok/step (¼ of full 524,288) · LR 4.5e-4 · warmup 128 · warmdown 160 (or 40 with EMA) · val every 100 · bf16 · compile 1 · zloss 1e-4 (F1 drops zloss).
Full finals: 4768 steps · batch 16 · accum 32 · LR 0.0018 · warmup 256 · warmdown 1024 stock / 256 stack · val every 128.
Cost safety, idempotency, serverless semantics
Serverless: no long-lived GPU. timeouts 1h (train) / 8h (finals). scaledown_window=10s. Worst hung arm ~$2.10; absolute multi-hang cap ~$15.
Safe re-run of any phase. score_docs skips per-shard .npz. prep skips prior/features/triage if present.
LAUNCH + log.csv freshness <15 min → fail loud instead of racing twin trainers (learned after a silent double score ~$2 tax).
Volume v1 vs v2: scripts request version=2; creating with default v1 breaks concurrent writers — recreate with --version=2.
Cloudflare 403 on kit download: default Python-urllib UA blocked inside Modal; send curl UA from ensure_kit().
Image build free, cold start billed: smoke in-container 14s ≈ $0.008 GPU; actual bill ~$0.009 with pull.
Live state — 2026-07-09
| Milestone | Result | Status |
|---|---|---|
| Wave-1 (Colab T4) | fp16 parity, prior, sliding Δ growth, warmdown waste found | DONE |
| Smoke (Modal) | A100 lifecycle, volume write, ~$0.009 | DONE |
| Wave-2 (Modal) | m0 4.7788 · stack 4.7232 · databand −0.051 · MTP/shape kill | DONE |
| Wave-3 (Modal) | Canon −0.078 but +30% time · parallel banked · SwiGLU kill | DONE |
| Scale (acct 1) | 28 scores · prior_full · triage_full 1.92B tokens | DONE |
| F1 (acct 2) | Crosses 3.3821 @ step 4736 = 229.1 min · final 3.3795 | DONE |
| F2 record (acct 1) | Stack on triage_full + prior_full + short WD + EMA dual-eval | PENDING |
| Official speedup | (13748s − F2_time) / 13748s | AFTER F2 |
Notebook: notebooks/finals/nocap_f2_account1.ipynb (or modal run … --phase f2). Asserts 20 curated shards + prior_full present. Insurance ckpt at step 4352 before final warmdown. Success = first official val ≤ 3.3821 before step 4768; earlier crossing = larger speedup.
Future architecture (post-finals leads)
From FUTURE.md — ranked design work that does not block F2.
| # | Lead | Why it matters |
|---|---|---|
| 1 | Fused Canon layers | Validated −0.078 nats; kill the +30% transpose/conv tax (elementwise k=4 / channels-last / careful Triton). ≤2% step cost → headline win. |
| 2 | Parallel attn+FFN (+ fused projections) | Already 1.6% faster, loss-neutral; fused [QKV|FFN] GEMMs toward 3–6%. |
| 3 | Top-K head + prior tail-bucket + late unlock | Original idea; ~5–6× head FLOPs cut for most of training; extends prior narrative. |
| 4 | Adaptive two-tier softmax | Conservative cousin of #3; −16–18% step time niche. |
| 5 | Harder band / curriculum order | Already implemented knobs; never raced at finals scale (~$0.35 A/B). |
| 6 | EMA re-test in flat endgame | F2 dual-eval may claim free early cross on EMA weights. |
| 7–8 | Sizing race · 4090 replication | Write-up / board-comparable absolute hours. |
How to run the agent
# once per workspace pip install modal modal setup modal volume create nocap --version=2 # cheap proof (~$0.01) modal run kit/modal_nocap.py --phase smoke # proxy DAG (~$4–6, 60–80 min wall) modal run --detach kit/modal_nocap.py --phase all # full curated dataset for F2 modal run --detach kit/modal_nocap.py --phase scale --chunks 28 # finals modal run --detach kit/modal_nocap.py --phase f1 # stock baseline modal run --detach kit/modal_nocap.py --phase f2 # record stack # fetch evidence modal volume get nocap runs_logs.zip . python kit/summarize_runs.py runs
01 Briefing → 02 Playbook → 03 Wave-1 retro → 04 Campaign report → 05 this architecture map → FUTURE.md → RUNBOOK.md
modal_nocap.py · smoke-test-writeup · scale_verify_bundle · monitor.py · nocap_archive · modal-reference.md
One-line summary
A serverless, volume-backed experiment runtime that runs pre-registered keep/kill races on an instrumented GPT-2 trainer and a two-stage FineWeb triage pipeline — now holding a validated stack (curated data + unigram prior + short warmdown + dual eval) and a measured F1 denominator (3.82 h to 3.3821), waiting only on F2 to mint the official speedup claim.