An agent that finds a faster path to val loss ≤ 3.3821

The environment is the BottleCapAI NoCap-Test hiring challenge: train GPT-2 124M on FineWeb to the baseline’s final validation loss, faster.

Competition rules require a honest relative speedup on our hardware. That is necessary but not sufficient for forensic-grade claims: unpinned images, compile warmup, and GPU-class noise can move wall-clock by several percent — enough to fake a “win” or hide a real one. This section states both the official rules and the extra methodology we enforce.

A. What the competition already requires

B. What we add for forensic accuracy (beyond the README)

C. Torch audit (why indeea shows 2.13)

D. Claim classes (what is allowed to publish)

E. Seal procedure (mandatory before any speed claim)

1. Code pin: TORCH_PIN = "torch==2.13.0" in modal_nocap.py + indeea_proxy.py (done; log full 2.13.0+cu130 string).
2. Smoke on clock account rainbowpuffpuff → fingerprint torch + A100-40.
3. F1_seal on rainbowpuffpuff: full 4768 stock F1 — DONE · cross @4736 · \(T\)=3.866 h · final 3.3798 · torch 2.13.0+cu130.
4. Research proxies on indeea = ordinal ranking only.
5. Claim final also on rainbowpuffpuff vs that F1_seal — no published cross-account cardinal %.
6. If claim is 6–10%, prefer 2nd seed; if <~6% single-seed, do not publish as win.

F. Meta AI forensic review (2026-07-11)

External skeptical methods review (Meta AI browser skill). Thread e5e853d3-e8e8-4afa-91f3-e8f525078f98. We adopt the fixes; we do not adopt factual errors.

We call it an “agent” because it is a goal-directed, multi-phase system with memory, tools, and a control loop — not because it chatters. Humans set kill gates and recipes; Modal executes the DAG; the volume is durable memory; the kit is the tool surface.

Private working repo until email submission (synced 2026-07-11, commit family 17b9e95+). Mirrors brain (docs + literature), body (kit), hands (notebooks + indeea scripts), and memory (results logs — no multi‑GB weights in git).

Not in git (local / Modal only): multi‑GB *.pt, FineWeb bins, score npz, Modal tokens. Restore via volume_backup / CLI get-put.

Companion machine workspace: software_i_built/modal/ — full archives, Cloudflare deploy source (cf-pages-nocap-agent/), live pulls (pull_wave4_*, nocap_archive/).

Design contract of the trainer: with all new flags at defaults, semantics match the stock baseline (model, loss, schedule, timing protocol). Every experiment is opt-in.

Trainer flag groups (opt-in ideas)

FineWeb10B pre-tokenized GPT-2 BPE: up to 50×100M tokens. Competition budget cap 5B; baseline consumes 2.5B. Our F2 path uses a curated subset scored and banded from 28 chunks.

App name: nocap-wave2. Image: debian slim + torch==2.13.0 (pinned) + numpy/huggingface_hub/tiktoken. Volume: nocap version=2. GPU string: A100-40GB (never bare A100). Fingerprint on every smoke.

Phase catalog

Invocation shape

# local entrypoint only spawns the remote orchestrator — safe to close the terminal
modal run --detach kit/modal_nocap.py --phase all
modal run --detach kit/modal_nocap.py --phase scale --chunks 28 --band 0.10,0.80
modal run --detach kit/modal_nocap.py --phase f2

# panic / observe
modal app stop nocap-wave2
modal app list
modal volume get nocap runs_logs.zip ./runs_logs_modal.zip

Every component proxy-validated at matched steps / hardware. Stack additivity ~70–100% at proxy scale.

Killed with evidence (submission material)

Closed-loop agent + 72 literature objects in academic_lit/ (core BIB.md + tangential BIB_TANGENTIAL.md). Time-to-target decomposes as:

T_cross ≈ N_steps(data, opt, schedule, arch) × t_step(arch, systems) + T_fixed(compile, val)

What the 2026-focused literature implies

Ranked experiments (indeea) — literature-backed

Full cards: academic_lit/CARDS/ · synthesis: academic_lit/SYNTHESIS.md · related work draft: RELATED_WORK.md

Two active ~$30 Modal workspaces split by forensic role, not by GPU class. Both stay on A100-40GB + pin torch==2.13.0 until campaign end (VRAM ~10/40 GiB underused; we accept that for a stable clock).

What runs where (checklist)

Retired / archive accounts (do not burn for new science)

rainbowpuffpuff — clock commands (when approved)

# profile is local name = workspace rainbowpuffpuff
MODAL_PROFILE=rainbowpuffpuff modal volume create nocap --version=2
MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase smoke
# confirm /vol/env_fingerprint.txt → torch 2.13.0 + A100-40

MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase data --chunks 28
MODAL_PROFILE=rainbowpuffpuff modal run --detach modal_nocap.py --phase f1
# → runs/f1_baseline (or f1_seal name) = sole T denominator

indeea — research commands (when approved)

# always dry-run first; --execute only when approved
cd software_i_built/modal/indeea_scripts
./00_check.sh
./01_seed_val_data.sh --execute
./10_proxy_control.sh --execute
./15_proxy_parallel.sh --execute
# mild schedule only — not extreme short-WD finals
ARM=wd160 ./12_proxy_schedule.sh --execute

MODAL_PROFILE=indeea modal run indeea_proxy.py --arm control --iters 800 --input curated

Legacy anchor #indeea redirects conceptually to §13 Dual accounts.

Goal: disjoint val ≤ 3.3821 with \(T < T_{F1\_seal}\) by a margin that survives ~6% noise, A100-40 + torch==2.13, claim on rainbowpuffpuff, story that is not “paste Modded-NanoGPT.” F2b proved quality stack alone does not win the clock — we need \(N^*\) and/or \(t_{step}\) levers under fair-clock rules.

Tier 0 — clock (mandatory)

Tier 1 — highest EV for a real win (implement then run)

Tier 2 — ordinal screens (indeea)

Kill schedule/data fast if Δ < 0.01 CE; put remaining $ into Muon + one full claim. Proxies never publish board %.

Phased execution

# PHASE A — CLOCK (rpp)  ~$10
MODAL_PROFILE=rainbowpuffpuff modal volume create nocap --version=2
MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase smoke
MODAL_PROFILE=rainbowpuffpuff modal run modal_nocap.py --phase data --chunks 28
MODAL_PROFILE=rainbowpuffpuff modal run --detach modal_nocap.py --phase f1

# PHASE B — SCREEN (indeea) while coding Muon  ~$8–12
cd indeea_scripts
./00_check.sh
./01_seed_val_data.sh --execute
./10_proxy_control.sh --execute
./15_proxy_parallel.sh --execute
# + implement Muon in train_gpt2_t4.py

# PHASE C — MUON DECISION (indeea)  ~$3–5 after kit
./11_proxy_muon.sh --execute   # unblocked only after port

# PHASE D — CLAIM (rpp)  ~$9–18
# if Muon promotes: Muon + parallel + stack
# else: parallel + stack + best data/schedule nominee
# if |X| in 6–10%: 2nd seed; if <~6%: do not publish as win

Skip (do not burn credits)

• Extreme short-WD finals · re-score whole FineWeb for another static PPL band
• Historical F1 as denominator · cross-account published %
• Single-seed <~6% “win” · torch/GPU shopping · blind Modded-NanoGPT dump
• 3× F1_seal “for science” without a claim plan

This is the campaign’s main optimizer bet. After data+prior+EMA+long warmdown (F2b) hit quality but not earlier first-cross, we needed a lever that cuts steps-to-target \(N^*\), not just step time. Muon is that lever at proxy scale. Full claim still waits on sealed \(T_{F1}\) on rainbowpuffpuff.

What Muon is

Why Muon for NoCap (campaign logic)

Pre-registered card: private docs/academic_lit/CARDS/B-Muon.md — promote if proxy dominates AdamW early; claim only with sealed fair-clock \(T\).

Literature we cite (not a paste of Modded-NanoGPT)

Kit implementation (what we actually run)

# indeea proxy shape (ordinal)
MODAL_PROFILE=indeea ./11_proxy_muon.sh --execute
# or direct flags once kit override is live:
# --optimizer muon --muon_lr 0.02 --iters 800 --input curated (+ prior)

# claim shape (after F1_seal) on rainbowpuffpuff only
# Muon + parallel + curated + prior · full 4768 · report train_time to first disjoint ≤3.3821

indeea protocol (how the board was run)

• Hardware / pin: A100-40 · torch==2.13.0 (same CLI stack as F2b / Wave-4 — not a cheat kit).
• Proxy length: 800 steps primary; one 1600-step horizon check on best compose.
• Base recipe: d12 · batch 8 · grad_accum 16 · seq 1024 · bf16 · compile 1 · val every 100.
• Denominator: ix_control AdamW + curated + prior → val@800 4.731 (step0 val ~7.88 from prior).
• Metric for ranking: disjoint val at matched steps. Never published as % vs historical F1.
• Kill gates (card): worse than AdamW by >0.02 CE; or step tax >10% without ≥0.03 better val; or NaN/explosion.

Full ordinal board (2026-07-11) — COMPLETE

What the board taught us

Fair-clock constraints (Muon-specific)

Status vs next action

Offline study corpus: research/ai-ml-foundations-books/ (HF ai-ml-foundations-book-collection, 25 PDFs). Per-book TOCs: tocs/*.md · full dossier: NOCAP_RESEARCH_FROM_BOOKS.md. Not training data — NoCap allows FineWeb only; this section is method research for FineWeb-legal flags.

High-relevance books (why these, not all 25)

Park for this race: multi-agent RL, fairness institutional chapters, pure product AI-eng without a trainer flag.

Empirical board ↔ book themes

Ideas derived → future experiments (gated on claim_muon_wsd)

Live run: lobitodser · claim_muon_wsd · app ap-cEFn… · Muon+par+curated+prior · no EMA · WSD-like. Adaptations below fire only after that result is known.

Chapter → flag sketch (implementable on our kit)

Reading order (team, not full curriculum)

1. Algorithms for Optimization — momentum, Adam, Newton (tocs + PDF)
2. Understanding Deep Learning — optim + gen + norm
3. Build LLM from Scratch — pretrain hparams
4. Murphy Intro / Bishop — MAP, priors
5. MML — matrix tools for Muon writeup

Structured ideas from Algorithms for Optimization, Understanding Deep Learning, Build a Large Language Model (From Scratch), and Murphy PML Intro. Each idea: why it fits our FineWeb d12 one-GPU setup, why not, and what experiment to run (or park). Source: research/book-ideas/ideas_by_book.json.

Algorithms for Optimization (53 ideas)

PDF: Algorithms for Optimization.pdf

Understanding Deep Learning (52 ideas)

PDF: Understanding Deep Learning.pdf

Build a Large Language Model (From Scratch) (52 ideas)

PDF: Build-a-large-language-models.pdf

Probabilistic Machine Learning: An Introduction (Murphy) (55 ideas)

PDF: book1 - Probabilistic Machine Learning - An Introduction.pdf

After F2b, Wave-4, indeea Muon board, claim miss, horizon screen, and book research §13d: primary path is live claim_muon_wsd; next bets depend on HIT/MISS (Tier B/C/D above).

# once per workspace
pip install modal
modal setup
modal volume create nocap --version=2

# cheap proof (~$0.01)
modal run kit/modal_nocap.py --phase smoke

# proxy DAG (~$4–6, 60–80 min wall)
modal run --detach kit/modal_nocap.py --phase all

# full curated dataset for F2
modal run --detach kit/modal_nocap.py --phase scale --chunks 28

# finals (per-account profiles — volumes do NOT share)
MODAL_PROFILE=stufflaters   modal run --detach kit/modal_nocap.py --phase f1
MODAL_PROFILE=acalincarol   modal run --detach kit/modal_nocap.py --phase f2
MODAL_PROFILE=deeferentleeg modal run --detach kit/modal_nocap.py --phase f2b
MODAL_PROFILE=stufflaters   modal run --detach kit/modal_nocap.py --phase wave4
MODAL_PROFILE=indeea        modal run kit/modal_nocap.py --phase smoke

# indeea research proxies (dry-run unless --execute)
cd indeea_scripts && ./10_proxy_control.sh   # add --execute when approved

# readiness / transfer (CPU only — no GPU)
MODAL_PROFILE=<name> modal run kit/modal_nocap.py --phase readiness
MODAL_PROFILE=acalincarol   modal volume get nocap triage_full/ ./triage_full/
MODAL_PROFILE=indeea        modal volume put nocap ./triage_full triage_full -f

# fetch evidence
modal volume get nocap runs_logs.zip .
python kit/summarize_runs.py runs