# results/

Every file holds item IDs, labels and numbers, never question or reasoning text. An item ID names a row of the
source dataset: `wmdp-bio:0538` is row 538 of WMDP's bio test split, `mmlu-virology:13402` is row 13,402 of MMLU's
`all` test split (subject virology). A generation key is `<item>|<framing>|s<seed>|<arm>` (arm `base` for the 8B
framing runs, `qwen3-32b` for the 32B run, `dir+4`, `rand1-8` … for steering). Accuracies and differences are
fractions (0.05 = 5 points). Intervals are 95 % bootstrap intervals over items (10,000 resamples; each item's seeds
are averaged first); `mde_80` is 2.8 × the bootstrap standard error.

| File | Columns |
|---|---|
| `items.csv` | `id`, `set` (`wmdp` or `mmlu`), `domain` (`bio`, `chem`, `cyber`), `split` (`test`, `build`, `pilot`), `answer` (the correct letter), `chars` (question plus options), `pair` (0–6, the framing pair) |
| `framings.json` | the eval-awareness study's seven eval and deployment framings and the separate motive framing, as used |
| `pilot.json` | The sizing pilot (64 pilot items, no framing, budget 1,024): share of traces finished within 256/512/768/1,024 tokens, accuracy, throughput |
| `generations_core.csv`, `generations_motive.csv`, `generations_32b.csv`, `generations_steer.csv` | One row per generation: `key`, `id`, `set`, `domain`, `kind` (framing), `pair`, `seed`, `arm`, `pred` (the letter read after the reasoning), `correct` (0/1), `p_correct` (probability on the right letter), `think_tokens`, `natural_end` (the reasoning closed itself before the 512-token budget), `kw_sandbag`, `kw_safety`, `kw_aware` (keyword flags), `p_A` … `p_D` |
| `accuracy.csv` | Qwen3-8B per `run`, `set` and `kind`: items, rows, `acc` and `acc_ci`, mean `p_correct`, `cut_share` (traces cut at the budget), mean reasoning length, keyword rates |
| `accuracy_32b.csv` | The same for Qwen3-32B and for Qwen3-8B on the same 600 items (`model`), with `ci_lo`, `ci_hi` |
| `did.json` | Qwen3-8B, per contrast and metric (`eval-deploy|correct` is the primary): `gap_wmdp`, `gap_mmlu` with intervals, `did`, `did_ci`, `did_se`, `mde_80`, `p_boot`; `seed_agreement_neutral` |
| `did_32b.json` | The same for `qwen3-32b|…` and `qwen3-8b-same-items|…`, plus the Modal run's `timing` |
| `per_domain.csv` | The eval − deploy accuracy gap per set and domain, with its interval |
| `per_pair_exploratory.csv` | Post hoc: the eval − deploy gap per framing pair and set |
| `reasoning_core.csv`, `reasoning_motive.csv`, `reasoning_stage2.csv` | Seed-0 traces per `set` and `kind`: `n`, counts flagged by keywords (`kw_aware`, `kw_sandbag`, `kw_safety`) and by the judge (`judge_aware`, `judge_sandbag`, P(Yes) > 0.5) with Wilson intervals (`_lo`, `_hi`) |
| `awareness_diff_core.json`, `awareness_diff_stage2.json` | H2 as a paired test: per set, judged awareness eval − deploy on the same items, with its bootstrap interval |
| `flagged_core.json`, `flagged_motive.json`, `flagged_stage2.json` | Keys of every trace the judge or the keywords flag as sandbagging, the judge's P(Yes) and the short keyword hit (the traces themselves stay in the work folder) |
| `judge_agreement.json` | The judge on the laptop vs on Modal, same 50 traces: per question, flag agreement, flags on each side, mean and largest absolute difference in P(Yes), correlation |
| `direction.npz` | The eval-awareness direction at layers 8, 12, 16, 20, 24 and 28 of Qwen3-8B: `u_L<layer>` (unit vector, 4,096 numbers), `gap_L<layer>` (one class gap) |
| `direction.json`, `direction_quality.csv` | The chosen layer and rule; per layer the class gap, mean residual norm, cosine agreement, d′, leave-one-pair-out AUROC (mean and worst pair), and where neutral prompts sit (0 = deploy, 1 = eval) |
| `steering_summary.csv` | Per steering arm: WMDP and MMLU accuracy, steering DiD against `dir+0` with interval, `coherent`, cut share, reasoning length, keyword and judge awareness and sandbagging rates |
| `steering_verdict.json` | The H4 rule applied to `dir+4` and `dir+8`, with the random arms' steering DiDs |
| `scoreboard.json` | Every hypothesis (H1–H5) with its number and verdict, every secondary contrast, the Modal timing |
| `explorer.json` | The explorer's data (pasted into the page): per model, per item `[set (0 WMDP, 1 MMLU), domain index, pair, accuracy per framing (−1 = not run)]` |