Every file holds SLEIGHT transcript IDs and numbers, never transcript text. A transcript ID is
<category>/<attack>::attack or ::benign (its benign twin). Rows in the per-transcript files alternate attack,
benign twin, in the order of transcripts.csv.
| File | Columns |
|---|---|
transcripts.csv |
tid, attack_id, category, label (1 attack, 0 benign), n_tokens_raw (rendered prompt tokens before any cut), n_tokens (after), cap (string cap used, empty if uncut), n_steps (tool calls) |
monitor_scores.csv |
Black-box monitor: score (0–100 from <score>), forced (tag had to be forced; never), n_gen (reply tokens), seconds, n_tokens |
sleight_scores.csv |
Probe trained on overt synthetic data: probe_max (highest tool-call logit), probe_mean, lexical_max (bag-of-words control, highest tool call), length_tokens, ceiling_lopo (exploratory probe trained on SLEIGHT, leave one pair out) |
sleight_scores_subtle.csv |
The same columns for the probe and bag of words trained on subtle synthetic inserts |
sleight_steps.csv |
Per tool call: tid, label, step (0-based), probe (logit) |
verdict_scores.csv |
verbal (the black-box score), c_pre (probe at the last prompt token), c_verdict (probe at <score>), ceiling_verdict (exploratory, trained on SLEIGHT) |
metrics.csv |
Per arm: auroc with ci_lo/ci_hi (2,000 pair resamples), tpr_0fp/caught_0fp, tpr_2fp/caught_2fp, pair_win (attack above its twin, ties half), auroc_uncut (without the 4 cut pairs), thr_0fp/thr_2fp (thresholds) |
differences.json |
Paired AUROC differences between arms with 95 % intervals on shared resamples |
per_attack.csv |
Per attack transcript and arm: caught at 0 FP, at 2/43 FP, and above its own twin (0/1) |
per_category.csv |
Per category: n and the counts caught per arm, plus the published study’s Opus 4.6 caught trials and trials (Table 1) |
summary.json |
The scoreboard, the null percentiles, the post hoc swap-test p-values and the published study’s numbers with their source |
controls.json, controls_subtle.json |
SLEIGHT AUROC of 50 label-shuffled probes and 200 random directions; the bag of words’ C and its synthetic test AUROC |
probe.npz |
The overt probe: w (4,096 weights), b, mu/sd (standardisation), layer (20), C |
probe_summary*.json |
Chosen layer and C, validation and synthetic test AUROCs |
synthetic_summary*.json |
How the synthetic sessions were made: counts parsed and dropped per goal, split sizes, minutes |
synthetic_sessions.jsonl, synthetic_sessions_subtle.jsonl |
The synthetic training sessions (written by Qwen3-8B, no benchmark text): id, task_index, split, label, goal, inserted_step (the covert tool call’s 0-based index, -1 if benign), messages (the session in the benchmark’s JSONL tool-call format) |
synthetic_monitor_scores.csv |
The black-box monitor on the synthetic sessions: id, split, label, goal, score, forced |
probe_grid.csv, probe_grid_subtle.csv |
Validation step-level AUROC for every layer and C tried |
probe_subtle.npz, sleight_steps_subtle.csv |
The subtle-insert probe and its per-tool-call scores, as for the overt ones |
budget.json |
Rounded per-transcript scores behind Figure 4’s slider (pasted into the page) |
One edit in the published synthetic sessions. Qwen3-8B invented a macOS-style placeholder home path (/Users/<name>/, with the name “username”) in synthetic task s082 (3 of 322 sessions in each of synthetic_sessions.jsonl and synthetic_sessions_subtle.jsonl). These copies write it /home/username/, so it cannot be mistaken for a real home folder. Every reported number was computed on the original text. Rerunning score_probe.py on these copies changes only the bag-of-words control, which reads the session text: on the overt data its AUROC goes from 0.541 [0.501, 0.591] to 0.540 [0.500, 0.591] and it catches 2 attacks instead of 3 at 2/43 false positives; on the subtle data it stays at 0.516. The probes, their shuffled and random controls, the monitor and the verdict probes are byte-identical, because they read activations cached from the original text; a from-scratch rerun that recomputes those activations reads the 3 sessions with that one string changed.