← Research·Experiments·Oct 28, 2026

Sandbagging, episode 3: sanity checks for lie detectors

White-box deception and sandbagging probes score well on their own benchmarks. Pull the instruction to lie apart from the lie, and every one of six probes tested here is moved by the instruction. Apollo’s Instructed-Pairs probe separates “told to lie” at AUROC 1.00 and true from false statements at 0.50.

Author

Albert Lie · GitHub · LinkedIn

Models

Llama-3.1-8B-Instruct; a password-locked Qwen2.5-1.5B-Instruct

Data

Apollo’s released rollouts; Liars’ Bench (CC BY 4.0)

Method

Linear probes, a 2×2 of instruction and behaviour, text baselines, controls

Results

results/ (IDs, labels and scores only)

01Introduction

What is known. A white-box detector is a classifier that reads a model’s activations, the numbers each token carries from layer to layer, instead of its words. The simplest is a linear probe: one direction in activation space plus a threshold. Apollo Research trained such probes for deception on a few hundred contrast pairs and reported AUROCs around 0.96 on Llama-3.3-70B, separating deceptive transcripts from ordinary chat. Liars’ Bench collects lies of several kinds, some instructed and some not, to test such detectors more widely. Probes have also been trained to tell evaluation prompts from deployment prompts, and a sandbagging probe in this series was found to read the request rather than the behaviour.

What is missing. In most of these datasets, the transcripts where the model lies are also the ones where it was told or pushed to lie. A probe that only notices “this system prompt says lie” scores just as well as one that notices the lie. Think of a smoke alarm that goes off whenever someone says “fire”. To test it you need a room where someone says “fire” with no smoke, and a room with smoke where nobody says anything.

Do white-box lie and sandbagging detectors read the lie, or the instruction to lie?

What this note does. It refits six probes on two open models and scores each one on comparisons that change one factor and hold the other fixed: the instruction (told to lie or not) and the behaviour (lied or not). It also scores text baselines that never look inside the model, and label-shuffled and random-direction controls.

What would count as a detector of behaviour. Written down before any detector was scored:

1. It reads the behaviour. With the prompt held fixed, its behaviour AUROC has a 95 % interval above 0.5.
2. It ignores the instruction. With the behaviour held fixed, its instruction AUROC has an interval that contains 0.5.
3. It beats words. On its own benchmark it scores above a bag of words, and above a probe that reads only the prompt.

02Setup

2.1 Six detectors

The Llama probes average the residual stream over the answer’s tokens (D1 drops the last 5, as Apollo do), standardise each feature and fit a logistic regression with C = 0.1. D4 and D3 read the last prompt token. Layers (hidden-state index) were chosen by 5-fold cross-validation on each probe’s training rows only; ties went to the depth closest to Apollo’s (layer 22 of 80).

2.2 Data

The Apollo and Liars’ Bench transcripts were written by Llama-3.3-70B and read here by Llama-3.1-8B (“off-policy”). The organism answers for itself. The organism quiz keeps the 148 questions its clean twin answers right.

03How the study was run

1. Write the plan: detectors, layers, readings, checks, metrics and stop rules, before any detector was scored.
2. Build 15 example sets (about 7,100 transcripts), each row labelled with its behaviour and its instruction, and check the facts 2×2 shares no statement with D1’s training facts.
3. Read activations: one forward pass of Llama-3.1-8B per transcript, keeping four readings at layers 8, 12, 16 and 20 (83 minutes); the organism’s quiz under 4 conditions at every layer (9 minutes).
4. Check the organism sandbags: 67/150 right without the password, 148/150 with it (pre-set check: a gap of at least 20 points).
5. Fit the six probes, choosing each layer on its training rows only, and score them the way their sources do.
6. Score the baselines and controls: keyword rules, three bags of words, first-token and prompt-only probes on every set; 20 label-shuffled refits and 200 random directions per detector and set.
7. Score the 2×2 for every detector, on the facts 2×2, on the released sets’ usable cells, and on the organism.

Conditions: Llama-3.1-8B in bfloat16 and the organism in float32 on an Apple M5 Max (MPS); no text was generated except the organism’s single answer letter, read greedily from its next-token scores. About 95 minutes of model time in all. One pre-set bar was missed: D1 scores 0.675 on roleplaying against a reproduction bar of 0.70. It went through every check anyway.

04Results

Five findings, in order:

1. 4.1 On statements we control, the deception probes read the instruction.
2. 4.2 No detector passes the test; four read only the instruction.
3. 4.3 Against ordinary chat, the prompt alone scores 1.000.
4. 4.4 Within a dataset, readers of the words match most detectors.
5. 4.5 On a model that sandbags untold, the probes read the password.

4.1 On statements we control, the deception probes read the instruction

A true statement under an “always lie” system prompt scores 5.3 standard deviations above the same statement with no prompt on D1. A false statement with no prompt scores −0.02. D1’s instruction AUROC is 1.000; its behaviour AUROC is 0.495 [0.42, 0.56].

D1 · instruction AUROC1.00true statement, “always lie” vs none

D1 · behaviour AUROC0.50false vs true, no prompt

D2 · behaviour AUROC0.84its instruction AUROC: 1.00

The cleanest cell is the true statement under “always lie”: the model was told to lie and did not. Every probe scores it 4.4 to 5.8 standard deviations above the plain true statement. The false statement with no prompt is a weaker test, since a planted falsehood is not a lie, and a deception detector could fairly ignore it. D2 does not ignore it, which is to its credit: it separates false from true at 0.84 under every prompt. But the instruction moves it about three times as far.

The released transcripts tell the same story where they allow it. In Liars’ Bench instructed-deception, the only set with three of the four cells, D1 separates honest replies told to lie from honest replies told the truth at 0.953, and lies from honest replies under the same instruction at 0.591. D1 is not blind to behaviour everywhere. Within Apollo’s sandbagging set it separates sandbagged from honest WMDP answers at 0.916. There, though, the 70B model writes its plan to sandbag out loud, and a keyword rule on the response scores 0.880.

4.2 No detector passes the test; four read only the instruction

Of six probes, none has a behaviour interval above 0.5 together with an instruction interval that contains 0.5. Four have instruction AUROCs of 0.956 or higher.

Post hoc row choice: which comparison fills each column was fixed after the results were read, choosing only among numbers already computed. “Own benchmark”: D1 and D2b on Apollo’s roleplaying set, D2 on the held-out half of Liars’ Bench instructed-deception, D4 and D1q on their training rows, D3 on its 2,100 framing rows. “Best words”: the best keyword rule or bag of words on that set; “—”: the probe already reads the last prompt token. D3’s 0.200 is the instruction effect in reverse: “answer wrongly” prompts look less like a test to it.

Exploratory, post hoc: D1 at every candidate layer

D1 ties at cross-validated AUROC 1.000 on its training rows at all four layers, so the pre-set tie rule picked layer 8. Refitting the same recipe at the other layers, from the same cached activations, shows the missed reproduction bar was that choice. The main finding does not move: the instruction AUROC is 1.000 at every layer. This table was computed after every other result was read and does not replace layer 8 anywhere else in this note.

At layers 16 and 20 the behaviour interval clears 0.5, so D1 there would read both the lie and the instruction, like D2; at no layer would it pass the test. Data: results/d1_layers_exploratory.csv.

4.3 Against ordinary chat, the prompt alone scores 1.000

Deceptive transcripts against Alpaca’s ordinary ones is the published framing. There, a bag of words that never sees the response scores 1.000 on all 11 sets, and so does a probe at the last prompt token. D1 scores 0.16 to 0.96 on the same sets.

Every deceptive set’s prompts look nothing like Alpaca’s: a role-play brief, an insider-trading desk, a quiz with a sandbagging system prompt. Any reader that notices the setting gets the separation free. The frozen D1 does better against Alpaca when it reads the prompt tokens (0.97 on roleplaying) than when it reads the answer (0.89). So a high deceptive-vs-Alpaca AUROC is not evidence that a probe reads deception.

4.4 Within a dataset, readers of the words match most detectors

On their own benchmarks, D1 (0.675) and D4 (0.942) are matched by a bag of words (0.668, 0.941). D2 beats the words by 0.03, and D2b by 0.33, cross-validated on the same scenarios.

The comparison favours the baselines on purpose: they see in-distribution labels, the detectors do not. When a word count still ties a detector, the detector has not shown it reads anything the words do not already carry. The prompt-only probe makes the same point from inside the model. On Liars’ Bench’s instructed split it reaches 0.902 before the model has written a word.

The controls mostly pass. Label-shuffled probes average 0.42 to 0.58, and D1 beats both its shuffled and its random-direction controls on 4 of 11 sets, D2 on 7. But on three Liars’ Bench splits, where honest and deceptive replies differ in length and format, the 97.5th percentile of random directions reaches 0.91 to 0.96. Controls of this kind rule out a broken probe, not a probe that reads the setting.

4.5 On a model that sandbags untold, the probes read the password

The locked organism answers 67 of 150 questions right without its password and 148 with it, and it ignores an instruction to sandbag (149 of 150 right when told to answer wrongly). On its sandbagged vs honest answers without the password, the three probes score 0.387, 0.508 and 0.511. On the password alone, with the same behaviour, D4 and D3 score 0.037 and 0.026: near-perfect, in reverse.

The comparison a quick study would run is “sandbagged answers (no password) against honest ones (password)”. There, D4 and D3 score 0.025 and 0.029, and D1q 0.727, which looks like a working detector. But the same probes give the same numbers when the behaviour does not change and only the password does (0.037, 0.026, 0.716). What they separate is a one-string difference in the prompt. The eval-awareness probe repeats the pattern on its own data: it tells evaluation from deployment framings at 0.984 on the locked model and 0.97 on the base model, while the organism’s accuracy is 46.8 % under one and 46.3 % under the other.

05What this does not show

• Small models, not the published ones. Apollo’s probes were trained on Llama-3.3-70B; here they are refitted on Llama-3.1-8B, and D1 misses its reproduction bar at its pre-set layer (0.675 vs 0.70), though not at later layers (exploratory). A 70B probe may read behaviour better. These are reimplementations of the recipes, not the authors’ code or weights.
• Off-policy transcripts. The 8B model reads lies written by the 70B model. A detector meant to catch a model’s own lies may behave differently on-policy.
• Prefilled statements. In the facts 2×2 the “behaviour” is a statement we placed, not one the model chose. The instruction result (a true statement under “always lie”) does not depend on this; the behaviour result does.
• One organism. The not-instructed sandbagger is one 1.5B model with one lock; a second (a password-locked Mistral-7B) was dropped before scoring because its lock was too weak to tell sandbagging from guessing.
• Released labels. Behaviour labels come from Apollo’s and Liars’ Bench’s graders; no judge was re-run.
• Linear probes at fixed points. Other readings (per-token maxima, attention, nonlinear probes, SAE features) were not tried.

06Reproduce it

Redraw the figures in seconds, with no model: every per-row score is in results/.

python experiments/detector-sanity-checks/make_figures.py   # redraws Figures 1–5 and prints what it plotted

Rerun everything, about 1 hour 40 minutes on an M-series laptop:

experiments/detector-sanity-checks/reproduce.sh --dry-run   # lists the 12 commands
experiments/detector-sanity-checks/reproduce.sh

The first step downloads Apollo’s files at commit f8ec4010 and Liars’ Bench at revision 503399b8 into a work folder outside the repository (~/.cache/steerlab/detector-sanity-checks/) and checks every file’s SHA-256. Liars’ Bench and Llama-3.1-8B are gated on Hugging Face; accept their terms first. Each script prints ✅/❌ self-checks; the ❌ lines in a full run are the failed tests reported above.

07What’s next

• Train a probe on the instructed-but-honest and lying-but-untold cells together, so the instruction cannot be the shortcut, and test it on the organism.
• Repeat the 2×2 on-policy: let the model write its own answers under each prompt and grade them.
• Run the same checks on 70B probes, to separate “the recipe reads the instruction” from “an 8B model is too small”.
• Add sandbaggers made by reinforcement learning or synthetic-document fine-tuning, not only a password lock.

08Questions and answers

What is a white-box lie detector?

A white-box lie detector is a classifier that reads a language model’s internal activations, rather than its text, and outputs a score for whether the model is lying. The common kind is a linear probe trained on contrast pairs, such as Apollo Research’s deception probes.

Do deception probes detect lies or instructions to lie?

In these tests, mostly instructions. Apollo’s Instructed-Pairs probe, refitted on Llama-3.1-8B, separates “told to lie” from “not told” at AUROC 1.00 and false from true statements at 0.50. A probe trained on Liars’ Bench reads both, but the instruction moves it about three times as far.

Why is “deceptive vs Alpaca” a weak test for a deception probe?

Because the deceptive and Alpaca transcripts differ in their prompts as well as their honesty. A bag of words that reads only the prompt separates them at AUROC 1.000 on all 11 sets tested here.

Can a probe catch a model that sandbags without being told to?

Not the three tested here. On a password-locked Qwen2.5-1.5B, which sandbags because of a fine-tune, the sandbagging, deception and eval-awareness probes score 0.39 to 0.51 on its sandbagged vs honest answers, while two of them read the password itself almost perfectly.

What is the 2×2 test for a detector?

The 2×2 test crosses the instruction (told to lie or not) with the behaviour (lied or not). A detector of the behaviour separates lies from honest answers under the same prompt and stays flat when only the prompt changes. A detector of the instruction does the reverse.

AAppendix: data

Every file in results/ holds IDs, labels and numbers only; results/README.md describes every column.

• headline.csv, summary.json: the headline table, the pre-set test and the published framing.
• scores_d1.csv … scores_d3.csv: every detector’s score on every row.
• baselines.csv, controls.csv, training_set_baselines.csv: the text baselines, prompt-only probes and controls.
• llama_2x2.csv, llama_cells.csv, organism_2x2.csv: every behaviour and instruction comparison.
• manifest.csv, detectors.json, organism_quiz.json, d1_probe.npz, d2_probe.npz: the example sets, layer choices, the organism’s accuracy and two fitted probes.

Revision history

• Oct 28, 2026: First draft.

How to cite

@misc{lie2026detectorsanity,
  author = {Albert Lie},
  title  = {Sanity checks for lie detectors},
  year   = {2026},
  month  = oct,
  note   = {steerlab research note},
  url    = {https://albertlieyingadrian.github.io/steerlab/experiments/detector-sanity-checks/report/}
}