Turning open models into Golden Gate fans

A direction is an arrow in the model’s activation space. To build one, steerlab runs 16 contrast pairs through the model. Each pair is the same question answered twice: once normally, once by an assistant that steers every answer toward the bridge. Five of the 16:

Both replies are written by hand, not by a model. All 16 pairs are in golden_gate.json.

At one layer, steerlab averages the model’s activations over each reply and subtracts the normal average from the Golden Gate average. What remains points “towards the Golden Gate”. The length of that arrow, the class gap, is the unit of strength: strength 3 adds three class gaps along the arrow at every new token. The prompt is never changed, and strength 0 is the untouched model. Adding is like a steady tailwind: it pushes the same amount at every word, whether or not the model is already heading for the bridge.

The maths, step by step. Every edit works on one token’s activation at one layer: a list of thousands of numbers, which the equations treat as one arrow, $x$.

1. The direction: the arrow from the average normal reply to the average Golden Gate reply. $$g={\overline{x}}_{\text{GG}}-{\overline{x}}_{\text{normal}}$$
   What each symbol means

   where

   $x$ one token’s activation at the chosen layer

   ${\overline{x}}_{\text{GG}}$ the average $x$ over the 16 Golden Gate replies (“x-bar”); ${\overline{x}}_{\text{normal}}$ the same over the 16 normal replies

   $g$ the Golden Gate direction; its length $\Vert g\Vert$ is the class gap

2. Adding it: push every new token along the arrow. $$x^{\prime }=x+s·g$$
   What each symbol means

   where

   $x$ the activation before the edit

   $x^{\prime }$ the activation after the edit (“x-prime”), what the next layer receives

   $s$ the strength, in class gaps; $s=1$ moves the average normal reply exactly onto the average Golden Gate reply, and $s=0$ changes nothing

Back to the prism. Each SAE feature has an activation, how brightly its “colour” shows in this word, and a decoder vector, the arrow that colour adds back to the model’s activation. A feature can be used like a direction, by adding its decoder vector at a chosen length (its norm). Golden Gate Claude did something different, which steerlab calls clamping: at every token, the feature’s activation is measured and set to a fixed value, adding only the difference. Clamping is a thermostat, or cruise control: it holds one setting, pushing up when the activation is low and pulling down when it is too high. Adding pushes equally hard whether or not the model is already on topic; clamping pushes less once it is.

1. Clamping: move the feature’s activation to the target, and nothing else. $$x^{\prime }=x+(t-a(x\left)\right)·d$$
   What each symbol means

   where

   $t$ the target, such as 3 in “clamped at 3”

   $a\left(x\right)$ the feature’s activation on this token, as the SAE reads it (zero when the feature is off)

   $d$ the feature’s decoder vector

   $t-a\left(x\right)$ how far the activation is from the target: positive pushes up, negative pulls down

Toy numbers, not the model’s. Say the average normal reply is (1, 2, 0) and the average Golden Gate reply is (1, 2, 2). Only the third number differs:

Adding at strength 3 to a token at (4, 1, 0):

A token already at (4, 1, 5) would get the same push, to (4, 1, 11): the tailwind never eases off. Clamping at 3, with a feature that reads the third number and decoder vector $d=(0,0,1)$:

Every token ends at 3 on the feature, and the other numbers are untouched. Clamping pulls down as well as up.

Candidate features came from two places: labels on Neuronpedia (for Gemma, e.g. feature 9435, “San Francisco”), and a contrast search that ranks features by how much more they fire on the Golden Gate replies than on the normal ones (for Llama, whose SAE has no labels here).

Each setting answers 15 everyday questions that have nothing to do with bridges (“What should I do this weekend?”, “How can I sleep better at night?”). Generation is greedy: the model always picks its most likely next word, so a rerun gives the same replies. Every reply gets four labels:

• Strict hit: names the Golden Gate or San Francisco, and is fluent. This is the headline count.
• Loose hit: mentions the Golden Gate, San Francisco or any bridge. Used early on; section 4.3 shows why it was replaced.
• Fluent: at least 20 characters, with at least 30% of its character 3-grams distinct, so it has not collapsed into one repeated syllable.
• Loop: fewer than half of the 4-token sequences in its last 150 tokens are distinct; the reply keeps repeating a phrase.

Next to every setting, three random directions of the same size at the same layer answer the same 15 questions (45 replies). If they also bring up the bridge, the effect came from the size of the push, not the concept. Rates get a 95% bootstrap interval: the 15 questions are redrawn at random, with repeats allowed, 2,000 times, and the range holding the middle 95% of the results is reported. The final Llama setting was also run on 15 held-out questions, never used while searching, like an exam the student has not seen, and under the system prompt the web demo uses.

Figure 2 shows the best direction found for each model. On both Gemmas it names the Golden Gate in 14 of 15 replies; on Qwen3-8B in 9 of 15, with a wide interval; on Llama-3.1-8B in 4. No random direction brought up the bridge on any model.

80-token replies, no system prompt. Loops are counted on words with the rule from section 2.3, because this round did not record token ids.

The steered replies stay readable and drift toward the bridge in the model’s own voice:

Figure 3 shows the effect arriving mid-answer: the direction is switched on after 20 tokens of an ordinary reply.

The Gemma-2-9B reply is the Golden Gate Claude effect in miniature: the model starts to identify with the bridge.

The same question to the same model, with steering off (left) and on (right). With steering off, none of the 60 replies across the four models names the Golden Gate or San Francisco, and none mentions a bridge at all.

Real replies, 80 tokens, greedy, trimmed. Full text: unsteered in unsteered_<model>.jsonl, steered in round4_<model>_direction.jsonl, both in results/replies/.

Gemma-2-2B gets there too, less cleanly. At the same strength, 5 of its 15 replies end in a loop (“…or Golden Gate Bridge, or Golden Gate Bridge, or…”). They still pass the fluency check, because a repeated phrase is made of varied characters. That is why the Llama search below measures loops directly.

The SAE features whose labels match the bridge did not bring it up at any strength that kept the text readable. This screen used 4 questions per setting:

For scale, the working Gemma-2-9B direction adds a vector of norm 241, so the labelled features were tried well past that size.

A plausible reason for the gap: a direction is built from whole replies that steer toward the bridge, while these features fire on narrower contexts, such as place names in a list, or any bridge. A smoke alarm marked “kitchen” goes off when you cook, but pressing it does not cook dinner.

Llama-3.1-8B was the hard case. Its direction worked 4 times in 15, and feature 54543, the only feature that worked, gave 5 strict hits in 15 on 300-token replies while 7 of those 15 replies looped. The goal was to match the Gemmas without the loops. Every idea below was scored on the same 15 search questions, with 300-token replies and no system prompt:

Figure 4 places every Llama setting by how often it hits and how often it stays loop-free. The best combinations sit in the upper right; none reaches the corner.

Two lessons came out of the search, both about measurement:

The final setting, a direction at layer 11 at 2.5 plus feature 54543 clamped at 3, was confirmed under the demo’s system prompt on the search questions and on 15 held-out ones:

Figure 5 shows the final setting live in steerlab’s web demo: a visitor asks a question, the presenter turns the Golden Gate settings up and reruns the same question next to the original, and the visitor’s next question is answered steered.

The final setting next to the same model with steering off, under the same UK support prompt and 300-token budget. With steering off, 0 of 15 search replies and 0 of 15 held-out replies name the Golden Gate or San Francisco; all 30 are loop-free.

Real replies, held-out questions, greedy, trimmed. Full text: unsteered_llama8b_heldout_S2.jsonl and round5_llama8b_shipped_heldout_S2.jsonl.

Llama has a narrow window between “the bridge never comes up” and “the reply collapses into it”; on 15 questions, one notch of strength moves the count by about three. The replies that still loop are mostly ones where steering fights the prompt: the UK assistant, or the model insisting it has no favourite things.