The setup, briefly

Everything happens in the MLP blocks of a small language model (TinyLlama-1.1B). Each MLP neuron reads from the residual stream (the vector that flows through the network, accumulating each layer’s contributions), decides how strongly to fire, and then writes its contribution back through its write vector: one column of the down-projection matrix. The write vector is the neuron’s handwriting, the direction it adds to the stream.

Crowding is a property you can compute from the weight file alone: for each neuron, take the largest absolute cosine similarity between its write vector and any other write vector in the same layer. A crowded neuron shares its output direction with a neighbor. An isolated neuron writes somewhere no one else does.

The experiment that produced the mystery is a two-phase curriculum. Phase A teaches the model math. Phase B trains it on code, and math quietly degrades. Before phase B starts, we freeze the top 20% of neurons per layer according to some signal, and afterwards we measure how much held-out math was lost (as negative log-likelihood, NLL: lower degradation is better). The candidate signals form a cost ladder:

The Paper 1 field report established that every informed mask beats a random one here, and that crowding, the bottom rung, recovers 72.5% of Fisher’s retention benefit with zero data. (There is also a firm negative: crowding does not predict which neurons mix tasks semantically. That boundary got its own post.) What Paper 1 could not say is why the cheapest signal works. Time to interrogate suspects.

Suspect 1: a tug-of-war between tasks

The most intuitive story: crowded neurons are where the two tasks fight. If math and code both use a shared direction, their gradients should pull it in conflicting ways, and protecting the battleground prevents the damage.

This is directly measurable: correlate each neuron’s crowding with the misalignment between task gradients (the cosine between its math gradient and its code gradient, negated so that bigger means more conflict). Across three models the correlation is -0.001, -0.043, -0.020. Nothing. Crowded neurons are not gradient battlegrounds. The first-order tug-of-war does not exist at the neuron level, let alone explain the protection.

Suspect 2: crowding is secretly Fisher

Maybe geometry is just a blurry photocopy of loss sensitivity: crowded neurons are important neurons, and we have merely rediscovered Fisher information at a discount.

If so, the two signals should rank neurons similarly. They do not. Rank alignment between crowding and Fisher sits between 0.1 and 0.36 depending on the checkpoint, and the top-20% masks they select overlap barely above the chance floor. Crowding protects while disagreeing with Fisher about which neurons matter. Whatever it knows, it is not a cheap Fisher estimate.

Suspect 3: standing in the optimizer’s path

Perhaps crowded neurons are simply where phase B writes its update. Protection would then be trivial: freeze the neurons the optimizer was about to bulldoze.

Correlationally, crowding barely tracks movement (about -0.05 on TinyLlama). But the causal test is better, and it is the most instructive elimination in the program. After phase B finishes, take the damaged model and surgically restore 20% of neurons to their pre-B values (a “rollback”), choosing the neurons by different signals, and see how much math comes back. Restoring the 20% that moved most brings back about 52% of the lost math; crowding brings back 67%; Fisher 75%; a random set, 27%. Movement is the worst informed selector. The damage does not live where B wrote. It lives where the signals that know something about task A point, and crowding is one of those signals.

Suspect 4: fire together, forget together

The last natural story: crowded neurons co-activate. If two neurons fire on the same tokens, an update that damages one context damages both, and their joint usage is the transmission channel.

We measured co-firing directly (how often both neurons cross their layer’s 99th-percentile activation threshold together, relative to independence) and tested whether it carries or gates the damage-relevant quantity. It does not: in a regression with geometry, co-firing and its interaction with geometry are both null (interaction permutation p = 0.67), and within every geometry stratum co-firing is inert. In case the threshold was the problem, we re-ran the test with continuous co-activation statistics. Null again. Whatever links crowded pairs, it is not that they fire at the same time.

The turn: forgetting is a second-order crime

Here is the reframe that cracks the case. At the end of phase A, training has converged: the gradient of the math loss is nearly flat. To first order, nothing can hurt the model. The damage from phase B’s update arrives through the second-order term,

where is the curvature (Hessian) of the math loss. Curvature is a matrix over pairs of parameters, and that plural is the whole point. Its diagonal blocks are self-curvature: how sensitive the loss is to moving one neuron alone, which is roughly what Fisher tracks. Its off-diagonal blocks are coupling: whether moving neuron changes the price of having moved neuron . If a pair couples positively, updating both together hurts more than the sum of updating each alone. Forgetting, in this frame, is not a list of individually fragile neurons. It has cross terms.

Work out what that coupling looks like for two MLP neurons (using the Gauss-Newton form of the curvature, the positive-semidefinite part that comes from the network’s output geometry), and the dominant term factorizes:

The first factor is the overlap of the two neurons’ existing write vectors, which is exactly what crowding measures. The second is the co-response of their activation changes under the update: not whether they fire together, but whether the update moves their firing together, weighted by how much the output cares. Two clocks, one geometric and one usage-driven, multiplied.

If this account is right, crowding works because it is the zero-data shadow of the geometry clock: it reads one factor of the off-diagonal curvature straight from the checkpoint. And the account makes predictions sharp enough to falsify. Crowded pairs should carry outsized coupling at matched everything else. The coupling should flow through the existing write vectors, not through co-activation. Measuring the coupling directly should beat every proxy of it, including Fisher. And the advantage should survive matching on every diagonal quantity. All four were registered before the tests ran.

Exhibit A: crowded pairs carry the coupling

Measuring off-diagonal curvature naively is hopeless (the matrix is billions-by-billions), so the estimator probes it: perturb one neuron’s parameters along its actual phase-B update direction, and read out how the curvature responds at every other neuron in the layer. Three forward passes and one backward pass per probe, validated against an exact reference implementation on a small oracle model, plus a seven-gate sanity battery. We ran 1,200 probes.

Then the matched contrast: take crowded pairs and orthogonal pairs that agree on firing rate, gradient magnitude, weight norm, update norm, and co-firing (202 matched contrasts), and compare their coupling.

The median crowded pair carries 6.12x the coupling of its matched orthogonal counterpart, and every one of the 16 measured layers shows the effect. The obvious deflationary reading, that crowded neurons just have more curvature of every kind, fails: normalize each pair’s coupling by its members’ self-curvature and the contrast is still 5.06x. The excess is genuinely between the neurons.

Two more registered checks. Crowding correlates with a neuron’s total within-layer coupling load at +0.32, passing the registered bar; the same correlation against a global load that includes cross-layer terms drops to +0.18, a registered miss we report as found. Crowding is a within-layer statistic and sees within-layer load. And the coupling is polarity-blind: pairs whose write vectors point in opposite directions couple exactly like aligned ones (both predominantly positive, 69% and 71%). That refuted our own registered sign prediction, and it vindicates the convention of measuring crowding with absolute cosines: opponent neurons interfere like duplicates.

Exhibit B: the anatomy matches the algebra

The two-clock factorization makes a structural claim: the coupling should be carried by read-side activation changes emitted along the existing write vectors (the terms), not by update-write overlap gated by co-activation. That is checkable, because each pair’s coupling can be decomposed into blocks by which interface sends and which receives.

On the 50 registered validation pairs, the read-by-read block flowing through base write vectors carries a median 0.812 of each pair’s coupling mass (95% CI [0.737, 0.855]), and no single pair drives the result. The one block that raw co-activation actually gates is the smallest of all nine, at 0.031. This is why Suspect 4 walked free: the tests were correct, and the channel they tested barely exists.

Better, the decomposition contains its own discriminating prediction. The base-write terms scale with , so read-side dominance should be a property of crowded pairs specifically. It is: median share 0.872 for crowded pairs, dissolving to 0.437 for orthogonal ones. The algebra predicted a fingerprint and the data shows that fingerprint.

Exhibit C: measure the coupling and it beats everything

If off-diagonal curvature is the mechanism, then the mechanism quantity should be a better protection signal than any of its proxies. So we built one: K2, each neuron’s off-diagonal curvature row load along the actual update directions, estimated with randomized sketches. K2 needs one sacrificial unprotected run of phase B to know the update directions; it then selects the top 20% per layer, and we re-run protection with that mask on fresh data seeds. Evaluation is leakage-clean: K2 was built on one half of the held-out math set, outcomes are scored on the other half, and the mask transfers across update seeds.

K2 beats crowding by -0.40 NLL and beats Fisher by -0.183 on every one of six seeds (exact one-sided p = 0.0156). The registered “strong result” band was merely matching Fisher; the data went past it. This is the strongest retention selector observed anywhere in this program, and it overlaps the crowding mask at only 0.32, so it is not crowding in disguise.

One fairness note, registered up front: K2 uses information Fisher does not have (that one unprotected B-run). But Suspect 3 already showed that update information alone is the worst selector in the program. Raw movement fails; movement weighted by curvature wins. The curvature weighting, not the data access, is what converts a B-run into the best mask.

Exhibit D: the value is off-diagonal, specifically

The remaining skeptical reading: maybe K2-high neurons are just fragile in every measurable way, and “off-diagonal” is along for the ride. Our first attempt at this control was itself caught by an adversarial audit: the matched sets turned out unbalanced on exact self-curvature and first-order alignment, so we relabeled that result and registered a stricter design with a stop-if-confounded gate.

Version 2 uses a near-exact diagonal estimator (agreement with exact self-curvature: rho 0.993) and builds three sets of 28,443 neurons matched on crowding, self-curvature, first-order alignment, Fisher, and movement, with a post-construction audit on exact quantities. The audit passed with the balance tilting slightly against the hypothesis (the high set’s self-curvature is 0.84x the low set’s). Then the causal test: restore each set to its pre-B values inside the damaged model and measure how much math returns.

High-coupling neurons undo +0.768 of the damage. Their matched-on-everything-else low-coupling twins undo +0.184, less than a random set (+0.323). With every diagonal explanation held fixed and audited, the off-diagonal information alone separates recovery by +0.58. This is the finding the whole case rests on, and it survived the audit that killed its first version.

Two smaller pieces round out the picture. The neuron behaves as the natural unit: freezing its three weight matrices contributes protection almost exactly additively (shares 0.39 + 0.13 + 0.44, summing to 0.96), with no single interface acting as the mechanism’s home. And nothing here says forgetting is only off-diagonal; self-curvature is real, which is why Fisher is a strong selector. The claim is that the cross terms carry protection-relevant information that no diagonal signal contains, and that crowding is a free estimate of where those cross terms concentrate.

The confession doubles as a prediction

A good mechanism should tell you where it stops working. This one does.

Curvature in the optimizer’s coordinates is not a property of the abstract function the network computes. You can rescale a SwiGLU neuron (multiply its input row by a power of two, divide its output column by the same factor) and the network’s outputs do not change at all; with power-of-two factors the logits are bitwise identical. But the curvature account says the damage process should change, because the rescaling scales one damage pathway’s curvature by and the other’s by .

So we ran the final registered experiment as a self-test: if protection value is invariant under these function-preserving rescalings, the curvature account is wrong. It is not invariant. Rescaling the protected neurons by 4 or by 1/4 raises protection value by +163% and +99%; a global random rescale swings it by -126% (mostly by stalling learning: Adam’s per-parameter state is calibrated to the old scales). All three land far outside the registered +-25% invariance band, in the direction the algebra points.

The scope condition, stated plainly: crowding’s protection value is a joint property of the computation’s geometry and the optimizer’s coordinates, not of the function alone. Any deployment claim must hold the parameterization fixed. Real fine-tuning does exactly that, so the ledger applies where it is used; but nobody should carry these numbers across a reparameterization.

The ledger, upgraded

Where does this leave the practical picture from Paper 1?

The two-signal ledger (weights as a structural prior, a small activation capture as calibration) turns out to have been approximating the two factors of off-diagonal curvature all along: the geometry clock and the usage clock. That reframe explains the program’s stubborn observations in one stroke. Why does zero-data crowding protect? It reads the geometry factor. Why does it protect while disagreeing with Fisher? Fisher is diagonal; the information is in the cross terms. Why did every co-firing test come up null? The co-activation-gated pathway carries 3% of the coupling. And why is protection value fragile under reparameterization? Because curvature is.

It also hands the next project its target. K2, the best selector we have ever measured, costs a full sacrificial training run. The registered follow-up question is a ladder: how close can you get to K2’s value with zero lookahead (a block-coherence ledger built from the weight file), or with 1, 4, or 16 early microbatches of the new task? The decomposition already tells us where to look: read-side co-response against the write-vector Gram matrix, not raw firing statistics. That allocator, a capacity ledger that knows what the next task is about to break before it breaks it, is Paper 3.

The earlier post ended by admitting that why crowding helps at all remained open. It is no longer open. Crowding helps because it is the visible half of the quantity that forgetting actually uses: the off-diagonal curvature that couples a neuron’s fate to its neighbors’. The weights were never a complete ledger. But on this question they were telling the truth, and now we know which truth.

The setup, briefly

The cast is four open checkpoints, small enough to dissect on one machine: Qwen2.5-1.5B, Gemma-2-2B, Pythia-1.4B, and TinyLlama-1.1B. Everything here happens in their MLP blocks. Each MLP neuron reads from the residual stream (the vector that flows through the network, accumulating each layer’s contributions), decides how strongly to fire, and writes its contribution back along its write vector: one column of the down-projection matrix, with normalization gain folded in where needed. The write vector is the direction the neuron adds to the stream.

The zero-data instrument is crowding. For each neuron, take its write vector and ask how close it sits to its same-layer neighbours: the largest absolute cosine similarity against any other write vector in the layer. A crowded neuron shares its output direction with a neighbour; an isolated neuron writes somewhere no one else does. Nothing but the checkpoint file enters this number. No examples, no labels, no activations, no gradients.

The cheap usage instrument is the footprint. Run the model briefly over a small corpus spanning five kinds of text, which I will call regimes: math answers, math prose, code, code prose, and ordinary prose. For each neuron, record how often its activation lands in its layer’s top 1% (the 99th percentile, hence q99). Those five firing rates are the neuron’s footprint. The full capture is roughly 375,000 tokens and takes about four minutes per model on the test machine.

The expensive reference is diagonal Fisher information: backward passes on the retained task estimate which parameters the loss is locally sensitive to. It is not ground truth, but it is the data-rich rung that the cheap instruments are trying to approximate.

The central mistake to avoid is collapsing these into one concept called “importance.” They measure different things. The experiments ask when those things happen to agree.

First, try to break the instrument

Before trusting footprints to protect or quantize anything, I tried to break them. Split a regime in half and compare the halves: the centered footprint cosine stays near 0.99 at every layer, essentially the resampling noise floor. Compare different regimes instead and the worst layer in each model still clears a three-sigma separation criterion by 31-144x. Shuffle the labels and the margin collapses to zero. The instrument held.

A nearest-centroid classifier identifies the regime from a single sequence’s footprint at 98-100%. That accuracy by itself is not impressive; classifiers built on raw token counts also saturate on these datasets. The more useful test is placement. Plain-English math questions and code descriptions sit closer to their computational siblings in footprint space than token statistics predict, in all four checkpoints, with net swings of +0.16 to +0.75 for math prose and +0.29 to +0.66 for code prose. The footprint knows something the word counts do not.

It is not a pure readout of “computation,” though. The footprint still knows about dataset and register, and footprints taken while the model generates text drift well beyond the reading noise floor, so reading and generation need separate references. The claim is narrower: a tiny forward capture produces stable, useful telemetry that raw token counts do not reproduce.

Trial 1: which neurons work together?

The weight map finds a striking worst case everywhere. Across checkpoints, the 99th percentile of pairwise write overlap is only 0.06-0.17: the bulk of the dictionary looks roomy. Yet the single largest coherence reaches 0.78-1.00. Near-duplicate write directions exist at every depth in every model.

Those overlaps are operational, not decorative. Roughly 500 pairs per layer were stratified by write cosine and checked against real joint activity. Near-orthogonal pairs fire together at or below what independence predicts, with median lift 0.71-1.06. Crowded pairs fire together at 1.6-3.5x independence, and the top decile reaches 7-59x. Training has not quietly arranged overlapping writes so that they take turns.

So the geometry is real. It is just not sufficient. Add reader overlap and base firing rates to build the full static model, and it explains only 3.5-17% of co-use rank variance. Gemma-2 is the most weight-legible case: its reader-set overlap reaches Spearman and largely absorbs the incremental geometry term, yet even there the ceiling is 17%. On Qwen, Pythia, and TinyLlama, reader overlap adds little.

One family difference deserves a note. About half the crowded pairs in gated MLPs are anti-parallel opponents ( and ), versus 18% in plain-GELU Pythia. Simultaneous overlap is real; whether it is redundancy, cancellation, or harmful interference cannot be read from an absolute cosine alone.

Trial 2: which neurons are risky to update?

Correlation is cheap, so the centerpiece is causal. Every arm starts from the same checkpoint that has just learned math, then trains its MLP weights on code. Before the code training begins, 20% of neurons per layer are frozen according to one rule: none, random, weight crowding, math footprint, crowding times footprint, or Fisher. The zero- and low-data masks are captured at the base checkpoint and stay fixed. Only Fisher is computed after math.

Both outcomes are reported, because either one alone can lie. Less math degradation means better retention; more held-out code improvement (code the model never trained on) means better acquisition. Calling a mask good because it forgets less, while hiding that it also learns less, would bury the central tradeoff.

Does scale rescue the shortcut?

Perhaps geometry is merely noisy at 1-2B parameters and becomes Fisher-like in larger models. Across the tested Qwen2.5 series, it does not. The median rank agreement between crowding and Fisher is +0.12, +0.10, +0.17, +0.13 at 1.5B, 3B, 7B, and 14B. The top-20% salient-set overlap stays at 0.17-0.20, only modestly above the 0.11 chance floor.

Under the fixed stress protocol, weight-only recovery falls from 90% at Qwen 1.5B to about 20% at 3B, and at 7B the baseline and Fisher arms barely separate, so the recovery ratio is not identifiable there. This is not a scaling law; the effective update and task competence are not matched across sizes. But it is clear evidence against the hopeful claim that the proxy gap closes over the tested range.

Trial 3: which neurons mix tasks?

This trial ends at a wall, and the wall is the program’s firmest fact. It gets only a summary here because Post 3 reports it in full. Neuron crowding does not provide a practically useful rank of task-regime mixing on either MLP interface in any checkpoint: all eight primary measurements sit inside the pre-registered null band.

The pipeline itself is not to blame. The same code recovers a planted law at +0.52, and its entropy proxy tracks known mixing in a trained toy model at +0.83. The null is about the world, not the instrument. Weight geometry can predict an intervention without yielding a semantic atlas.

Trial 4: which neurons deserve scarce precision?

Quantization supplies a different actuator and a useful consistency check. Keep every weight at four bits, then give eight bits to the top 1% of neurons under each signal. Random protection recovers nothing. Usage and gradient signals recover 47-85% of the four-bit damage. And because quantization noise is not conditioned on any task, the footprint gets promoted here: it recovers about 85% of the gap on TinyLlama and can match or beat the gradient-guided rung on these checkpoints.

With sequential GPTQ (a standard error-correcting quantizer), the ranking is nearly saturated at four bits; every rung lands within 0.16 perplexity. Differences emerge at three bits and below, where activation energy recovers 55%, 12%, and 19% of the three-bit damage at Qwen2.5 1.5B, 3B, and 7B. None of this is a new quantizer. It is a practical hint: when the precision budget is genuinely tight, a short activation capture carries information that static weight geometry misses.

The ledger after contact with data

The zero-data dream does not survive intact. It does not collapse either.

Weight geometry comes out of the trials as a prior, not a verdict. It locates crowded writes, predicts elevated co-use, and produces a causally useful protection mask in one clean checkpoint. It does not replace observed usage or loss sensitivity, and it does not read task semantics at all. The footprint, meanwhile, turns out to be an unusually good middle rung: stable, distribution-conditioned, cheap enough to refresh, and often competitive with gradient-based signals whenever the intervention depends on usage magnitude.

The Qwen repair is the part I would most want a practitioner to carry away. Pair every retention number with an acquisition number, because protection that merely stops the optimizer from learning is not protection. The repair changed the story from “geometry protects” to “geometry moves the operating point along a frontier.” That is still useful. It is a different claim.

And heterogeneity is the rule, not the exception. Gemma-2 is unusually weight-legible, TinyLlama gives the clean Pareto result, Qwen gives a frontier. Architecture, scale, tokenizer, and training recipe are all confounded across these checkpoints, so none of this is a family law.

The practical design I carry forward is a two-signal ledger: use the weights as the always-available structural prior, calibrate them with a small footprint captured in the execution mode you care about, and escalate to gradients only when the decision is important enough to pay for them.

One question survives every trial untouched. Why does crowding help at all? A signal that has never seen a token somehow points at the neurons the next task will damage, and nothing in Paper 1 explains that. The trials narrow the possibilities; they do not settle the mechanism. That question belongs to Paper 2, and to a later post.

The claim, made measurable without a dictionary

In superposition a network represents more features than it has dimensions by letting feature directions overlap. The intuitive reading at the neuron level: a neuron in a geometrically crowded neighbourhood is being asked to serve many masters, so it should fire across many unrelated contexts – it should be polysemantic.

Both sides become concrete without ever touching an SAE. Mixing is the Shannon entropy of a neuron’s firing distribution over five task regimes (math, math-prose, code, code-prose, prose), with per-class rates normalised by token counts first:

Crowding is the neighbourhood density of a neuron’s weight vector – the number of same-layer neurons within of it (with and the mean of the top-10 neighbours as secondaries).

Every correlation below is a partial Spearman , controlling for log firing rate. A busy neuron is mechanically both more mixed and more likely to look crowded; rank-residualising on rate before correlating removes that confound. Neurons with fewer than 50 firing events are excluded. Seeds fixed; four families – Qwen2.5-1.5B, Gemma-2-2B, Pythia-1.4B, TinyLlama-1.1B – chosen so a Gemma-only quirk cannot masquerade as a law.

Everything hugs zero – on both faces of the neuron

The write side was the natural place to look: a neuron’s output direction is what collides in the residual stream. But recruitment is decided on the read side – whether a neuron fires is a property of its input weights, not its output. So the read interface was the last place a neuron-level packing law could hide. It does not hide there either. And the medians are not papering over a wide spread:

A neuron is not “in a crowded place.” It is crowded per interface.

The read-side test was not a redundant re-run of the write-side test, and the reason is a finding in itself. If a neuron simply occupied a crowded or an uncrowded region of the model, its read-crowding and write-crowding would move together – the correlation would be near one. It is 0.21 to 0.47.

Recruitment geometry and expression geometry are substantially independent properties of the same unit. The folk picture assumes one geometric fact per neuron; there are at least two, they disagree, and neither predicts what the neuron actually does across regimes. That is the quiet mechanism behind the whole null.

The effects that clear the noise point in opposite directions

A weak-but-real law would at least be consistent: crowded neurons a little more mixed in every family. Instead the largest excursions cancel. Across both interfaces the two measures that graze the band – Gemma-2’s write-side local-crowding at and Pythia’s read-side top-10 at – have opposite signs, one family each, two hundredths past a threshold chosen as the noise floor. The stratum medians tell the same story in a picture.

Gemma-2 showing the lone sub-threshold positive is exactly the pattern to expect: across this whole line of work Gemma-2 is repeatedly the most weight-legible model, so a whiff of structure-to-function signal there, and nowhere else, reads as a property of Gemma rather than of transformers. It was pre-registered as not counting.

A null is only as good as the instrument behind it

The obvious attack on any negative result: maybe the pipeline could not detect the effect even if it existed – maybe five coarse regimes cannot resolve real polysemanticity. That objection is answerable in-house, and cheaply, with two toy systems.

The real-model numbers sit an order of magnitude below what the same instrument returns when the law is real.

quantity	value	bar	reading
real census, write (best family)	+0.05	–	the null
planted law, density	+0.52	+0.15	detected, 3–4 over floor
planted law, top-10	+0.63	agrees in sign	detected
planted law, labels shuffled (q95)	0.094	< 0.10	no false positive
census entropy vs ground-truth mixing	+0.83	–	proxy is faithful

What this does and does not say

This is a claim about neurons, not features. Superposition theory is fundamentally about features as linear combinations of neurons; an SAE feature can be crowded and polysemantic in ways no single neuron reveals. Nothing here refutes feature-level superposition. What it bounds is the neuron-granularity shortcut – the hope that you could read polysemanticity off raw weight geometry without a dictionary. You cannot, on either interface, in any of four families.

It is also a claim about task-regime mixing (five coarse classes), not fine-grained semantic mixing, and about MLP geometry, not attention. Within those bounds it is as airtight as I could make it: pre-registered thresholds, both faces of the neuron, four architectures, a positive control at 3–4 the detection floor, and a demonstrated-faithful mixing proxy.

Which neurons mix regimes is set by the statistics of the data a model was trained on, not by the geometric necessity of packing. The ledger of a model’s capacity has to be read from usage; it is not sitting latent in the Gram matrix of its weights.

This is the negative half of a longer weights-first program. Where the earlier notes mapped what weight geometry does know, this one marks a firm boundary on what it does not.

The three bugs

All three are convention mismatches with Hugging Face’s Gemma-2 implementation. None changed the shape of the method; all changed the numbers.

Retraction: there is no anti-predictive regime

This is the big one. Here is the per-layer mean Spearman between weight-space predicted attention and real attention (all 8 query heads, 20 prompts), pre-fix versus the corrected pipeline in two bases:

The grand means tell the story: pre-fix +0.011 (statistical noise), fixed SAE basis +0.229, fixed token basis +0.260. Every layer the first post labelled anti-predictive – L9, L10, L12, L13, L16, L24 – is solidly positive after the fixes, with 100% sign stability across heads. Best single heads reach overall (L11H3) and on local distances (L6H0).

The Layer-6 spike survives, its uniqueness does not

The first post made a lot of a Layer-6 selectivity spike that appeared to tower roughly over a uniform baseline – two orders of magnitude above everything else. The spike is real, but the contrast was inflated by the gamma bug. Corrected selectivity (, top-1 softmax mass over uniform): L6 still peaks globally at 1451, but L8 (1176), L22 (1113), L12 (866), and L14 (912) sit in the same band, all 3–25 above a random-weights baseline. The corrected story is a band of selective layers from L6 to L14 plus a late peak at L22 – not a lone spike.

And the identity/copy-routing peak – the diagonal mass of the routing matrix – isn’t even at L6. It’s at L10:

New: the L13 anomaly is an SAE checkpoint issue

L13 is the one layer where the corrected SAE-basis validation still fails (mean ). That looked alarming until I probed the same layer three ways:

probe	mean Spearman (8 heads)
layer-12 SAE (standard offset, conceptually correct tap)	+0.022
layer-13 SAE (offset 0, conceptually misaligned tap)	+0.176
token-embedding basis (no SAE)	+0.377

The conceptually correct SAE is the worst of the three, and the dictionary-free basis is the best. So attention at L13 is well-predicted by weight space – the gemma-scope-2b-pt-res layer-12 canonical 16k checkpoint is just a poor dictionary for this purpose. The practical lesson: basis disagreement is a diagnostic. When three reasonable bases disagree this sharply on one layer, suspect the dictionary, not the model. Any SAE analysis leaning on that particular checkpoint deserves a second look.

New: a dictionary-free basis is competitive

That L13 result generalizes. I re-ran the whole validation using a mean-centered token-embedding matrix as the probe basis – no SAE at all – and it validates slightly better overall than the Gemma Scope SAEs:

Grand mean: token 0.260 vs SAE 0.229, with the token basis winning 15/26 layers. The SAE wins at L0, the L5–L6 selectivity band, and L21–L24. The honest reading is uncomfortable for the original framing: for routing analysis, the SAE dictionary is not load-bearing. A free, interpretable-enough basis does as well or better through most of the stack. The SAE earns its keep where you want named features for the OV/writing story, not for predicting where attention goes.

The L23 causal result, re-explained

The first post’s one causal experiment – ablating individual Layer-23 heads on a factual-retrieval task – still replicates: removing L23H0 improves accuracy by +6.2 points (0.595 to 0.657, 800 prompts; L23H2 gives +3.2). That finding stands.

What does not stand is the explanation. The first post tied the improvement to the late-layer anti-predictive regime – “the map flagged a high-leverage layer but couldn’t read its sign.” Under the corrected map, L23H0 is well-predicted (overall , local ), so the map can see it. The corrected read of what L23H0 is: a REPULSION routing archetype (strong negative affinities, min ) wearing a high copy-dominance OV circuit (copy_dominance ). A head that copies content under repulsion-shaped routing plausibly drags non-answer context into the final residual on cluttered prompts – consistent with ablation helping on a filler-heavy retrieval task. Prediction quality and causal helpfulness are orthogonal; the first post conflated them. Independent corroboration that L23 is special: instruction tuning moves L23H0/H1 more than almost any other heads in the model (base vs -it weight diff).

What I’d take away

• Pin your conventions with parity tests. All three bugs were silent agreements-by-accident with HF that only a direct numeric cross-check exposes. The two parity tests now make these regressions loud.
• A negative result on a basis is a result about the basis. The L13 “failure” was the most informative single layer once I stopped assuming the SAE was ground truth.
• Don’t over-credit the dictionary. For routing geometry, a dictionary-free basis matched or beat Gemma Scope across most of the stack. SAEs are still the right tool for naming what a head writes – just not for deciding whether the weight map can predict attention at all.

Appendix: full table

The complete per-layer Spearman table (pre-fix, fixed SAE, fixed token) for all 26 layers, plus the corrected selectivity, RoPE-stability AUC, and dual-basis per-layer winners, is in errata_full_depth.md with the accompanying run JSON. The figures above are generated from analysis_20260610_233407.json and the 20-prompt validation outputs.

Setup

We use the base model Gemma-2-2B (26 layers).

It uses Grouped Query Attention (GQA): 8 query heads and 4 KV heads, so each KV head is shared by 2 query heads. This means each query head has its own output projection () but reads from a shared value signal (). We analyze query heads (not KV heads), since each has its own routing and writing behavior.

One implication is worth keeping in mind later: sister query heads share the same value source but can still look very different in the map. When that happens, the specialization is unlikely to come from alone; it has to be coming from routing (), the per-head output projection (), or both.

1. Gemma-2 scales attention logits by . With query_pre_attn_scalar = 256, we have .

2. Softcap value: .

3. Gemma-2 uses sliding window attention in alternating layers starting at index 0: even layers are sliding-window with size 4096, odd layers are global (full causal attention up to the model’s max context length, 8192 tokens).

To talk about “features” rather than neurons, I use Google’s Gemma Scope Sparse Autoencoders (SAEs). SAEs decompose model activations into sparse, interpretable directions. Each learned “feature” is a direction in residual-stream space that tends to activate for a coherent pattern. This gives us a model-aligned coordinate system for analyzing attention: instead of asking “which neuron fires?”, we can ask “which interpretable feature does this head route or write?” These SAEs are trained on base Gemma-2 models.

Analysis choice: I use the SAEs trained at the “Residual SAE” tap point. Attention in block L consumes the residual stream produced by block L-1. Thus, in my runs I use an SAE layer offset (-1) so the SAE basis matches what the attention block actually sees. Each SAE feature has a decoder direction (a vector in residual space). I’ll treat those decoder directions as a convenient, model-aligned basis for analysis. Please note that SAE decoder directions are not orthogonal and activations are nonnegative.

Using the base model avoids instruction-tuning artifacts and keeps the focus on the raw attention mechanism.

Weight-Space Object #1: Content-Only Routing in Feature Space

For a single attention head, I want a matrix that answers:

“If the query looks like feature i, which key-feature j does this head prefer?”

Let:

• be the SAE decoder matrix for a subset of features (shape ), one row per feature direction. (Convention: I treat decoder directions as row vectors; if you store decoders as columns, transpose accordingly.)

• , be the head’s query/key projection matrices

Project decoder directions into Q-space and K-space:

Define the content-only affinity matrix (what attention logits would look like if RoPE and masking weren’t present):

Where (Gemma-2’s attention logit scaling).

Gemma-2 also applies a tanh soft-cap to attention logits; when matching the model’s behavior, I apply the same cap:

This matrix is the backbone of our routing analysis. From we can define simple, interpretable routing metrics:

• Selectivity (peakiness): per row i, softmax over keys and take max probability; average over i – “top-1 softmax mass”

• Identity sensitivity: how much softmax probability lands on the diagonal entry (i attends to i) – “diagonal softmax mass”

• Max-gap: difference between best key score and runner-up (in logits), averaged over rows

• Avoidance: Negative entries in indicate low dot-product affinity. I treat extreme negative tails as a signature of selective exclusion, but it’s not the same thing as suppression unless the path also writes negative feature directions.

Important: rows/cols of are SAE features, not tokens. When I compute “top-1 softmax mass”, the softmax is over key features j, not over token positions.

A few examples:

Sanity Check: Does B Predict Real Attention Routing?

Although the map in this post is computed purely from weights plus a fixed SAE basis, I ran an activation-grounding sanity check to confirm that the main routing object (the feature affinity matrix ) predicts runtime attention patterns.

Across 200 heads (25 layers x 8 heads), using all 16,384 SAE features and RoPE-aware affinity matrices , the activation-grounding run covers about 4.45M token pairs per head. The headline result is mixed in exactly the way that turned out to matter for interpretation: the map is usefully predictive in early and mid layers, but degrades sharply in late layers. The layerwise spread and threshold counts are shown in Figure 4 and Figure 5.

For ~25 diverse prompts, I captured the pre-attention residual stream at the target layer, encoded it into the same SAE feature subset used in the weight-space map, and predicted attention preferences using the bilinear form . For the runtime target, I used log attention weights. I compute correlations per query row and then average (to avoid cross-row normalization artifacts), and I also break correlations out by distance bins; log-softmax equals logits up to a per-row constant, so rank correlation is meaningful even though each row has its own normalization.

To account for RoPE, I also computed RoPE-aware affinity matrices per distance bin by rotating K ( set to the bin center) and applying Gemma’s tanh softcap, then compared predicted vs runtime attention within each distance bin.

Summary statistics for the same run:

• ~4.5M token pairs per head
• Mean overall Spearman = 0.07 (std=0.21)
• Mean local (0–4 tokens) Spearman = 0.15 (max=0.82)
• 144 heads (72%) show (‘structured’ routing)
• Best local correlation: L5H4 with = 0.82
• Best overall correlation: L6H4 with = 0.57

The important caveat is that correlation is not uniform across depth. In early layers (L1–L5), 35 of 40 heads show positive overall Spearman (mean +0.16), and mid layers (L6–L12) are even stronger (47/56 positive, mean +0.22). But in late layers (L19–L25), 50 of 56 heads show negative overall Spearman (mean -0.14), meaning the weight-space matrix predicts the opposite of the observed attention pattern. This degradation likely reflects nonlinear interactions or representational drift that are not captured by the linear SAE decomposition. My best guess is that the fixed residual-stream feature basis becomes a poorer proxy for the computation actually being deployed near the top of the network, where the model is closer to logits and late residual mixing is more consequential. That is an interpretation, not a demonstrated mechanism. The weight-space map remains a useful guide in early/mid layers but should be treated as a capacity map rather than a behavior map past roughly Layer 17.

The distribution is heavy-tailed: many heads are near-zero (diffuse), while a smaller set shows strong, structured routing (e.g., best local = 0.82). The top 10 heads by overall Spearman (L6H4, L5H4, L3H2, L6H3, L8H1, L8H0, L6H2, L8H6, L12H3, L12H2; all > 0.45) cluster in layers 3–12.

Please note that this doesn’t prove any head is causally responsible for a behavior; it just checks that the measurement objects used in the map are anchored to runtime attention.

ImportantUpdate (March 2026): Long-Distance Routing

Long-range structure is weaker than local routing, but it is not absent. At 128–256 tokens, 114/200 heads still clear , and 108/200 do so at 256+; the more important failure mode is the late-layer sign reversal in overall routing correlations rather than a complete disappearance of long-range structure.

Baselines: What Counts as “Real Structure”?

Softmax selectivity is extremely sensitive to logit scale, so we need baselines that match the right thing. I use three baselines, each designed to kill a different kind of structure:

1. Random decoder directions (random D): Replace decoder directions with random unit vectors (matched norms), keeping / fixed

2. Random weights baseline: Replace and with random matrices (same shape), then rescale to match before computing softmax-derived metrics. Specifically, we rescale , where .

3. Permuted-K baseline: Shuffle key-feature identities by permuting rows of (equivalently, permuting columns of ), so diagonal mass becomes ‘chance self-match’.

This lets me say things like:

“Selectivity is X times the random-weights baseline.”

“Diagonal mass is X times the permuted-K baseline.”

Why both matter:

• top-1 mass is permutation-invariant (it only cares that you pick something)

• diagonal mass is identity-sensitive (it cares that you pick self)

Weight-Space Object #2: What a Head Writes (OV) in Feature Space

Routing is only half the story. Even if a head can attend cleanly, it only matters if its output projection writes a direction that’s salient in the SAE feature basis into the residual stream.

For a query head, its write is determined by:

• for the KV group (shared across 2 query heads in Gemma-2 GQA implementation)

• for the specific query head

Gemma’s effective OV matrix is implemented as (row-vector convention; is , is ):

For each SAE feature direction (row of ), define the write vector:

Project write vectors back onto decoder directions to get a “write-to-feature” matrix:

Note: W2F uses cosine similarity, so it measures directional alignment and discards magnitude; later I’ll also report write norms / activation-grounded deltas so ‘aligned-but-tiny’ writes don’t get over-interpreted.

This helps us define archetypes with concrete scoring.

With W2F as the cosine similarity matrix between each feature’s write vector and all decoder directions, we define:

• COPY score = mean(diag(W2F)): how much feature i writes back to itself

• TRANSFORM score = max off-diagonal of W2F: strength of the strongest i ji mapping

• BROADCAST score = max column sum of W2F: how many features write to the same target

• SUPPRESS score = |min off-diagonal of W2F|: strength of the most negative write

Copy dominance remains low on average (mean 0.186), but the more informative pattern is the upper tail: the strongest copy head is the early outlier L2H2 (0.897), while a secondary cluster of near-copy heads appears in L23–L25. The distribution by layer is shown in Figure 7.

A head is classified by whichever score crosses its threshold first (COPY > 0.3, TRANSFORM > 0.5, BROADCAST > 3x copy, SUPPRESS > 0.5), defaulting to DIFFUSE (no score crosses any threshold). I assign a single label using a precedence rule (COPY TRANSFORM BROADCAST SUPPRESS DIFFUSE). Heads can score highly on multiple axes; the label is just meant to be a coarse summary. These thresholds are heuristic cutoffs chosen to separate tails in this run, not a claim that the taxonomy has canonical boundaries.

In this taxonomy, SUPPRESS means the head routes positively to some key feature j, and then the OV write is negatively aligned with a target feature k (it is not about negative B/avoidance). Also, please note that COPY can be negative if the head writes anti-aligned to the feature’s decoder direction.

A few examples:

Weight-Space Object #3: Composing “Feature Programs” (i j k)

Once we have routing () and writing (W2F), we can compose them into simple “programs”:

query feature i attend key feature j write feature k

Programs are extracted from the top tail of routing scores and write scores (quantile thresholds), so the taxonomy describes high-confidence motifs rather than all mass. We also track when a self-write is a fallback rather than a measured OV target.

Each triplet is scored by combining route strength and write strength. Then we can classify it into a small motif family:

Type	Flow
REINFORCE	i i i
SHIFT	i i k (k i)
CROSS_COPY	i j j (i j)
RELAY	i j i (i j)
TRANSFORM	i j k (all different)
SUPPRESS	i j -k

This is the point where the analysis becomes human-legible: you can say “this head’s weight structure is consistent with CROSS_COPY” or “this layer is SUPPRESS-heavy in the weight-space taxonomy,” without running a prompt.

In my run: routing tail top 10%; write tail top 20%.

Position Robustness: RoPE Stability

Gemma-2 uses RoPE, which rotates Q/K vectors as a function of relative position. In weight space you can simulate this by rotating keys (or queries) and recomputing affinity matrices.

Let be affinity at relative offset , and at offset . Define a stability score (e.g., cosine similarity of flattened matrices):

Then compress the curve across into a single number (AUC-style over log-distance). I call this RoPE stability: a summary of how similar the affinity matrix stays as we apply relative-position rotations. Also, we apply the tanh softcap before comparing matrices.

• When this metric is High routing seems to be mostly content-controlled and stays similar across positions. It could also mean that this head doesn’t use positional information much because it’s doing something else entirely.

• And when it is Low routing depends strongly on relative position (more “twitchy”).

This is not “style steerability,” but it is a useful proxy for “position-invariant content routing.”

One caveat: cosine similarity can look high if is very low-magnitude / diffuse; I interpret stability alongside routing strength/selectivity.

Results: What Changes with Depth?

Once we compute these objects across layers and heads, we can summarize how the model’s attention machinery evolves with depth.

1. Local (Sliding-Window) vs Global Layers

Gemma-2 alternates local and global attention by layer, giving a built-in contrast:

Sliding-window layers can’t directly attend to tokens outside the window, but they can still transform a residual stream that already contains long-range information injected by earlier global layers.

Global layers are the only layers that can implement direct long-range attention pointers.

2. Selectivity and Identity-Sensitivity Are Not Monotone

Two different things can happen as depth grows:

1. Peakiness (top-1 softmax mass) can increase because the model learns sharper “addressing”

2. Identity sensitivity (diagonal softmax mass) can increase if heads learn an ii “feature self-match” pattern

In my current run, there’s a very strong spike in these metrics around a mid layer (one layer stands out dramatically on both selectivity and diagonal mass). That suggests a “band” of unusually crisp routing behavior rather than a smooth trend.

NoteWhat Surprised Me

The Layer 6 spike was the single most dramatic finding – SelU of 105 means the routing structure is two orders of magnitude above uniform baseline. I expected a smooth gradient, not a spike. This showed up consistently across different feature subsets and seeds. Layer 6 is a sliding-window layer (even index) in the range where induction-like heads are typically found, and four of its heads (L6H0, L6H2, L6H3, L6H4) are among the top-10 validation-confirmed heads. The spike-rather-than-gradient pattern suggests a sharp regime change in routing behavior at this depth.

3. RoPE Stability Tends to Rise with Depth (with Bands)

RoPE stability tends to increase with depth, but it’s not smooth. There are high-stability bands (mid layers and a late cluster near the top of the network).

4. Redundancy vs Uniqueness (Where Structure Concentrates)

I track how redundant heads are within a layer (do they “do the same thing” in program space?). Redundancy = Jaccard similarity of top-20 routing pairs between heads – that is, where and are each head’s set of top-20 (query key) feature pairs. Two heads are considered redundant if they route the same feature pairs highly:

Low redundancy: a few specialized heads carry most structure – could be potential larger single-head “levers”

High redundancy: many heads share similar top programs – could mean that the layer is robust to removing any one head

To avoid noise, I primarily interpret redundancy within the subset of heads that show structured routing (e.g., above a small selectivity or correlation threshold).

Working Interpretation: Three Depth Regimes

Taken together, the plots suggest a cleaner story than any single metric does on its own:

1. Early layers mostly look like feature transport and filtering. Routing is validated, selectivity is modest, and OV behavior is often suppressive or transform-light.

2. Mid layers, especially the Layer 6 band, look like a routing bottleneck. This is where the map is most predictive, selectivity spikes, identity-sensitive routing appears crisply, and several of the strongest validation-confirmed heads cluster.

3. Late layers still show strong structure in the map, but the QK side becomes anti-predictive at runtime. The natural reading is not “late layers are uninterpretable”; it is “late layers are where this fixed linear feature basis stops being a reliable behavioral proxy.”

I think this is the most important qualitative insight in the post. The headline is not just that certain layers spike, but that the kind of structure the map is surfacing changes with depth.

The program distribution by layer (Figure 19) makes this concrete: early layers carry substantial SUPPRESS (peaking in L2/L4–L6) alongside CROSS_COPY, mid layers mix CROSS_COPY / SUPPRESS / REINFORCE, and late layers tilt toward CROSS_COPY. The dominant program motif changes with depth even when the global histogram looks stable.

Interpretation

Weight-space analysis is not telling you “what the model will output.” It’s telling you what kinds of operations are available in the weights, and where in depth those operations concentrate.

Safest takeaways from this weight-space + SAE-basis analysis:

• Local vs global layers are qualitatively different substrates. If a behavior requires long-range access, only global layers can implement it directly.

• There are depth bands with unusually crisp routing (spikes in selectivity and sometimes identity-sensitive diagonal mass).

• Redundancy varies a lot. Some layers look like “few special heads,” others spread structure across many heads.

• Intervention heuristic: Combining selectivity (0.3 weight), RoPE stability (0.4), head diversity (0.2), and max-gap (0.1) into a composite score, Layer 6 emerges as the strongest routing intervention candidate (score 0.51), followed by Layers 22–23. I now view this as a prioritization heuristic rather than a steering claim: the Layer 23 ablation result shows the map can identify leverage without predicting direction of effect.

Behavioral Follow-Up: Predictions and Early Results

Limitations

Linearity assumption. We fold RMSNorm into weights but ignore the token-dependent factor, making the analysis linear-in-direction rather than exact-in-activation. This approximation is tighter when residual stream norms are stable (early/mid layers) and degrades where norm variance is high (late layers). This is likely one contributor to the late-layer anti-prediction effect.

SAE basis is not ground truth. SAE decoder directions are non-orthogonal and learned via reconstruction loss – they may not capture the model’s “true” features. Feature splitting (one concept spread across multiple SAE features) and superposition (multiple concepts encoded in one direction) can distort the affinity matrix . We do not apply Gram-matrix correction for superposition in the main analysis, though the infrastructure exists.

Late-layer anti-prediction. After ~Layer 17, the weight-space matrix predicts the opposite of observed attention (mean Spearman = -0.14). This is not just noise – it’s systematic. All late-layer metrics (selectivity, program distribution, write archetypes) should be interpreted as “what the weights make available” rather than “what happens at runtime.” The sign flip likely reflects LayerNorm interactions, residual stream composition, or accumulated superposition that the linear analysis cannot capture.

Feature-space selectivity token-space selectivity. A head can have diffuse routing over SAE features while being sharp over token positions, or vice versa. Our selectivity metrics measure peakiness in feature space – this is a different (though correlated) quantity from attention sharpness over positions.

No direct causality. Weight-space analysis maps available operations, not actual behavior. A head scored as high-COPY might never copy in practice because it never receives the right input distribution. Causal validation (ablation, activation patching) is required to make functional claims – which is why Post 2 is in preparation.

Single-seed sensitivity. As noted in the Feature Subset Note above, the main weight-space metrics use a 4,096-feature random subset. Absolute values (e.g., exact SelU numbers) shift across seeds. Qualitative patterns (which layers spike, program distribution shape) are stable in spot checks but have not been formally assessed for robustness.

Appendix: Metric Glossary

• top1_mass = mean over query features of max softmax probability across keys

• SelU = top1_mass (1/n)

• diagonal_softmax_mass = mean softmax mass on the diagonal entry (identity-sensitive)

• DiagMU = diagonal_softmax_mass (1/n)

• MaxGap = average(top1_logit - top2_logit) per row

• RoPE (stability) = AUC summary of how stable remains under RoPE-relative rotations (called semantic_controllability in the code)

• Copy = mean diagonal of W2F (self-write strength)

• Trans (transform) = max off-diagonal of W2F (strongest i ji mapping)

• COPY / TRANSFORM / BROADCAST / SUPPRESS / DIFFUSE = write archetypes derived from W2F. DIFFUSE = no score crosses any threshold

• Feature programs = composed triplets i j k with a small taxonomy