Can a Little Lookahead Replace a Full Training Run?
The best shortcut never looked at the new task
lookahead ladder preregistered, with amendments results: July 20, 2026
At the end of Why Crowding Protects, I had a protection signal that worked unusually well and an awkward bill to pay. To decide which neurons to protect during training on a new task, the signal first needed to see an entire unprotected run of that training.
That signal, K2, measures how changes to different neurons couple through the old task’s loss curvature. It beat the cheaper selectors in the tested setting. But its information came from the future we were trying to avoid: let the model learn the new task and forget some of the old one, inspect the update, then start again with a protection mask.
The obvious next question was whether a small preview would be enough. One training step? Four? Sixteen? Or could the mechanism suggest a better selector before seeing any new-task gradients at all?
The experiment is now complete. The shortcuts improved on weight crowding, but the ordering was unexpected: the best shortcut never looked at the new task. A little lookahead made it worse. More lookahead partly recovered the loss, without catching up to the zero-lookahead estimator, let alone K2.
A zero-lookahead estimator beats crowding on 6/6 evaluation seeds. The tested 1-, 4-, and 16-step previews leave a large gap to the full-run selector; no rung satisfies the registered saturation rule. The preview scores become more similar to K2 as lookahead increases, but their protection quality does not follow the same ordering.
What the ladder buys
The setup is the same small language model, TinyLlama-1.1B, first trained on math and then on code. During code training, each selector freezes 20% of the MLP neurons in every layer. Afterwards, we measure the increase in held-out math negative log-likelihood, or NLL: how much worse the model has become at predicting the old task. Lower degradation means better retention.
The selectors differ in what they know when choosing that mask.
| Selector | Information used |
|---|---|
| Crowding | The weight file: which neurons have nearly aligned output directions |
| L0: zero lookahead | The weights and a forward capture on old-task math contexts |
| L1, L4, L16 | K2-style curvature estimates along updates from 1, 4, or 16 early code-training steps |
| K2: full run | Curvature along the update from a full, 500-step unprotected code-training run |
The early previews are cumulative AdamW updates from the same starting checkpoint. The masks are constructed using seed 0; their retention value is evaluated on six separate training seeds. The old-task contexts used for L0 exclude the held-out outcome set.
There is a clarification to the previous post’s closing sketch. I described zero lookahead as a ledger built from the weight file. The implemented L0 also uses old-task contexts. It has zero access to new-task gradients, but it is not a zero-data estimator. And this is a ladder of information access, not a measured wall-clock cost comparison: constructing the curvature scores has a cost of its own.
L0 follows the mechanism’s anatomy. It combines overlap between neurons’ write vectors with how their two read paths respond on math contexts, then ranks neurons by their total coupling score to other neurons in the layer. This is more specific than asking which neurons fire together. It estimates coherence between their sensitivities without knowing the new task’s actual update direction.
Data and provenance. The result report, preregistration and amendments, per-seed outcomes, and recorded statistical analysis accompany this post. The evidence bundle includes the experiment’s source scripts and a figure renderer. These are the July results; writing this post did not involve new model training. See the bundle notes for its scope and hashes.
The first step goes backwards
Here is the retention result. Every dot is an evaluation seed; the short lines mark means. The figure uses the same fp32 evaluation convention for every arm, rather than mixing numbers from the earlier post with the new runs.
L0 reduces math degradation by 0.1667 NLL relative to crowding, with a 96.7% bootstrap interval of [0.1613, 0.1715] for the reduction. It wins all six pairs; the exact one-sided sign-test p-value is 0.015625, below the registered per-comparison threshold of 0.05/3.
So the mechanism does give us a useful improvement over weight geometry alone. Old-task contexts add information even before we consult the new task.
The surprise comes when we do consult it. L1, the one-step preview, is worse than L0 on every seed, by 0.049–0.068 NLL. Four steps improve on one, and sixteen improve on four, again on every seed. But neither catches L0. The observed mean ordering, from best retention to worst, is:
K2 → L0 → L16 → L4 → L1 → crowding.
That is not the smooth progression from cheap approximation to expensive oracle that motivated the ladder.
L0 uses a different estimator from the K2-style preview arms. Their comparison therefore does not isolate the effect of adding new-task information to one fixed algorithm; it compares the selectors we actually constructed.
A passing trend is not a monotone ladder
There is an easily misread result in the registered analysis: the ladder’s trend test passes.
The test asks whether the rank correlation between the proposed rung order and retention benefit is positive in every seed. It is: Spearman’s rho is +0.400 in all six. The excellent full-run endpoint is enough to make the overall trend positive even though the first step goes backwards.
The separate descriptive check asks whether every adjacent step improves retention. That check fails immediately at L0 to L1. These are different questions, and the passing answer to the first cannot substitute for the failed answer to the second.
The saturation test is less ambiguous. Before running, I defined practical equivalence to K2 as a difference within ±0.09 NLL. The test starts at L16 and proceeds to cheaper rungs only if L16 passes. Its gap to K2 is 0.2623 NLL, with a 96.7% interval of [0.2571, 0.2674]. It fails by a wide margin, so the sequence stops: no saturation rung.
The remaining rung means are descriptive comparisons, not additional equivalence tests. Even L0’s mean gap is 0.2342 NLL. The tested previews have not bought back the full-run selector’s value; this experiment does not tell us how many further steps would be enough.
Looking like K2 is not the same as protecting like K2
The score diagnostic makes the result more interesting. As the amount of lookahead increases, neuron rankings become more similar to the full-run K2 ranking: median within-layer Spearman correlation rises from 0.306 for L0 to 0.370 for L16.
That is what we hoped the previews would do. Yet the least similar score, L0, supplies the best protection among them.
A score can approximate another score more closely without improving the decision made from its top 20%. This experiment demonstrates that mismatch; it does not isolate its cause. It also explains why score agreement was registered as a diagnostic rather than a gate for selecting a winner.
One possible contributor is noise in the early update. Repeating the one-step capture produced about 6.3% relative direction jitter. L1’s curvature estimate inherits that uncertainty. This makes a noisy preview a plausible explanation for its poor showing, but we did not run the independent capture experiment needed to distinguish that explanation from a limitation of the estimator itself.
Does the ordering survive a different new task?
A small transfer experiment replaced code with fiction. L0 and crowding reused their masks because neither depends on the new task; the four-step preview and full-run K2 were rebuilt on fiction. A random mask supplied the remaining reference arm.
Across the five arms, the mean ordering was identical to the corresponding code experiment: K2 → L0 → L4 → crowding → random. The original prediction had L4 ahead of L0. The same unexpected reversal therefore appears on both new tasks.
That transfer used three evaluation seeds and was registered as estimation only. It is encouraging evidence that the ordering is not specific to code, not a cross-model claim or a precise estimate of how often it will recur.
What this experiment leaves open
The result supports a narrower practical conclusion than the allocator I sketched last time. In this setting, an old-task-aware estimator improves retention over static crowding. The tested early-update approximations do not reproduce the value of curvature measured along the full update.
The distinction matters because retention is the endpoint here. This post does not establish an improved tradeoff at matched new-task acquisition, or an end-to-end compute advantage. The primary evidence is one model, one math-to-code curriculum, a fixed 20% neuron budget, and the native parameterization with AdamW. The optimizer-coordinate boundary from the previous post still applies.
The remaining K2 gap is a research question. We have not determined whether closing it requires a longer preview, a more stable direction estimate, a different way to rank neurons, or information that these shortcuts omit. The finding worth keeping is already concrete: the mechanism helps build a better shortcut, but similarity to the expensive signal is an unreliable guide to how well that shortcut protects.
The protocol was frozen July 18 and amended July 19; results were recorded July 20. Three primary comparisons share a Bonferroni budget of 0.05/3. The stored analysis uses 10,000 percentile bootstrap resamples, a fixed RNG stream, and 96.7% intervals. Seed 0 constructs masks and is excluded from evaluation. All displayed outcome values use fresh fp32 evaluation of the recorded after-training checkpoints relative to the same after-math checkpoint.
The original repeat-determinism gate failed under bf16 reduction-order nondeterminism. It was replaced by quantitative repeat-stability thresholds after the repeat-check values were observed. That is a disclosed post-observation amendment, not a pristine preregistered gate pass. The other amendments addressed an unobservable checkpoint-prefix check and the mismatch between the earlier bf16 reproduction target and the new fp32 endpoint. The full protocol preserves their chronology.
The recorded report also notes missing persisted sketch-convergence margins for the one- and four-step arms. Those margins cannot be reconstructed from the saved score files. This limits the audit trail; it is not evidence for an explanation of the ordering. See the report for the gate results, logging gap, context-only comparison arms and transfer details. Its broad statement that amendments preceded affected numbers should be read with the explicit post-observation stability amendment above.