Latency weirdness with USBPre

I’m trying to use a Sound Devices “USBPre” - USB, 2 in and 2 out. It works with Jack and Ardour, all the latest versions of everything. But there’s a weird problem with latency compensation when overdubbing - the later tracks end up delayed a bit compared to what I’m listening to (the initial track). And it’s not consistent. I’ve been trying out different things using a metronome to record an initial track, then feeding the output back into the input to record an “overdub” for comparison. The first overdub tends to be delayed by 75 samples, then subsequent overdubs are another 40 or so each. So each overdub is even more delayed than the previous one, even though I’m recording from the very first metronome track.

Weird, huh? When I switch back to the internal line in and out (MacBook) everything is perfect. Any ideas?