Except with JACK there’s a problem… jackd has to be restarted in order to pick up newly measured systemic-latency. In case of USB-devices restarting JACK may change the systemic-latency (Ardour’s own ALSA audio-system does not have this issue).
Some users settle for some average, or simply use a small buffersize to work-around this jack issue.
Internal connections in Ardour are correctly compensated for. Other JACK apps are aligned if they report latency (many do, most also don’t introduce any latency). You can check with jack_lsp -l to list jack port latencies.