Yes, it is indeed very hardware dependent.
In case of @bachstudies requesting a low cpu-dma latency is what made all the difference!
Likely the CPU was unused for relatively long time, went into C3 or C6 sleep (both turn off bus clocks) and re-enabling hw clock takes too long.
Requesting the lowest non-zero latency can be sufficient. That allows the CPU to reach C1, bit no deeper power-saving states. This can still limit the temperature.
On Intel CPUs
sudo i7z is helpful to check.
PS. Compiling the Linux kernel (or Ardour) with constant 100% CPU usage of all cores can equally result in thermal throttling, but in that case the system will just compile more slowly. With audio it may result in dropouts.