Is it really possible to get down to 0 Xruns?

lenovens · June 16, 2020, 3:40pm

Some wifi drivers do that. Sometimes there is a better kernel module for the same wifi chip that works better or try a different wifi device or leave it powered off.

To get rid of sudden CPU usage… turn off cron while doing lowlatency work. Cron is supposed to run as a “nice” process but a high enough load is still a high load. I think there are some things done in the update process that end up running atomic.

So long as these things (that cause xruns) can be done when you are not doing recording/playback/audio that is not really a problem.

I have run machines over 24 hours even at forced 800mhz instead of full speed with 0 xruns but that was with no wifi and cron turned off as well as making sure the audio device was properly prioritized. A steady speed is more important than a high speed. Audio seems to tolerate the speed of the cpu core going up better than decreasing… so the xruns happen as the cpu load goes away and ondemand or powersave lowers the speed (boost will also do this). Definitely not plug and play…

topas · June 16, 2020, 5:09pm

I can also use a wired connection - not ideal in the living room - but possible. Another idea is to cache my learning videos and play them in jack using vlc.

I should do this to find out if this is the cause - good idea I’ll try that.

Already read about that in a github issue of cpufreq. I ran the XanMod kernel which set the CPU to steady AND high and I left it there. But this is good to know. I found out that I cannot get xruns by using stress-ng --cpu 4 (so sudden CPU stress seems to be no problem), but I got xruns by filling up memory with stress-ng --brk 1 -t 5. I don’t know if having xruns when memory is full is okay or not.

I think I have to go though the measures of “wiki.linuxaudio org - system_configuration” since I reverted all of these things since with the stock kernel they did not help. That means for example jack is running at realtime priority 10.

That means once it gets run it cannot be interrupted and blocks jackd or an audio interrupt? There are 4 CPU. jackd or the interrupts may get another CPU if there are no higher processes running and so on.

topas · June 18, 2020, 7:02am

Can you tell me how you do that?
I know about the options in the system_configuration of the linux wiki. I mentioned that in a post before:

The wiki links to http://subversion.ffado.org/wiki/IrqPriorities where there are more details about irq prioritizing. I have a problem in step 2 since I do not see so many entries in my list as they have in their list.

Step 1

tobias@tobias-pc:~$ cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       
  0:          6          0          0          0   IO-APIC   2-edge      timer
  8:          0          0          1          0   IO-APIC   8-edge      rtc0
  9:          0          0          0          0   IO-APIC   9-fasteoi   acpi
 16:     170413          0          0          0   IO-APIC  16-fasteoi   ehci_hcd:usb1, ath9k, cx88[0], cx88[0], cx88[0]
 17:          0          0          0        911   IO-APIC  17-fasteoi   snd_hda_intel:card2
 18:          0          0          0          0   IO-APIC  18-fasteoi   i801_smbus
 19:         52          0          0          0   IO-APIC  19-fasteoi   snd_ice1724
 23:          0          0          0         33   IO-APIC  23-fasteoi   ehci_hcd:usb4
 24:          0          0          0          0   PCI-MSI 1572864-edge      enp3s0
 25:          0      67837          0          0   PCI-MSI 512000-edge      ahci[0000:00:1f.2]
 26:          0          0      76501          0   PCI-MSI 327680-edge      xhci_hcd
 27:     172837          0          0          0   PCI-MSI 524288-edge      nvkm
 28:          0         17          0          0   PCI-MSI 360448-edge      mei_me
 29:          0          0        903          0   PCI-MSI 442368-edge      snd_hda_intel:card0
NMI:         10          9         10         10   Non-maskable interrupts
LOC:    4538007    4547056    5124427    4138157   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:         10          9         10         10   Performance monitoring interrupts
IWI:          0          0          0          0   IRQ work interrupts
RTR:          0          0          0          0   APIC ICR read retries
RES:      23675      22253      20577      19001   Rescheduling interrupts
CAL:      27139      22127      29797      23675   Function call interrupts
TLB:      22821      18400      25576      19394   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:          3          4          4          4   Machine check polls
HYP:          0          0          0          0   Hypervisor callback interrupts
HRE:          0          0          0          0   Hyper-V reenlightenment interrupts
HVS:          0          0          0          0   Hyper-V stimer0 interrupts
ERR:          0
MIS:          0
PIN:          0          0          0          0   Posted-interrupt notification event
NPI:          0          0          0          0   Nested posted-interrupt event
PIW:          0          0          0          0   Posted-interrupt wakeup event

Step 2

tobias@tobias-pc:~$ ps -eLo pid,cls,rtprio,pri,nice,cmd | grep -i "irq"
    9  TS      -  19   0 [ksoftirqd/0]
   17  TS      -  19   0 [ksoftirqd/1]
   22  TS      -  19   0 [ksoftirqd/2]
   27  TS      -  19   0 [ksoftirqd/3]
  359  FF     50  90   - [irq/28-mei_me]
 9864  TS      -  19   0 grep -i irq

I have a USB audio interface (focusrite Scarlett 2i4).

lenovens · June 18, 2020, 6:42pm

Are you sure that is the right filename? I think rtirq these days uses /etc/default/rtirq anyway…

and it appears you have an internal audio and an ice1724 as well. Do you also use these?
Looking at your interrupts I see two USB2.0 buses: ehci_hcd:usb1 and ehci_hcd:usb4. The USB1 bus should absolutely not be used for audio work as it is on irq16 along with 4 other devices. Yet it appears that is where the USB sound device is plugged in. Try plugging your USB audio device into any other USB plug until you get it in USB4. I see there is also an xhci process which will have more usb buses… I would not use those either.
Having said that, Assuming you will find a way to get your USB audio plugged into USB4 (and have made sure nothing else is) in /etc/default/rtirq there is a line:
RTIRQ_NAME_LIST=“snd usb i8042”
There is some debate if the i8042 needs to be there at all, but seeing it is last that is fine.
using “snd” or “usb” in here will not help and may even make things worse.you may be able to get away with: “usb4 snd_isc snd_hda i8042 usb”. Then reboot and run in a terminal: /etc/init.d/rtirq status. You should find ehci_hcd:usb4 has the highest priority of all your USB ports with the ice 5 down and the hda 5 more down and the rest of the usb stuff below that. If that does not work as expected… it may be needed to use 23-ehci instead of usb4 in the line above. usb4 is better if it works because it will work even if you change the bios and the irq assigned to USB4 changes.

topas · June 28, 2020, 6:41am

No more - I left them inside since I might use them again. I might get rid of them.

I tried all unused USB ports, but I only got the audio interface isolated on bus 3. Is that okay? It is a USB 3.0 port.

lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 4: Dev 2, If 0, Class=Audio, Driver=snd-usb-audio, 480M
    |__ Port 4: Dev 2, If 1, Class=Audio, Driver=snd-usb-audio, 480M
    |__ Port 4: Dev 2, If 2, Class=Audio, Driver=snd-usb-audio, 480M
    |__ Port 4: Dev 2, If 3, Class=Audio, Driver=snd-usb-audio, 480M
    |__ Port 4: Dev 2, If 4, Class=Audio, Driver=snd-usb-audio, 480M
    |__ Port 4: Dev 2, If 5, Class=Vendor Specific Class, Driver=, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
        |__ Port 1: Dev 3, If 2, Class=Vendor Specific Class, Driver=btusb, 12M
        |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=btusb, 12M
        |__ Port 1: Dev 3, If 3, Class=Application Specific Interface, Driver=, 12M
        |__ Port 1: Dev 3, If 1, Class=Vendor Specific Class, Driver=btusb, 12M
        |__ Port 2: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 2: Dev 4, If 2, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 2: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 12M
        |__ Port 5: Dev 5, If 0, Class=Vendor Specific Class, Driver=usbfs, 12M

I need to find out device names for the name list in rtirq next. (I don’t know where you get your suggestions from - but I would like to know where the name come from. rtirq readme states:

The term service seems to refer to module names and sound device designations (so the output of lsmod and aplay -l respectively) and doesn’t have to correspond to the full output, part of the output may suffice as the rtirq script does the matching itself

)
Then prioritise them.

topas · June 28, 2020, 7:45am

Can someone tell me how to prioritise a USB audio device correctly with rtirq?
I cannot see what to specify in rtirq config file that references the device I want to prioritise. I tried USB1 (For USB bus 1). Can I use snd_usb_audio? That seems to be the closest hit. However whatever I choose it doesn’t appear in the rtirq status.

lenovens · June 28, 2020, 6:21pm

That should be fine. The hard part is finding the service name. Looking at you irqs above, xhci_hcd is on irq 26 so the process you are interested in is: irq/26-xhci_hcd. rtirq does not need the whole thing just a unique part like 26-xhci. so in /etc/default/rtirq at about line 30 you will find: RTIRQ_NAME_LIST=“rtc snd usb i8042” or something like that. you want to change that to RTIRQ_NAME_LIST=“26-xhci snd-ice snd-hda i8042 usb”. That is you want the USB bus your audio device is on to be highest priority. the ICE1724 should be next and the HDA after because HDA could interfere with the others. (being higher minimum latency). I add usb again right at the end because I have seen times where if one USB port is raised to say 90, all the rest end up right under it not at 50. Putting it at the end makes sure they are lower than everything else. Please note that anything else plugged into any USB3.0 port may affect your audio. Even though you have two USB 3.0 buses, I only see one xhci (USB3) process.
Also note that all motherboards are different. finding a good setup for one does not just map to another. The minimum latency varies from system to system. I do not think there are any motherboards that are designed from the ground up for low latency (possibly some of the compaq server setups… but they tend to optimize around network rather than audio). There is just not enough demand and through put is so much easier to measure and advertise, while low latency means lower throughput which to most PR people is bad.

topas · July 6, 2020, 6:53am

I cannot really get a specific IRQ process to be highest priority. Do you know how to solve this? If not I’ll ask the devs of rtirq.
With RTIRQ_NAME_LIST="29-xhci snd-ice snd-hda i8042 usb" I end up with:

/etc/init.d/rtirq status`:
  PID CLS RTPRIO  NI PRI %CPU STAT COMMAND    
  120 FF      70   - 110  0.3 S    irq/16-ehci_hcd    
  122 FF      70   - 110  0.4 S    irq/29-xhci_hcd    
  121 FF      69   - 109  0.0 S    irq/23-ehci_hcd    
  105 FF      50   -  90  0.0 S    irq/9-acpi    
  123 FF      50   -  90  0.0 S    irq/8-rtc0    
  220 FF      50   -  90  0.0 S    irq/18-i801_smb

So irq process 16 is still higher than 29, but why? Do they belong together from the hardware side?

lenovens · July 6, 2020, 5:44pm

irq 16 is not higher but it is the same just listed first. (70 for both) I do not know why this is but suspect some kind of mother board thing. It would be interesting to try:
RTIRQ_NAME_LIST=“29-xhci snd-ice snd-hda i8042 usb 16-ehci”
but I don’t have high hopes. It appears the designer of your MB have made some decisions that are less than stellar for low latency USB use. Being able to use The USB that goes to 23-ehci seems the best solution if it is possible. How does this affect how low of a buffer size you can use without xruns?

topas · July 8, 2020, 12:51pm

I already did that without success; IRQ 16 process was still at the same priority as IRQ 29 process.

I thought to shorten this USB problem I could use the Quartet PCI soundcard and properly adjust the IRQ priorities to see if there is an IRQ priority problem.

I thought…

Then I stumbled over the Linux System configuration page again to see what else is there which I tested with the stock Debian kernel. And I read about the kernel tick ( timer ) rate again. At least that page suggests a kernel with at least 1000Hz tick rate. I wanted to give this a try since the two kernels I tested (liqourix and XanMod) use 250Hz or 500Hz, but not 1000Hz for scheduling. I thought I’ll recompile the xanmod kernel, which I used since I discovered it.
Then i accidentally saw that liqourix has just changed ( back?!) to a 1000Hz tick. I am not 100% sure if I’ve used liqourix with that thick rate, but I’ve never done a ( for me so called) six hour plus (6h+) test. So getting a 1000Hz kernel just got easier. Additionally I thought the two milliseconds scheduling time of 500Hz might be not enough granular for my 2.6 ms latency reported by Cadence. So I wanted to try that before switching the sound card from USB to PCI.
So I updated the liqourix kernel and checked it has a 1000Hz config and did a 6h+ ( in this case nine hours) test.
The result was ( the title mentions it) 0 xruns.

That never happened before but my experience is that one time is not enough so I need to repeat that test at least three times to be more sure that 1000Hz fixed at least my problem.
In addition : the kernel also got other updates which might be the cause. I think the tick rate is a big change (250Hz to 1000Hz) and therefore the cause for the successful test. We will see and I’ll follow up on this.

topas · July 11, 2020, 5:47am

After three tests for at least six hours and not a single xrun I will stop my tests for now and consider this as good enough for my needs.
The cause for the successful tests for me seems to be the change to the liqourix kernel with 1000Hz tick rate for scheduling. I also disabled WIFI and gave my user the ability to use realtime. (WIFI turning off is not needed I think - realtime setup was discussed above - I did not use the audio group - see above)

I will briefly repeat my current test setup:

JACK runs with about 2.7 ms (128 frames / 2 periods) (I didn’t try lower settings)
CPU governor left default
CPU frequency left default
Ardour connected in jack and recording a two track mix
Guitarix running
~~Musescore connected in jack playing a loop~~
Pulseaudio bridged to jack
~~QLC running~~
~~aubiotrack running~~
~~ALSA Midi bridge active~~

I hope my experience helps others.

topas · July 11, 2020, 5:55am

I want to say thanks to everyone who gave me feedback and helped me and others.

topas · June 16, 2022, 5:47pm

There is an update regarding my experience with this topic which I want to cross-link:
https://discourse.ardour.org/t/upgrading-ardour-5-to-ardour-6-on-debian-introduces-xruns

TL;DR: I endet up exchanging liquorix with a RT patched kernel with a 1000Hz timer.

topas · August 19, 2022, 1:04pm

I can add another experience.

I recently set up arch Linux and found that I don’t need special tweaking for it to do low latency audio.

Low latency audio setup works with the stock kernel with a buffer size of 128 (cadence reports 2,7ms) 48kHz.

I turned wifi off (like a reflex, don’t know if it works with wifi on) and granted realtime capabilities to the user, but that’s it.

Works for hours without a single xrun. Never had that with a stock kernel before.