Slow (or none at all) reaction to MIDI CC data

I’m using the very cool 24 x 24 matrix mixer plugin (Robin Gareus) to control output sends to 24 speakers from various chanels.
To use this with a custom-designed MIDI controler for spatial panning, I am assigning CC messages from the controler to each of the virtual pots (matrix points) using “learn”. The learning can be done, but there is a delay of about 2 seconds between the time when I start to send messages, and the time when the parameter in Ardour starts to react.
Unfortunately the matrix mixer does not respond at all to MIDI input after the pots have been assigned.
The controler sends the following MIDI CC data:
“Pan (coarse)” on chanel 1
“Expression (coarse)” on chanel 2
“Effect Control 1 (coarse)” on chanel 3
“Effect Control 2 (coarse)” on chanel 4
“Controller 14” on chanel 5
“Controller 15” on chanel 6
“General Purpose 1 (coarse)” on chanel 7
“General Purpose 2 (coarse)” on chanel 8
… and on and on…
I have tried to control the matrix mixer using the rotary encoders on an M-Audio MIDI keyboard instead, and the reaction seems to be pretty similar (not useable).
When I coresponded with Gareus a couple of years ago (about this project) it was mentioned that HUI might be easier, but the guy who is helping me with the coding for the controler wanted to stick to MIDI if possible.
Any help would be very much apreciated.
Here’s a quick video!
Here’s a page about the project/installation - mostly just for interest, but scroll down to “panning interface” section for the relevant part.

I am also experiencing this using a Korg nanoKONTROL2 as a control surface. It seems like if a controller is idle for some seconds, it takes 2 seconds of vigorous movement for a bound automation control to respond to incoming midi CC. Once it starts responding it keeps tracking the value changes as long as there is no long idle period.

It can make automation Write mode quite difficult.

Just following up here, I think this only happens when the MIDI System is (legacy) ALSA raw devices and a2jmidid. It doesn’t happen with ALSA (JACK2, 1.9.8 and later)

This doesn’t make any sense. If you’re using ALSA for MIDI, you’re not using JACK for MIDI.

Generally, the best performance comes from ALSA sequencer; next best is using JACK1 with it’s builtin version of a2jmidid. After that, all other options are not really worth considering.

I was just typing the label verbatim for that MIDI System option in the Audio/MIDI Setup dialog.

When I try MIDI System: ALSA(JACK1…) I get a connect error, ALSA(JACK2… ) works and properly exposes the MIDI devices