If other LV2 plugins do not have this issue it’s more likely a surge issue.
A shot in the dark: It could be an event-loop bug: the plugin constantly processes events or redraws, perhaps due to animations, instead of using lv2 idle calls.
Carla completely separates the LV2 plugin GUIs in a dedicated process. – Ardour doesn’t. So the plugin shares resources (X11) with the Ardour and other plugins. Does the issue exist in other hosts like jalv.gtk (that behaves like Ardour for the most part)?
Just as a footnote, I’ve seen similar behaviour in commercial, proprietary plugins using both AU and VST formats. The plugins are written incorrectly (as was acknowledged by the plugin devs in the two cases I saw).