I am looking for a flexible FM synthesizer for Ardour, under Linux, meaning I could put carriers, multiple layers of modulators and wire them if possible the way I want.
What is the right solution for me? A modular synthesizer?
In Octasine operators can be routed with great flexibility. But there are only four of them. Great synth synth, still, and much more flexible than it may look. The controls are a bit fiddly, though.
For a modular system, I would have a look at Bespoke Synth. Though it is also a non-traditional DAW, i.e. you can create whole arrangements with it, you can also just use it as a modular synth.
If you want full flexibility, I recommend learning the FAUST DSP language. This would be a significant investment in learning, but something that can be incredibly useful to know, if you are into creating your own DSP systems.