But when grotesque hacks are needed to make stuff work for users, we do them.
As a developer, and someone who genuinely holds ardour and its developers in high esteem, that kind of adhoc, sticking plaster approach to engineering, and problem solving is deeply disappointing to hear. As a user, that would not make me feel reassured. I had thought (perhaps wrongly) that the open source community prided itself on avoiding such attitudes, and instead designing well engineered software. All of the components contributing to this particular issue are open source, so all of the tools required to fix this in a good consistent reliable way are there, but for some reason there is a nonsensical attitude not to use them. If nothing else, I hope perhaps the calf developers will be able to engineer a better solution.
@linuxdsp I don’t fully understand the technical issues here (day job: I write software for embedded systems but not Linux), but can you explain what would be your ideal solution?
You give the impression that even if the CALF developers “engineer a better solution” that still wouldn’t be quite enough: what else is needed?
(my solution as a user, so far, is to use Linuxdsp plugins mostly )
@anahata: In order of preference, the best solution would be if libfftw could be completely threadsafe - I don’t know the details well enough to say if that would be possible, but perhaps there should be some communication with the libfftw developers to outline this use case. Paul obviously has a solution which he thinks would make libfftw work, even though this seems to be thought of as ‘a hack’ - again I don’t know the details.
If that isn’t possible or acceptable to the libfftw devs, then its really an issue about whether that library is appropriate for the plugin, which it seems it isn’t, and in some cases you just have to write new plugin code rather than use a general library - its difficult but that’s part of the job, its one of the reasons why I have over 50,000 lines of code in the average ‘simple’ plugin, just managing “that kind of thing”, from custom X11 and GUI related stuff to necessary and complex DSP code (make no mistake, I’m not trying to say that anything I’ve created is just perfect or be so conceited as to guarantee it to be trouble free, but it is well tested within the limits of our resources, and it is designed from the ground up with an awareness of these kinds of issues, so I hope it provides a good solution).
What I’m concerned about, and think is wrong is arbitarily ‘hacking’ libfftw and including this in the ardour binary bundle - this doesn’t address the issue for other host applications which use the system libs and it opens up the possibility of an unmaintainable and complex ‘Mix’ of the stuff that may or may not go wrong, depending upon whether ardour is built from source, downloaded pre-built, installed as part of some ‘custom’ audio distro etc etc - and because it is not a ‘hard’ fault, a user with the wrong mix of these components will not be aware of an issue until something disastrous happens.
Its important to fix the problem for users, but its important to fix it in the right way (its a tired analogy but I wouldn’t want to drive a car or fly in a airplane maintained or built by anyone with the attitude that “if grotesque hacks are needed to make stuff work for users, we do them…”
And I don’t want to see ardour move in the direction of only permitting the use of custom bundled plugins or ardour specific plugins which is the only case in which including some custom hack in the ardour bundle would make any technical sense.
just a related topic from the musicians department from the ludwigsburg filmacademy. amogst my fellow students the producing software logic 12 for instance is not being taken seriously as much as cubase anymore, because of the consumer direction apple is heading. although it is a top of the line software, it is more and more smiled upon here because people believe its professional integrity is getting undermined by commercial interests. how true this is, is of course debatable… but in other words, with ardour one gets the feeling that you are dealing with genuinely crafted code considering the dev circumstances. Also the closeness to mixbus and harrison gives the ardour a sustainable face value . In my opinion, the notion of hacky code on the long term, would be a huge sell out and blow for the project regardless of where ardour is currently standing. ardour with the ,slow and steady wins the race" project has an unique image which suggests quality and durability on the long term. I would strongly suggest to keep at it, see the subscribers as furure investors, and not sell out to a few impatient minds (no offence, honestly).
Hmmm, this topic really could be spun out to epic proportions…
I think this is a very clear example of how ‘free’ software can show some vulnerability when placed in comparison with commercial counterparts. My cranium is too small to grasp the literal fullness of Paul and linuxdsp’s debate however I think it is kind of obvious that when people think of ‘professional’ Audio production on Linux Ardour is the first place they will look. I don’t want to belittle anyone’s hard work but some of the FLOSS plugins out there are really not up to par with Ardour’s level of achievement and something like this fftw debacle really shows that in a ‘professional’ versus ‘free’ arena sometimes you get what you pay for. I think linuxDSP and to a lesser known extent Harrison are providing the quality of work that is commensurate with where Ardour is at and are doing the necessary homework and groundwork as well as being there to mop up their own messes. As someone who is a user, distributor and who currently independently packages Ardour I would much rather have plugin developers be held to a higher and more rigid standard than Ardour shipping hacked crutch libraries to hide vulnerabilities in their plugins. In the longview this will produce better plugins of higher quality, I also think the potential for professional production credibility puts Ardour in a different paradigm than more casual ‘bedroom production’ offerings…
GMaq: the problem with the approach you outline is that there is really only one way to “hold plugin developers to a more rigid standard” and that is to prevent them from using the most tested, fastest, most reliable, overwhelming common FFT library in world.
I don’t consider such an approach to be in anyone’s interests.
although i’m risking to turn into mr. smartypants here… i think its not about preventing people from using the fft library but motivating them to choose otherwise. what if the dev team gave out some sort of certificate to plugins that are ,ardour compatible’?
my thought was that plugin developers could use it to advertise their products an give their costumers more confidence in their products. In the end this would also be a measure to spread common knowledge about certain quality standards. as gmaq said, ardour is practically out of competition with other daws on linux, so why not let requirements be heard at this point?
what if the dev team gave out some sort of certificate to plugins that are ,ardour compatible’?
In a very real sense, that is exactly what I’m hoping wouldn’t happen - its not that I object to the idea of giving users that confidence, but, because it sets out a framework / potential roadmap for ‘Ardour specific’ plugins - and that’s at the root of the reason why I disagree with Paul’s proposed method for solving this particular issue. I actually agree with (a) and (b) in Paul’s reply, but I think we disagree on the implementation of it. The whole point of my argument is that by including an ardour specific fftw, if we’re covinced that really is the issue with these plugins, then it inherently makes those plugins ardour-only (and ardour binary only)
As a developer I want to see a diverse range of plugins, and I want users to have compatibility (within the - slightly loose - definition of current linux plugin standards) with as many hosts as possible. When you buy / build / download a plugin you should be able to use that on any ‘compatible’ host - not only on some or other with the right libraries or the right LV2 extensions or the right mix of this or that - that’s exactly the kind of ‘walled garden’ approach people object to about other platforms (its just that on linux it often happens through lack of design competence rather than for deliberate commercial advantage)
We need to move to a more inclusive approach, not one which encourages segregation by design choices - if linux audio is ever to be more than a very small minority pursuit and if there is to be broader interest from a wide range of developers (which in the end is sustainably of benefit to users)
In my opinion LV2 is already in danger of being co-opted as a convenient means of lock-in on other operating systems by at least one well known commercial developer - most likely as an accidental but commercially useful advantage of its lack of adoption on those platforms, but I don’t want to see that kind of thing or anything like it happen (for commercial benefit or just by accident of bad design choices) on linux.
The authors of fftw seem to think that thread safety (where not implemented in some parts of fftw) is the responsibility of software developers using it, so presumably they’d agree that it’s the CALF plugin developers need to take care of this.
I’m still trying to understand the issues here (thanks linuxdsp for the explanation)
It seems that making FFTW thread safe would make life easier for all plugin developers who need an FFT function, but I presume the downsides are
(a) it’s a lot of work for somebody to do (close inspection of a lot of complex code to identify problem areas) and
(b) risk of a performance hit, when many applications don’t need the thread safety (is that even true?)
Has anyone tried persuading the FFTW maintainers to do this?
risk of a performance hit, when many applications don't need the thread safety (is that even true?)
Very much depends on the implementation and / or the nature of the issue - it should not be assumed that locks are explicitly required to provide thread safety in every case, and even so, in a well designed implementation, the performance impact on a single threaded application should be minimal if at all (especially as we are told the issues cited here appear to be in the initial setup functions - not in the main processing)
In a very real sense, that is exactly what I'm hoping wouldn't happen - its not that I object to the idea of giving users that confidence, but, because it sets out a framework / potential roadmap for 'Ardour specific' plugins - and that's at the root of the reason why I disagree with Paul's proposed method for solving this particular issue. I actually agree with (a) and (b) in Paul's reply, but I think we disagree on the implementation of it. The whole point of my argument is that by including an ardour specific fftw, if we're covinced that really is the issue with these plugins, then it inherently makes those plugins ardour-only (and ardour binary only)
And this sums up my other thoughts I couldn’t type out earlier effectively. This is exactly why I would disagree with a specific packaged version of this lib for this purpose.
In my opinion LV2 is already in danger of being co-opted as a convenient means of lock-in on other operating systems by at least one well known commercial developer - most likely as an accidental but commercially useful advantage of its lack of adoption on those platforms, but I don't want to see that kind of thing or anything like it happen (for commercial benefit or just by accident of bad design choices) on linux.
Now this I disagree with:) I am fairly sure I know exactly what you are talking about, but at best there are differences, but more correctly the plugins for the most part can be used on any LV2 capable host, including Ardour. If no other host chooses to be able to host it, that is one thing, but I wouldn’t call that lock in per-say when I can use it or write my own host to use it. I’ll admit it may seem ambiguous at first glance, but I would argue from discussions with them that the goal is very different, and in fact close to the opposite of lock in as they do have testing happen on Ardour as well.
The authors of fftw seem to think that thread safety (where not implemented in some parts of fftw) is the responsibility of software developers using it, so presumably they'd agree that it's the CALF plugin developers need to take care of this.
The issue, as Paul correctly pointed out, is that it really isn’t possible for it to be handled in the plugin development, short of static compilation of the lib (fftw) into the plugin I suppose so that each plugin used their own version of fftw. This of course brings with it a host of other concerns, but on the flip side, considering these are plugins we are talking about, I honestly wonder how much weight those other concerns should have at this point. Otherwise the plugin can’t know the status of other plugins to make multiple thread safe calls to the same library, particularly the more you try to isolate the plugins from the DAW to prevent a plugin from crashing the DAW for instance, the more difficult this becomes and it is already pretty close to impossible.
So let me ask the question I mused about above… Would it make MORE sense for the plugin to statically compile/link fftw in place in it’s own binary?
To throw out my 2 cents, I agree with linuxDSP. The problem is upstream. Trying to fix it downstream won’t ever fix it, only work around it. The ideal situation would be fftw becomes thread safe. Second best I think would be if there were sufficient resources to create an identically api’d fork of fftw (say the fastest fourier transform for plugins, fftp) such that upstream fftw changes can be easily merged in so it has little additional maintenance or the fork can be merged into upstream. Regardless I think the fftw maintainers need to be consulted before the decision is made. If they aren’t interested in making this special use case thread safe, then it isn’t the right library for plugins to use, but a minimally changed fork could be. Perhaps this is trivializing the changes required or too idealistic but it seems the most correct approach to me.
@seablade: I’m just far too cynical for my own good sometimes, put it down to life experience etc I stress this is just my opinion, and I invite people to draw their own conclusions (but perhaps not in this thread…) it still remains that while in theory LV2 is crossplatform, the reality is that, as far as I know, there isn’t a free build of ardour for Windows, and there isn’t yet a released build of A3 for OS X, all of which narrows down the options significantly for users of LV2 plugins as a real crossplatform solution at the present time - however that may have occurred.
I just want to see diverse choice for users (and developers) and that includes choice of DAW and compatible plugins (on any OS)
fftw authors acknowledge the problem and a fix is underway.
Meanwhile calf-plugins have removed their fftw dependency and KXStudio provides statically linked binaries for many plugins to circumvent threading issues.
Being a recording noob, I recently discovered the Calf plugins and used them on an old project as a learning exercise, only to run into this issue when saving. As suggested in an earlier post, recovery was possible through editing the .ardour file. In my case, I only had to disable the 5 band EQ plugin instances, as I did not use any of the other EQs.
Will the fix to fftw/Calf be in the Ubuntu Studio 15.04 release, or should I install the Calf plugins independently?