Ardour 6.8 destroyed my MIDI regions!

Hi.

As you probably know I’ve made a lot of videos teaching people how to use Ardour, especially for MIDI-sequenced electronic music.

MIDI in Ardour was always buggy and frustrating to work with. However after the 6.0 release things have only been getting worse.

Newly introduced problems include an incredibly destructive bug that happens when editing MIDI regions, and the user’s actions can be applied to any amount of randomly selected MIDI regions in the session, making notes longer, shorter, offsetting pitch or deleting them.

So every time you edit a MIDI region, you may be messing something up in your project, and once the session gets a little larger, it’s impossible to manually make sure that nothing was destroyed. You re lucky if a regions was copied somewhere and only one copy is affected by this bug. Sometimes all copies get mangled.

Just now I’ve been working on a new piece of music and about 1~2 hours in I have had this problem again, where an entire MIDI part (several consecutive MIDI regions) on my timeline have been wiped out:

@paul , @rgareus - please do what you can to eliminate this problem. This is the most destructive bug I have ever encountered, and it feels like Ardour hates your guts. There was something similar before that just wiped out random regions, but this is even worse.

Also - I have already reported an extremely similar problem for Ardour 6.7 around 3 months ago:
https://mantis.ardour.org/view.php?id=8729

As far as reproduction steps go - I am afraid you just need to spend a few hours editing MIDI to encounter it.

Make 6~8 MIDI tracks, create regions with notes on different tracks, duplicate them, edit them, make notes longer, shorter, offset them up and down with the keyboard shortcuts. At some point (for me usually about 1-2 hours into a new project) Ardour will start applying edits to regions that you did not intend to modify. Sometimes if you spot them early you can undo your way to safety (in exchange of loosing recent work), most often you can’t.

Oh, and share you creation afterwards :wink:

When you enter the Edit mode Ardour highlights regions and notes that I am not intending to touch, Sometimes nothing happens, sometimes they get treated like actively selected notes and all my actions apply to them. Sometimes that happens out of view.

It’s hard for me to recommend Ardour for MIDI when stuff like that keeps happening, and updates seem to introduce more and worse MIDI bugs over time. I was hoping the Ardour 7 release will fix all the issues with the NuTempo branch being merged, but I am now doubting that’ll change that much - there’s much more wrong with Ardour MIDI editing than imprecise timing.

It seems to me that MIDI editing is severely undertested by the core team, and they just don’t use that part of the software at all. So it’s completely up to the users to report any issues and provide help fixing them. I’ve reported more of these bugs than I can count but it just doesn’t end.

Maybe you could figure out some automated tests with xnee or something to at least catch regressions? I don’t know, I’m not a developer…

If there’s something more I can do to help getting this (and other showstoper MIDI bugs) - please let me know.

PS: I’ve managed to more or less restore my composition - the notes had zero length so by using MIDI Transform → Set length to exactly 0.25 + trying to recover the lost note length information from memory I got to more or less where I was half an hour ago.

There’s something I won’t get back though - the creative flow.
How can I trust Ardour won’t destroy my work again in the next 5 minutes?
Every move can be the fatal one.
I could save the project in a new directory every 5 minutes or setup Btrfs snapshotting of my projects folder or rsync everything to a new subfolder to keep the MIDI regions files in case I need to restore them (also lots of manual work) spending my time trying to mitigate these bugs instead of making music…

One can’t help, but look up to Zrythm and hope it’ll let him be creative without this kind of problems.

Or go back to 5.12 because it seems to have been much more predictable with MIDI editing. Unfortunately.

4 Likes

It seems to me that MIDI editing is severely undertested by the core team, and they just don’t use that part of the software at all.

Your understanding of this is wrong. First of all, we do not test Ardour much at all. There’s simply far too much for a tiny development team to ever test. That’s one reason we value the user community as much as we do: the people willing to try Ardour as we work on it are often the only way anything will ever be properly tested. It is not atypical in contemporary native (desktop) software companies for there to be 2-3x more testers than developers. We have essentially no (paid) testers, but we do have many wonderful users who are willing to play some part in this.

Secondly, as for working on MIDI and nutempo etc., the reason we have no focused on MIDI before the nutempo merge is not that we expect the time representation to fix many of the shorts of problems you and others are pointing out. It’s because working on those problems before the nutempo merge will simply cause more development nightmares when that merge does happen. Once nutempo has been merged, probably by the end of this week, there’s no theoretical restrictions to what can be worked on with MIDI (we could even end up merging Nils’ tracker, for example).

4 Likes

Thank you. This gives me some hope this nightmare of data loss will end in the near future.

Does this merge mean you’ll bump the version number to 7.0 in nightly builds?

the version number will move to 7.0-pre0 and there will be regular warnings when you start it that it is an experimental version that should not be used for actual work (until we remove the “-preN” part of the version).

1 Like

and meanwhile, if you’re not follow the commit mailing list, Ardour 7.0 is starting to get Live/Bitwig-style clip launching features.

4 Likes

Quick question: are you using snapshots? specifically Session > Snapshot (and switch to new version)? That could explain some of this.

Details:

  1. MIDI data only get saved when the session is saved.
  2. Ardour forks the MIDI data, specific for each snapshot.

(2) Since MIDI data is destructive a version has to be saved specifically for each snapshot.

So if you create a snapshot, the MIDI data may not be saved for the original snapshot (unless you explicitly save), and when re-loading the snapshot it comes back empty.

I’ve tried to address this issue a few times but the bug keeps coming back. I suspect there are also a couple of related bugs, perhaps also re-using source names when switching snapshots…

I am not following the mailing list - that sounds great! Though I think it’d be great if the MIDI CC control system would also get a rework for this to shine - right now it’s extremely bare-bones (unless of course you have something in store already that I am not aware of).

I have not created any snapshots in this particular session.
It was created and then I worked on it non-stop until the issues came up, I was only saving the same original session file.

@unfa Is there a good reason to stick to Ardour for midi?
Besides imho the lack of vital features like a separate midi-editing window, I’ve encountered many (maybe) bugs like disappearing notes, weird quantizing if using different meters, duplicating errors and so on. I can work my way around these things but it costs extra time (and sometimes frustration) and I never get the feeling that Ardour is a good place for midi-editing (and Unfa, I have the impression that you are struggling too).
So, is there a problem with using an other apllication connected to Ardour (for example Rosegarden) to do all the midi-editing stuff? Edit in Rosegarden and play in Ardour synced with jack. Is there a downside to that?

It may be inconvenient at first when one is used to a different workflow, and yes, old habits die hard, but vital is a bit of an overstatement, surely.

There is no consistent session state, but two separate applications where state (save/reload) may or may not be sync.

2 Likes

OK. So there are at least 2 different, distinct bugs that can trigger this.
Not a big surprise, but isolating issues really helps to track things down.

Yeah, you’re right. Maybe It’s not vital but it’s really important to me and I think it has nothing to do with old habits. After several years of using Ardour, it is still very inconvenient the way midi-editing has to be done in Ardour. But I’m still happy using Ardour and these midi-issues have been broadly discussed elsewhere, I’m not intending to re-open discussions :wink:

1 Like

This is then an organisation/workflow issue? Manage two applications instead of one and syncing the saves and so. But there are no technical issues (maybe time-sync-problems or latency)?

Using a separate program to edit MIDI totally defeats the purpose of working inside an integrated environment. I need to have one project per musical piece and one program to load. Anything above that will cause endless pain of keeping the things in sync, and an update to one program can break the whole system easily. Not something I want to experience. That’s why I want to have everything as plug-ins in a single DAW.

LMMS (I wouldn’t really call it a DAW - more of an advanced sequencer) has reliable MIDI but lacks even in basic editing (I think only this year they’ve implemented the ability to select and delete more regions than one at a time).

Zrythm looks very promising but the session format is not stable yet (can’t load projects in a new version), it’s quite crashy overall and severely lacks in performance with bigger projects (I saw MyLoFy pushing it to it’s limits). It may become the best way to do sequenced electronic music with FOSS, but that’ll take at least 2-3 years IMO, assuming everything else will be stagnant. Maybe Ardour will fix it’s MIDI by that point? We’ll see :smiley:

Nothing else in the field seems to fit my baseline needs, which are: great MIDI editing, solid plug-in support, powerful mixing, capable automation, good audio capture and editing + flexible batch exporting (for SFX production and album mastering).

Overall - among integrated music production environments (FOSS OFC) - Ardour gives me the best workflow and most ways to design my signal paths so far the MIDI bus are it’s biggest downside and I deeply hope that will change forever in 2022. If it wasn’t for the extremely frustrating MIDI bugs that often cause data loss, I’d be fine with the in-line piano roll. I’ve given up trying to change Paul’s mind about this like half a decade ago, and I’ve just learned the Ardour’s way.

I am going to keep painkillers* handy and hope for the best.

*A boxing sack?

5 Likes

I believe Qtractor has pretty good MIDI handling but maybe it falls short on some of your other baseline needs?

I agree, separate programs is not the ideal working environment.

I guess I have to figure this out and see if this really is endless pain for me or if it’s worthwhile to avoid the pain of Ardour-midi-editing (and I sincerely hope that that is not endless). I could live (not happy) with inline piano-roll but bugs and lack of (IMO!) basic features add up to a point where I’m ready to try something differrent on the part of midi editing.

Me too. I really appreciate the work being done on Ardour and respect to the guys who put so much effort in it. (and I still hope Paul’s mind is going to change).

I haven’t one (yet) but that’s a good advice :slight_smile:

I normally use Rosegarden for the initial MIDI work (first takes are usually live performances) synced to Ardour/Mixbus32C. Mainly because it’s so easy to merge different takes into one in one track, also because it plays notes when they are starting at the same time as a MIDI “region”, making the workflow much more streamlined.

But it has it’s downsides, one is that you need to use a bridge for having VST plugins inside the DAW. And for me, the the biggest issue is that the whole setup is much more complicated. I really want to work with both MIDI and Audio in the the same environment and Ardour/Mixbus* are the ones I want for that. But I use both Rosegarden and Ardour/Mixbus32C for now and when version 7 is coming, I will skip Rosegarden.

@unfa If the symptom of the hard-to-reproduce bug is really “zero-length” MIDI notes in MIDI regions, then you might want to install my below Lua script as Action Hook via Edit / Lua Script / Script Manager / Action Hooks:

I would be very interested, if it triggers on your “broken” files?
(I tested by manually shrinking notes, which might not be the same situation you’re in??)

But if it works as expected, it then runs automatically on every session save, checks all tracks / active (and non-active) playlists / MIDI regions for MIDI notes with a length less then 0.001 bars. In case such short notes found it will present a GUI warning:
grafik

Ardours Error Log will give you then more details which tracks / playlists / regions had those very short notes in it.

If it works, it is clearly no solutions… but if you save often… then it
1.) will give you an early warning that might prevent complex undo scenarios (which might keep your flow running, when no such warning occurs!) and
2.) in case of warning you might remember your last actions after the last save which will then help Paul & the Ardour team to maybe narrow the problem further down?

Hope it helps…? (I had so much benefit from your Ardour tutorials (MIDI Masterclass anyone?) - so after reading you’re not a developer - this is my unworthy try to pay-back all your gifts to the Ardour community).

In case this bug is meanwhile fixed with Ardour 6.9?? Please somebody comment here…
Then my script was a nice exercise for me do dig deeper in Ardour and Lua Scripting…
:slight_smile:

WOK

2 Likes

@unfa some time ago I had similar issues with midi editing (can’t recall when or what version specifically) and what I observed was that sometimes, selecting notes in one region also resulted in notes in a different (not linked) region also being selected, so edits (like shortening note durations) were applied to those other notes even if they were not currently visible at the given zoom level.

This was super frustrating at the time!

I haven’t observed that happening recently (but I’ve not been doing much midi), so I wouldn’t jump to claiming it as a current bug but it might give you an idea of possible behaviour to look out for in case it explains how it’s happening (ie, it might mean the bug is in the gui areas of the code rather than deep in region management, say).

@coenplanetc I was using Musescore sync’d to ardour for a while when doing some scoring to picture stuff, and the biggest problem is the tempo map. Ardour already has a pretty advanced capability for tempo in that it does ramping and such, so any app that you want to sync with will mean you have to either always use non-ramped tempos and also manually sync the tempo changes between the apps, or the other app will have to be happy to sync to ardour and to just play catch-up when things change (this should be completely possible, but musescore at least doesn’t currently do so). The nutempo stuff that is coming in v7 will probably have more impacts on that, which is why I abandoned the idea of pushing for musescore to sync to ardour until nutempo was in and settled (that was a few years ago, I think).

Hopefully as v7 settles in we’ll see some big improvements in this area - as unfa says Ardour is pretty amazing and does just about everything brilliantly, which is what makes the midi support stand out a bit more as having room for improvement, imo.

1 Like

Thanks for pointing this out. I haven’t done any ramping so far so that’s not a real issue for me now. I do use meter changes but I assume that’s ok if it’s non-ramped.