I have previously had the misfortune of having formatted an external drive to a wrong format, so that all my ardour projects became un-openable after a linux distro-hop.
I really want this to not happen again, so what is the format I should choose to remformat my external disk to before backing up my projects?
If you want all three, then exFat is your choice. If Linux only, any number of options are good possibilities, ext4 probably being the simplest, but ask half a dozen people (That know and understand the topic) and you would get a half dozen answers as to what is ‘best’ on Linux these days.
plus you’d get a dozen demonstrations at why they’re telling you there are not much more than one or two people that understand and know the topic
I agree with Seablade, use exFat if you are absolutely sure you’ll need quite often (say one medium project per month or one big per year) to cross worlds, or play it cool and easy and use whatever ext4 you prefer and in case of a project needing to cross platform ask the cageyOSes admins to use their best compatibility wondrful tools to read/write ext2/4 filesys or their favorite rosita embedment (note that the good ones will actually do it easily plus being happy to do it )
The problem with all the FAT derivative filesystems is they do not guarantee case sensitive filenames, and I think Ardour needs case preserved.
According to this post by x42 that is the case (ha ha, get it? case?): Ardour sessions are case sensitive
The recommendation from that thread was to work with Ardour sessions on the local disk, using whatever case sensitive filesystem was available (i.e. any Linux or Unix native filesystem, NTFS on Windows) and then zip the project directory to store on an external disk which was FAT32.
The Wikipedia filesystem comparison chart says exFAT is not case sensitive, but is case preserving, so a little bit of a grey area. FAT32 was technically case preserving with an extension, but it seems to have caused problems in the past, so presumably that extension was not universally used.
I think in practice NTFS is practically case preserving, case insensitive (one of those weird Windows caveats, technically the filesystem is case sensitive, but the Windows APIs you use to access normally on Windows are not case sensitive), so maybe exFAT will work fine. Definitely give it some stress testing before you rely on it.
So it is a bit more complicated with ExFat here. The filesystem IS case-insensitive, but filenames can have case. What happens, at least according to a brief search, ais that when searching for files as part of the filesystem operation, the file name searched for is converted to upper case. What this means is that while the file system is case insensitive, since the conversion happens at the filesystem level, Ardour is unaware of this, and it appears that it works fine.
The practical end result, is that so long as all filenames are unique, the session works fine. I did just test this at a VERY basic level by creating a new session and recording a file on my HFS+ case sensitive drive, then copied the session to ExFAT, confirmed it was working, and finally renamed the audio file in the interchange folder, only changing the case of one letter, and the session remained working. Now if I copied the session back to a case sensitive file system AFTER making that change, it would obviously not work, but if I copied it back before then, the case is preserved and the session works.
This is different from say FAT32, which is also case insensitive, but does not recognize case at all in filenames, and what tends to happen is that when copying files, the case gets changed to upper case, so when copying back to a case sensitive filesystem, you have effectively changed the name of the file. That is why FAT32 doesn’t work great for an interchange filesystem.
So end result, it works much like your description of NTFS.
And end result of all this is that it tends to work well enough for interoperability functions honestly.
don’t know what happened, but as stated that’s basically impossible… unless you reverted to a much older distro, or a very “crippled” one, any Linux distro should be able to access any of the supported file systems (well, unless you’ve used a non-native file system, such as NTFS or exFAT, which may be not supported by default by every Linux distro).
if it is for Linux only, neither one. Use a native file system such as ext2 or (better) ext3.
(I’d rather avoid ext4 instead, as it’s not supported by older Linux systems and many 3rd party add-ons for other platforms).
If you need compatibility with every other system, AFAIK FAT32 is the only one which (to date) is well supported by all platforms, including older ones.
But… it does have a lot of limitations (max volume size, max file size, allowed characters and length of file/dir names, etc).
exFAT has less limitations, but it is also less generally supported (also by Linux itself…). Same goes for NTFS.
This is true, but I have never had good USB performance with a fat filesystem. If you’re saving a few small things, it’s fine. But thinking about a backup… brings tears to my eyes. I stick with ext2. If you need to transfer things to another machine, have a separate fat usb for that. With ext2, if your system dies, you can fire up a live Linux session, plug in your drive and do whatever. I would separate the utility of a “backup” drive from a “inter-system transfer” drive. Just my 2 cents.
Speaking of USB, I’ve just tested this with exFAT on a USB thumb drive.
I’ve copied an Arodur session on it, and it loads fine (using Linux).
Edit: I have initially used mkfs.exfat on Linux. The thumb-drive was not readable on macOS (Ventura 13.3.1). I then used Apple’s Disk Utility to create an exFAT system on the thumb-drive. And that worked on both macOS and Linux.
Basically when Windows formats a large-capacity drive in exFAT, it will default to formatting it using a 1024k allocation unit size - this is incompatible with OSX that requires a 128k allocation unit size for exFAT. "
Except in my case it was just a 32GB thumbdrive, not really “large capacity”.
As I said, it is indeed somewhat better than FAT32 (at least, it does not have its severe limitations), but support for it is not as “universal” as it is for FAT32. Do not forget that (just like NTFS) exFAT was born as a fully proprietary file system, and MS have not released its specs until the end of August, 2019. Previously exFAT on Linux was somehow supported (since 2009) only via FUSE (which implies a severe performance penalty per se), and early drivers were based solely on reverse-engineering. Native support have been added only since Kernel 5.4. Latest distros are likely to provide native support for it, but even slightly older ones may not. E.g., IIRC, even Ubuntu 20.04 LTS (and derivatives) supported it only via FUSE.
MacOS supports it only since “Snow Leopard”, 10.6.5.
So yes, (as I said myself) today exFAT may be a viable option… but only if you’re gonna need to access it only from recent versions of the most common platforms. If instead you need to be able to access your files from really any system, including old or “exotic” ones, than FAT32 still has better chances to be supported anywhere. This is what I meant.
OTOH, if he only need to access that disk from Linux, than a native file system such as ext3 is always a much better choice.
My point is that support for it IS fairly universal at this point (Caveat being what Robin pointed out).
You are correct, it was not released publicly until 2019, however exFAT support existed well before that, even on Linux via as you mentioned a Fuse driver.
Correct but the Fuse solution worked fine in practicality (As i was using it quite a ways back), and is the same way NTFS would be supported for instance for most people. Just because it isn’t supported in the kernel doesn’t mean it didn’t work fine for Linux.
You do realize that is well over a decade old right?
I don’t consider a decade old ‘recent’ especially when talking about AV production personally.
That was just a general consideration, for completeness, since he was talking about compatibility. Which I generally intend as the possibility to access the files on the external storage from “any” machine (not necessarily one where you’re actually gonna do any work on the files).
Actually, for an USB disk which is used for actual work / “production” rather than just for “static” file storage / transfer(*), personally I would not use anything but ext3.
Ext3 is the most “resilient” file system in case of accidents such as sudden disconnections / power off without proper un-mounting, even among other journaled file systems (let alone the non journaled ones), it’s basically free from fragmentation problems, etc.
(on fixed disks I’d rather use ext4, which is somewhat faster and more efficient, though a bit less resilient than ext3, or possibly other native *nix file systems, depending on the specific needs).
(*) a removable disk (USB or whatever) is not the best option per se for work/production uses, and not only for audio/video… but I understand that sometimes that can be the only viable one.