Mastodawn

ME: I want a 5 TB hard drive

Amazon: We can do that

Canada Computers: I can give you 12 TB for twice the price

Christine: Wait, Canada Computers has 12 TB drives for *how* much? Get two

Me, walking back from yonge-dundas square the next morning, absolutely twisted, carrying 24 TB of platter drives:

Hey if I want to format an HD for archival purposes, and I want it to be accessible from both Windows* and Linux** without problems, do I use… exfat? Will exfat freak out if I format it at absurdly high sizes like 12 TB, or give me an annoyingly high "minimum file size" or something? Are there any more-reliable/journaled FSes that both these OSes are happy with?

* 10
** Let's say Debian Trixie

Charlotte Mar 1

@mcc chaos option: ntfs as the Linux ntfs driver is pretty good these days.

@Foritus @mcc put one giant bcachefs file on ntfs, and mount into that.

mcc

@rotopenguin @Foritus is this a serious suggestion?

In what way would it be better than NTFS straight?

Why not bcachefs on exfat?

@mcc @Foritus nah I'm just joshing about a much-worse-case setup. Especially considering that Mr. Bcachefs is a vibe coder.

@rotopenguin @Foritus Okay.

Assuming I understand these things in principle but not in detail and am looking for actual help— used to, when I used macs, I could create a "Sparse Bundle Disk Image" and it was like a hard drive in a file, which could grow and and shrink returning space to the host disk as it shrank, and could be encrypted, and could be compressed. Is this a thing I can do from Linux? What's the best way? (Assume for this one question I no longer care about Windows.)

aeva Mar 1

@mcc @rotopenguin @Foritus so like... a folder?

@aeva @rotopenguin @Foritus A folder has several limitations by comparison. It cannot be compressed, it cannot be encrypted with a different key from the drive which contains it, it cannot be mounted read-only, and it cannot be unmounted.

@mcc @Foritus I don't know of any Linux thing that will automatically grow like a sparse bundle, or take kindly to shrinking at all.

@mcc @Foritus throw the concept of "putting a filesystem inside a file on another filesystem" out, it's just compounding the failure modes.

@rotopenguin @Foritus my experience, across 30-ish years of working with linux, is that filesystems are the worst, most incompatible, and most fragile things you can possibly have to deal with. this is why i am being tetchy about this.

@mcc @Foritus the simplest advice I have is - pick one OS that gets to write to the drive, and stick with that. If Windows is the writer, use NTFS. If Linux, use ext4 or btrfs. If *BSD, ZFS. When you want to put stuff on that drive from some other OS, don't do so directly. Do it as a network share. Or copy it to an exfat flash stick first. Let the appropriate OS do the disk talking.

Zimmie Mar 1

@mcc @rotopenguin @Foritus I don’t think I’ve ever seen a Mac OS sparse bundle shrink.

A ZFS bitstream file works roughly that way, though it’s not carved up into smaller, FAT32-friendly chunks. You get one using ‘zfs send’, and you should be able to mount it directly. I don’t personally use Btrfs, but I would expect it to have similar capabilities.

Shrinking such a file when data is removed would involve another ‘zfs send’ operation to a new file on the disk, so you would need at least the size of the file in free space.

Ken McLeod Mar 1

@bob_zim @mcc @rotopenguin @Foritus "I don’t think I’ve ever seen a Mac OS sparse bundle shrink" -- they don't automatically shrink; you have to do it manually by running `hdiutil compact <image>`.

Zimmie Mar 1

@_the_cloud Yeah, that’s what I was thinking.

Brad Mar 1

@mcc @rotopenguin @Foritus loop device?

Elladan Mar 1

@mcc @rotopenguin @Foritus you can absolutely do this in Linux using a loopback device. It creates a block device in a file, so you can do all the regular block device things like encrypt it, partition it or use lvm, etc. It can also be attached to a VM with no changes, just attach it as a raw disk image.

It is also clearly less reliable than using a partition directly because the host fs metadata is a SPOF.

unnick Mar 1

@mcc @rotopenguin @Foritus you could create a sparse file with truncate -s 15T the_file[1], create a btrfs filesystem on it with mkfs.btrfs the_file[2], and mount it with sudo mount -o discard the_file /mnt[3]. heres a demo of that (doas does basically the same thing as sudo, i just absentmindedly switched midway through)

[1] 15 TiB is the max that ext4 lets me create, you can try increasing it if your filesystem supports it. this size will limit how much you can store inside the file.
[2] i chose btrfs over ext4 because btrfs can support compression, and also doesnt need to preallocate space for inodes/metadata.
[3] the -o discard lets it shrink the file when you delete stuff.