A cautionary tale:

#Linux #RHEL #VideoServer

So, at @show we have two #45Drives servers... one is a storinator, running our main video production disk set, with about 700 TiB of drives. The second is a much older (pre-storinator) storage pod, with severely limited I/O but about 400 TiB for the urgent "DO NOT DELETE" stuff on it.

The first is called "Kennedy". It has shares on it named after the Shuttles.

The second is called "Baikonur" it has one share - Buran. It's used for backups.

-----

Before the thanksgiving break, I left both Kennedy and Baikonur up for an extended period, because I was 1) tweaking backups from Kennedy to Baikonur, and 2) I was doing some scrubs.
Turned it off on Friday. Didn't think much about it - other than about three weeks ago, I'd told the server

"Sudo Yum Update"
You can see where this is going.
Kennedy runs Rocky, which of course is RHEL-based.
RHEL doesn't have ZFS natively, so you end up using open-ZFS.
I told Kennedy to update. It did. Silently, building its kernel, without ZFS support.

OH BUT DEAR READER IT GETS BETTER.

Booting the machine, Samba goes "OK, I have THESE SHARES. It is THE LAW that I must MAKE THESE SHARES AVAILABLE". And Samba, being the pit of plague filled pestilence that it is, goes "I MUST SHARE THESE" and then goes "HUH, I CAN'T READ THESE SO LA LA LA LA I WILL MAKE THEM ANYWAY"
So, I logged in to the server, expecting files, and saw nothing. None of the 700TiB of files were there . NONE. No data, nothing. The share was there, it just wansn't mapped.

sigh

After ten minutes, I realized that the new RHEL kernel doesn't have ZFS modules yet that work with it, so... I rolled back 😉

And what have we learned children?

ALWAYS READ RELEASE NOTES and ALWAYS CHECK COMPATIBILITY before doing SERVER UPGRADES.