Stuart Longland (VK4MSL)Nov 1

Of course, after the power outage, one of my nodes decided not to come back. Another WD 120GB SSD bites the dust.

I've ratted the SSD out of the now dud `fluorine` node… it was set up identically aside from network, hostname and cryptography keys. I put the SSD in and it booted… I've renamed the node and updated `/etc/network/interfaces`.

Put it back on the rack, re-added it as a MON/MDS node, re-activated the OSD… and it seems all is well again.

Might need to get some new 120GB SSDs in stock since these old WD ones look to be reaching the end of their lives.

#SolarCluster #ceph

Show threadStuart Longland (VK4MSL)Nov 1

```
2025-11-01T16:30:16.382783+1000 mon.hydrogen [WRN] Health check update: Degraded data redundancy: 47/2157581 objects degraded (0.002%), 1 pg degraded, 1 pg undersized (PG_DEGRADED)
2025-11-01T16:30:19.585820+1000 mon.hydrogen [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/2157581 objects degraded (0.000%), 1 pg degraded, 1 pg undersized)
2025-11-01T16:30:19.585879+1000 mon.hydrogen [INF] Cluster is now healthy
```

Let's see how much M.2 SSDs are worth these days. There's been two recent failures, the others will likely fail at some point so I'll get some stock in early.

Show threadStuart Longland (VK4MSL)Nov 1

Just had a look at the local suppliers… seems M.2 SATA is going the way of the dodo.

This is being used as a boot drive… data is on 2.5" SATA for which I've got 2TB Samsung 870 EVOs (and a couple of 870QVOs for emergencies).

The boot drive does not need to be big… even 120GB is heaps. It will see a number of writes though. Years ago I tried 16GB USB drives (Sandisk UltraFit) … worked great for about 6 months then they went read-only!

Umart seems to only have Silicon Power drives in M.2 SATA… $25 for a 120GB SSD, but the comments suggest Silicon Power should not be trusted for their SSDs, lots of reports of reliability issues after months of mild use.

I'm thus hearing the siren song of NVMe… use a 2.5" SSD for the boot drive, and buy a 2TB NVMe SSD to store data. I'll have to think about that.

Show threadStuart Longland (VK4MSL)Nov 1

Performance comparison… this is the 2.5" 2TB SSD (Samsung 860EVO), 6Gbps SATA interface according to `hdparm -tT`:

```
Timing cached reads: 4124 MB in 2.00 seconds = 2064.64 MB/sec
Timing buffered disk reads: 1258 MB in 3.00 seconds = 418.68 MB/sec
```

and a NVMe disk over USB 3.0 (Samsung 970EVO+ 500GB):

```
Timing cached reads: 3880 MB in 2.00 seconds = 1941.74 MB/sec
Timing buffered disk reads: 1110 MB in 3.00 seconds = 369.91 MB/sec
```

…not promising. These servers' motherboards do not support NVMe, so I'd either have to find a way to shoehorn a PCIe extender in and connect it up to a drive in a 2.5" enclosure somehow, or connect it over USB 3.0 — which we've just identified is slower than the SATA route.

Show threadDr. David McBride (dwm)Nov 1
@stuartl Given the recent failure mode, one of the qualities you may care about in any future SSD purchase is, "Power loss protection."
Show threadStuart Longland (VK4MSL)Nov 1

@dwm Maybe. It was gracefully powered off in this case, I didn't just yank the power.

I know sudden power-off is dangerous for anything flash-based, but that isn't what was done here.

These are drives I bought 5+ years ago, so they might just be reaching their end-of-life, having pretty much run every single hour of those 5 years.

Show threadDr. David McBride (dwm)Nov 1
@stuartl Some makes of SSD also had firmware bugs that caused failure after a certain runtime. You may be unlucky enough to have one of those, but lucky enough that it's fixable post-failure with a firmware upgrade.
Show threadStuart Longland (VK4MSL)Nov 1

@dwm Yeah, I could give that a try I guess… not sure how to do a firmware update on these things.

I'm running Linux machines so whatever the process is, it'll have to be done from either that… or Windows 7, I don't have anything else.

Show threadStuart Longland (VK4MSL)Nov 1

@dwm … or maybe not.

```
[3364206.670387] sd 6:0:0:0: [sdb] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[3364206.670394] sd 6:0:0:0: [sdb] tag#15 Sense Key : Hardware Error [current]
[3364206.670397] sd 6:0:0:0: [sdb] tag#15 Add. Sense: Internal target failure
[3364206.670399] sd 6:0:0:0: [sdb] tag#15 CDB: Read(10) 28 00 00 7d 96 00 00 00 80 00
[3364206.670400] blk_print_req_error: 154 callbacks suppressed
[3364206.670402] critical target error, dev sdb, sector 8230400 op 0x0:(READ) flags 0x800 phys_seg 16 prio class 0
[3364206.687903] sd 6:0:0:0: [sdb] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s
[3364206.687912] sd 6:0:0:0: [sdb] tag#8 Sense Key : Hardware Error [current]
[3364206.687924] sd 6:0:0:0: [sdb] tag#8 Add. Sense: Internal target failure
[3364206.687952] sd 6:0:0:0: [sdb] tag#8 CDB: Read(10) 28 00 00 7d 96 28 00 00 80 00
[3364206.687955] critical target error, dev sdb, sector 8230440 op 0x0:(READ) flags 0x80700 phys_seg 16 prio class 0
[3364206.687987] sd 6:0:0:0: [sdb] tag#9 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
[3364206.687990] sd 6:0:0:0: [sdb] tag#9 Sense Key : Hardware Error [current]
[3364206.687993] sd 6:0:0:0: [sdb] tag#9 Add. Sense: Internal target failure
[3364206.687995] sd 6:0:0:0: [sdb] tag#9 CDB: Read(10) 28 00 00 7d 96 a8 00 02 00 00
[3364206.687997] I/O error, dev sdb, sector 8230568 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 0
```

No worry… let's try the other failed disk. It wouldn't even boot earlier today.

Show threadStuart Longland (VK4MSL)

@dwm

```
rikishi /tmp/wd_fw_update # fdisk /dev/sdb

Welcome to fdisk (util-linux 2.41.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

fdisk: cannot open /dev/sdb: No medium found
```

…ookay, second failed disk is proper stuffed now. Never mind, they were going to the e-waste bin anyway.

Nov 1 at 9:09amWeb