p̻̻̥r̥̻̥o̻j̤͛ec͔t̞dp Jan 24, 2023

I've been meaning to revisit running iSCSI multipathing in my Proxmox cluster. I previously had it set up with a TrueNAS machine providing storage but I'm now utilizing a Synology. During the initial cluster configuration I attempted the iSCSI config but failed to get it working across all cluster nodes.

Instead for several months I decided to go with NFS since it has the most options to store Proxmox data (Disk image, Container template, Container, Snippets, VZDump backup file, ISO image) whereas the iSCSI + LVM option has more limits (Disk image, Container).

I finally revisited this and was able to get iSCSI, Multipath, and the LVM overlay working. I think I have missed one concept though and it's an important one that I need to validate. Before I get there I wanted to share the config items:

1. Synology: Set up Storage Volume
2. Synology: Set up LUN
3. Synology: Generate iSCSI iqn
4. Synology: Add Cluster host iqn's of cluster machines
5. Proxmox: Add iSCSI target and iqn at the cluster level
6. Proxmox: Add iSCSI target 2 at the cluster level
7. Proxmox shell: Install open-iscsi and multipath-tools if you haven't already
8. Proxmox shell: verify wwid's of the newly generated /dev/sdb, /dev/sdc (example disk names) ensuring that the wwid's match and are the correct iSCSI targets.
9. Proxmox shell: Configure /etc/multipath.conf to match your storage device, including denying the multipath management of all devices except for the explicit wwid if your iSCSI devices.
10. Proxmox shell: Restart multipathd. Once the multipath alias device appears you will be able to see it as an LVM Physical Volume (PV) with pvdisplay.
11. Proxmox shell: You may now generate an LVM Volume Group (VG) which will appear across the whole cluster.
12. Proxmox: You can now add an LVM overlay at the cluster level by selecting your new Volume Group.

Now I'm able to use my iSCSI-backed LVM volume across all clustered nodes for HA of VMs and Containers.

#homelab #selfhosted #proxmox #synology #iscsi #cluster #storage

Show threadp̻̻̥r̥̻̥o̻j̤͛ec͔t̞dp 

I mentioned I missed a concept, so where am I failing?

Well I noticed after migrating several VMs from the NFS storage over to the iSCSI+LVM storage that I was still only using a single path to the NAS by watching the port activity on the NAS. There are two network paths to utilize and I figured Multipath would handle that.

I then started testing by manually unplugging cables to force the second path but that didn't work. I also tried toggling the second iSCSI connection within Proxmox with no luck. The way I am adding the iSCSI connections is likely incorrect. What I was doing was using two different iSCSI portal IPs -- the two different IPs on the NAS, however they both point to the same iSCSI iqn on the NAS.

I'm pretty sure I need to now generate a new iqn that targets the same LUN on the Synology side, then re-add the second path IP to point to the second iqn. I'm a bit hesitant to do this since last time I broke the multipathing when I removed an iSCSI connection from Proxmox and had to do a lot of work to fix it. To be continued...

It has been a good exercise in figuring out how this all works. If you have experience doing this it would be great to hear what your experience was like and how you've configured things.

#homelab #selfhosted #proxmox #synology #iscsi #cluster #storage

Jan 24, 2023 at 8:25pmWeb