virtualizing OPNsense is....not going great

https://sh.itjust.works/post/49824507

virtualizing OPNsense is....not going great - sh.itjust.works

I want to collapse all my little boxes into one powerful box. Ram is super pricy so I built a rig based on the ryzen 5800xt and bought a motherboard that cant PCI passthru the NIC by mistake. Before ordering another motherboard that can passthru the NIC, I booted up bare metal and compared the performance to how it ran virtualized in fedora server. it was better, but still not hitting line level. direct macbook to cable modem: 916/40 opnsense virtualized (with vlans and rules): 699/41 opnsense bare metal (with vlans and rules): 816/39 opnsense bare metal (with vlans and rules and hardware offload fully enabled): 824/40 the only rules in place were the defaults, the rule to block vlans from talking to eachother, and the rule to pass traffic to WAN. when virtualized, I cannot get PCI passthru so I was using macvtap interfaces and virtuio drivers with 4 threads and 4 pinned CPU threads. CPU is a ryzen 5800XT NIC is a dual port intel I226V when virtualized, it was running under fedora server with QEMU/KVM q35 and given 8gigs of ram with hugepage memory and tested in both 2 and 4 thread resource allocation (all confirmed to be on the same 1 or 2 physical cores as the threads) and eventually even giving 4 threads to the virtuio driver (it was only claiming 1 thread before). Bare metal IS definitely helping, so it looks like I need to swap out for a motherboard that can do proper PCI passthru of the NIC (now that I understand the limitations of IOMMU groups they specs of the board dont tell you about I hate them all the more.) but it still cant hit line rates. Theres no IDS or suricata or any of the fanciness turned on yet though, so I just dont understand why its this slow even bare metal.

Yeah, pci passthrough would probably do it. I assume you used the best-supported virtual adapter and drivers in the guest. But failing that, you could also try USB passthrough. You should still be able to get full gigabit (i.e. ~800Mbps) on one of those, even with passthrough.

But I still don’t recommend it, because if your host has issues, your firewall and router do too.

Pass through isn’t going to happen on this board because there are basically just 3 IOMMU groups with ACS, AER, and SR-IOV enabled. 1 for GPU slot, 1 for a single m2 slot, and 1 for everything else. It sucks.

Supposedly an IOMMU-aware NIC can still help me even if the groups are shitty but I’m not certain if that’s true.

It can make a big difference just in the processing power needed if there's anything more intense than a straight firewall. IPS tend to be a resource pig. What are the load numbers saying vs the number of CPU cores available?

I ran into similar (or even worse) choking trying to get it virtualized even with a proper passthrough that I eventually shelved but might take another run at someday. Knocking a couple hundred watts off the stack is always welcome.

Well I was going to drop to IDS instead of IPS and that’s good enough for home use. The load numbers on the host were 2 full cores used but that’s the NIC doing paravirtualizarion crap there. In bare metal, top shows nothing but the fans do spin up so it’s not telling me the whole story.

I think swapping to an i350 nic will help but I’m not certain if it will really help enough.

The performance drop from virtualizing nics shouldn’t be nearly as big. How are you passing the vlans to the VM? are you passing all over one virtio nic or one virtio nic for each.

The setup I ran for multiple years was basicly a bridge interface on the host for each vlan and a seperate virtio nic to the opnsense VM for each, I got almost 10 gbit/s like that with 8gigs of ram for opnsense and 4 or 8 cores (I cant remember) with hyperthreading of a 2nd gen epyc.

I’m not running an epyc. Way too spendy for me. I was using direct attachment pasture but that was failing over to Macvtap because this motherboard sucks and the IOMMU/ACS shit only actually works on the GPU slot and 1 M.2 slot.

Supposedly I can use IOMMU with an i350 and that will work good enough but I’m not certain if this as it’s not the same as a direct passthru so I’m worried I’ll have similar issues.

I’m also reading the i226v NIC I have is kind of ass anyway.

@muusemuuse my OPNsense instance is a pve VM w/ a realtek card passed through. I did everything wrong and committed every sin. It'll work but you'll hate life for a bit.

Have you tried disabling all offloading on the virtual interfaces?

See third general tips - (docs.opnsense.org/manual/virtuals.html)docs.opnsense.org/manual/virtuals.html

Virtual & Cloud based Installation — OPNsense documentation

Offloading was disabled everywhere since I won’t be using it in the long run.
Keep the firewall on dedicated hardware. You don’t want your whole network going down because you have to do some work on the server.
Well in this case I don’t mind it. It’s for home use.