Mastodawn

Hugo Slabbert ⚠️

OPNSense's 26.1 release bumps radvd to include PREF64 support, which I've been anxiously expecting.

It also moves isc-dhcp to a plugin, pointing towards either Kea or dnsmasq.

So, that's the push I needed to finally move to Kea.

But, Kea in OPNSense doesn't have DHCP lease DNS registration tied in. Kea does support that with RFC 2136 registration, but it's not available directly in the OPNSense config.

So, the net effect is that Hugo is now completely revamping the home network DHCP and DNS setup, moving it from the OPNSense gateway to the couple of little mini PCs with Adguard + unbound + Knot for DNS, with HA Kea for DHCPv4.

And then I'll finish the OPNSense upgrade 😂

Oh hey; where did this yak and shears come from?

Hugo Slabbert ⚠️Feb 1

kea is up with active/standby HA, and pools and options moved;

leases imported to keep a few things consistent across the move;

ISC dhcpd disabled and dhcrelay configured on opnsense;

DHCP scopes handing out the new Adguard endpoints for DNS;

kea is registering forward & reverse DNS with knot;

unbound is configured with stubs pointing at knot for the relevant zones;

Once we've got a critical mass of some DNS internal records registered to knot from DHCP renewals, I'll cut over AdGuard to point to unbound, to basically let things flow through the new paths.

Getting there!

Hugo Slabbert ⚠️Feb 2

Cutover complete!

Backups still TBD 😬

Hugo Slabbert ⚠️Feb 3

This was more painful that it should have been due to a bad opnsense upgrade (connectivity issues, some pilot error on my part). But, I guess on the positive side I now have a shiny, fresh opnsense installation.

Hugo Slabbert ⚠️Feb 4

ah, right:
Found the reason for the failed upgrade!

Today, I had trouble logging into the admin web UI. On looking into it, I found that the disk had filled up, with the hostwatch service being responsible with tying up over 20 GB overnight.

Being on a DOCSIS/cable connection, we get a lot of chatter on the WAN, with basically a constant stream of ARP hitting the WAN interface.

opnsense had introduced their hostwatch service recently.

With this massive WAN chatter, the hostwatch service basically built up a massive DB in very short order.

Just before I tried the upgrade to 26.1 yesterday, I had done a minor version bump to the latest 24.7 release. While hostwatch had been stopped (crashed?) on my instance before, this seems to have allowed it to start running. Presumably this filled up the disk and borked my upgrade attempt to 26.1, requiring that full reinstall.

I don't really see the benefit of the hostwatch service in my case; I don't really need a full historical / rewind view of all discovered hosts beyond what I can get from just realtime / current ARP and ND. I could probably still keep it around with just disabling it on the WAN interface, but for the time being (maybe because it caused me a bunch of wasted time) I've just disabled it.

Good times.

Hugo Slabbert ⚠️Feb 4

yea; that timing lines up.

"Hostwatch ate my system: A story in graphs"

It looks like hostwatch is actually not to blame for the initial issue, and that was just straight up the disk filling over history. There may be a slight chance of hostwatch being to blame here as I did start it myself for a while before this window. So, it could have brought the disk utilization up somewhat that then finally got tripped over during the upgrade attempt.

Keenan Tims Feb 4

@hugo Logging everything for a service like this is crazy. Enabling such an immature implementation of a useless thing by default is crazier. It should really be a plugin...

Glad you're running into this before me :p

Hugo Slabbert ⚠️Feb 4

@ktims yea, it seems like A Choice ™️ to turn this on by default, especially so early, without a clearer understanding of how it interacts in different environments.

Hugo Slabbert ⚠️Feb 4

@ktims looks like https://github.com/opnsense/hostwatch/issues/3 is recorded there. I dunno if in my case that will functionally improve things, but added a comment for my own experience in there at https://github.com/opnsense/hostwatch/issues/3#issuecomment-3849154493

Logging fills disk. · Issue #3 · opnsense/hostwatch

Before you add a new report, we ask you kindly to acknowledge the following: [X ] I have read the contributing guide lines at https://github.com/opnsense/hostwatch/blob/master/CONTRIBUTING.md [ X] ...

GitHub

"Musty Bits" McGee Feb 1

@hugo that's an awful mDNS implementation

Hugo Slabbert ⚠️Feb 1

@arichtman heh, yea.

Some of the gear that I need/want stable DNS names for doesn't support mDNS unfortunately, specifically the cameras. I definitely have been leaning into more mDNS, but it doesn't quite cover everything for me yet.

DNS for v6 hosts with just using SLAAC is still pants. But, funny enough: those devices that don't support mDNS also don't support v6. For the v6-capable hosts, mDNS seems to have better coverage.

I'm still in this weird space where I don't quite trust mDNS to "always just work" or put it in line for something critical. Maybe that's just something I need to battle test more. Eg I create mDNS names and service entries for just about anything I add in these days, but then still generally just use those for admin and also add in standard CNAMEs for the actual client configurations to point to. Which seems a bit silly and is probably just a "me" thing.

"Musty Bits" McGee Feb 1

@hugo nah mDNS is too patchily supported, noisy, and slow IME, I was jut yanking your chain

Hugo Slabbert ⚠️Feb 1

@arichtman fair. I've probably leaned on it a bit more as it's fairly ingrained in the Home Asssitant ecosystem.

Keenan Tims Feb 1

@hugo Kinda surprised they're deprecating ISC without this feature tbh. It's been blocking me from moving to Kea too. ISTR there was another thing too. It might have just been that it wasn't possible to run Kea on some interfaces and ISC on others simultaneously. I probably have room on the home assistant k8s box to run DNS, I guess (which is alongside the opnsense VM). I don't really have any other 'reliable' machines to put it on. I don't really *want* to though. This is like table stakes basic stuff.

Accept a dependency on my public internet DNS and put it there maybe? Hmm.

Hugo Slabbert ⚠️Feb 1

@ktims the answer seems to be that if you want DNS entries from DHCP to just use dnsmasq, with the language basically being that Kea is the option for more "advanced" deployments.

ISC isn't being totally deprecated in OPNSense yet, and they do appear to have something in place that if you upgrade from an earlier release and use ISC that the plugin is installed for you automatically. But, it does seem like the writing is on the wall.

For mixing and matching:
That might be possible? As I was switching, the ISC enablement is definitely per interface, as was dhcrelay, so I could move that an interface at a time. I'm not sure about the Kea side in OPNSense, but it does look from the docs to have a listening interface setting, as a well as an option to bypass the GUI config and just manage the config file manually.

Keenan Tims Feb 1

@hugo Fair enough I guess.

Both web UIs have configuration for selecting bind interfaces.

The problem is ISC listens on *:67 regardless of which interfaces it's enabled on so Kea can't bind the socket anywhere else. OpnSense does send the interface names on the dhcpd command line so I guess it's a limitation.

dhcpd    dhcpd      97009 12  udp4   *:67                  *:*

I thought maybe this was necessary to capture the broadcasts, but Kea itself only binds to the interface IP, so I suppose not.

Hugo Slabbert ⚠️Feb 1

The problem is ISC listens on *:67 regardless of which interfaces it's enabled on so Kea can't bind the socket anywhere else.

Ah, weird. I definitely had both ISC and dhcrelay active in my opnsense install on separate interfaces while messing around with things, and confirmed that functioning (test interface got relayed correctly).

But, I never got so far as a netstat to check there as things didn't block or throw any errors in the UI, and functioned as expected.

Keenan Tims Feb 1

@hugo Ah maybe they fixed it? I haven't upgraded yet.

Kea gives:

failed to open socket: Failed to open socket on interface vtnet0_vlan6, reason: failed to bind fallback socket to address 10.6.6.1, port 67, reason: Address already in use - is another DHCP server running?

Hugo Slabbert ⚠️Feb 1

@ktims weird, yea. I'm still on 25.7 at the moment. I did only flip to dhcrelay on OPNSense while ISC was still running on other interfaces, rather than Kea on OPNSense, but with that failure mode I would have assumed the issue is in ISC's binding, not something Kea is doing.

🤷

Keenan Tims Feb 1

@hugo Turns out I don't know my own infra. I am already running pdns-auth for k8s ExternalDNS. And already using RFC2136 from ISC DHCP. lol.

Unfortunately while Kea fully supports this, there's not even a knob to enable the kea-dhcp-ddns daemon in OpnSense (and accepting manual configuration). The only way to use it would be to manually edit the service template to enable the daemon and use manual config. But that won't survive upgrades, so eh... I want to keep all critical network infra inside OpnSense.

I will stick to legacy ISC for now.

Hugo Slabbert ⚠️Feb 1

@ktims yea, exactly that.

It's a bit funny that they don't have support for Kea ddns.

I've been moving some key services off to a couple of mini PCs over the last while, so that's a workable enough plan for my environment. But it would be nice if OPNSense supported RFC 2136 ddns for Kea. Perhaps it's something they'll roadmap for the future.

Keenan Tims Feb 1

@hugo Someone did a pretty well fleshed out PR for full config generation support. But interest from the core devs in accepting it is lukewarm at best.

Kea: Add DHCP-DDNS Support by ruifung · Pull Request #9401 · opnsense/core

This PR adds full support for enabling and configuring the Kea DHCP-DDNS server. This will enable Kea to perform Dynamic DNS updates over RFC2136 for both IPv4 and IPv6. Are there any docs changes ...

GitHub