Mastodawn

OPNSense's 26.1 release bumps radvd to include PREF64 support, which I've been anxiously expecting.

It also moves isc-dhcp to a plugin, pointing towards either Kea or dnsmasq.

So, that's the push I needed to finally move to Kea.

But, Kea in OPNSense doesn't have DHCP lease DNS registration tied in. Kea does support that with RFC 2136 registration, but it's not available directly in the OPNSense config.

So, the net effect is that Hugo is now completely revamping the home network DHCP and DNS setup, moving it from the OPNSense gateway to the couple of little mini PCs with Adguard + unbound + Knot for DNS, with HA Kea for DHCPv4.

And then I'll finish the OPNSense upgrade 😂

Oh hey; where did this yak and shears come from?

Show thread

Hugo Slabbert ⚠️Feb 1

kea is up with active/standby HA, and pools and options moved;

leases imported to keep a few things consistent across the move;

ISC dhcpd disabled and dhcrelay configured on opnsense;

DHCP scopes handing out the new Adguard endpoints for DNS;

kea is registering forward & reverse DNS with knot;

unbound is configured with stubs pointing at knot for the relevant zones;

Once we've got a critical mass of some DNS internal records registered to knot from DHCP renewals, I'll cut over AdGuard to point to unbound, to basically let things flow through the new paths.

Getting there!

Show thread

Hugo Slabbert ⚠️Feb 2

Cutover complete!

Backups still TBD 😬

Show thread

Hugo Slabbert ⚠️Feb 3

This was more painful that it should have been due to a bad opnsense upgrade (connectivity issues, some pilot error on my part). But, I guess on the positive side I now have a shiny, fresh opnsense installation.

Show thread

Hugo Slabbert ⚠️

ah, right:
Found the reason for the failed upgrade!

Today, I had trouble logging into the admin web UI. On looking into it, I found that the disk had filled up, with the hostwatch service being responsible with tying up over 20 GB overnight.

Being on a DOCSIS/cable connection, we get a lot of chatter on the WAN, with basically a constant stream of ARP hitting the WAN interface.

opnsense had introduced their hostwatch service recently.

With this massive WAN chatter, the hostwatch service basically built up a massive DB in very short order.

Just before I tried the upgrade to 26.1 yesterday, I had done a minor version bump to the latest 24.7 release. While hostwatch had been stopped (crashed?) on my instance before, this seems to have allowed it to start running. Presumably this filled up the disk and borked my upgrade attempt to 26.1, requiring that full reinstall.

I don't really see the benefit of the hostwatch service in my case; I don't really need a full historical / rewind view of all discovered hosts beyond what I can get from just realtime / current ARP and ND. I could probably still keep it around with just disabling it on the WAN interface, but for the time being (maybe because it caused me a bunch of wasted time) I've just disabled it.

Good times.

Show thread

Hugo Slabbert ⚠️Feb 4

yea; that timing lines up.

"Hostwatch ate my system: A story in graphs"

It looks like hostwatch is actually not to blame for the initial issue, and that was just straight up the disk filling over history. There may be a slight chance of hostwatch being to blame here as I did start it myself for a while before this window. So, it could have brought the disk utilization up somewhat that then finally got tripped over during the upgrade attempt.

Show thread

Keenan Tims Feb 4

@hugo Logging everything for a service like this is crazy. Enabling such an immature implementation of a useless thing by default is crazier. It should really be a plugin...

Glad you're running into this before me :p

Show thread

Hugo Slabbert ⚠️Feb 4

@ktims yea, it seems like A Choice ™️ to turn this on by default, especially so early, without a clearer understanding of how it interacts in different environments.

Show thread

Hugo Slabbert ⚠️Feb 4

@ktims looks like https://github.com/opnsense/hostwatch/issues/3 is recorded there. I dunno if in my case that will functionally improve things, but added a comment for my own experience in there at https://github.com/opnsense/hostwatch/issues/3#issuecomment-3849154493

Logging fills disk. · Issue #3 · opnsense/hostwatch

Before you add a new report, we ask you kindly to acknowledge the following: [X ] I have read the contributing guide lines at https://github.com/opnsense/hostwatch/blob/master/CONTRIBUTING.md [ X] ...

GitHub