Apparently there is no way to run dhcpd that binds to individual interface and #libvirt network on the same host. The solutions are:
1. Find a dhcpd that binds to *:67 (and then only responds to configured interfaces/networks), or
2. Manage the libvirt bridge network interface manually, or
3. Hack libvirt with a custom patch, or
4. Create a separate interface with different dhcpd port, and bind the dhcpd to that interface and prot. Forward dhcp packets from the desired interface to this custom interface and port.
I found this out the hard way when I planned to replace isc-dhcp-server with kea-dhcp4-server.
Apparently the kernel errors out if different applications want to bind to UDP *:port and UDP interface:port. Multiple apps binding to UDP *:port is fine and that is why isc-dhcp-server works with libvirt.
Two apps binding to different UDP interface0:port and interface1:port are of course ok, but unfortunately libvirt generates the bridge dnsmasq config in a way that doesn't allow specifying the specific interface for binding or excluded interfaces: https://github.com/libvirt/libvirt/blob/3f3cb3ab51740c30f1016c4fe657c48a14cc8462/src/network/bridge_driver.c#L1144