You can make it sooo cursed lol.

A KVM usually have circuitry that can handle a specific total bandwidth and a specific number of HDMI or DP ports (I’ve seen a few where using 2x 4K displays would disable the remaining ports until disconnected due to bandwidth).

To make this work as expected for a KVM you need circuitry to handle all ports being used for either standard (expensive, lol), and have each physical port connected to I/O on both the HDMI and DP controller. Or support half and half, but connect each port to even more I/O ports and start doing switching…