Did macOS Tahoe change how auth works for ssh connections to recently rebooted machines? I don't remember being able to use remote ssh until connecting a keyboard and mouse.
You can finally manage Macs with FileVault remotely in Tahoe | Jeff Geerling