@jpmens I don't really think it needs to be mutually exclusive with a Yubikey... e.g. I have a TPM key for SSH on one system *and* a Yubikey as a portable carrier *and* an eToken 5110 inside another system – and the rest of my security is still "yolo" level, really, but there are some things that the portable Yubikey isn't authorized to directly access while the laptop's key is, for example.
The `ssh` taking 1 second longer might be a hardware difference. On my ThinkPad it's 0.5s in total when using a TPM-backed RSA2048 key (it's not EC for silly reasons but I can't imagine EC being much slower).
Might also be a difference in which parent TPM key that SSH key was created under. Tradition carried over from TPM1 era is to generate a persistent root key (SRK) and store it in the TPM, so my test key is stored using the rough equivalent of "ssh-tpm-agent --parent-handle 0x81000001" (which is the agreed default handle for a SRK). At some point systemd-cryptsetup used a transient ECDSA SRK that it would generate deterministically each and every time. I'm not sure what ssh-tpm-agent does exactly, but I wonder if it might be using a transient parent key or something, which adds to the time needed to load the child key?...