UFBX is a single-header library to import FBX files and it's almost twice faster than the FBX SDK!
https://github.com/ufbx/ufbx
GitHub - ufbx/ufbx: Single source file FBX loader

Single source file FBX loader. Contribute to ufbx/ufbx development by creating an account on GitHub.

GitHub
I actually used a different hashing algorithm, so a part of the difference could come from this. Anyway, as a single-header library it's much faster to compile so even if it was on-par with FBX SDK's performance it would still be a great deal.