Depending on how the system as a whole is designed, which I don't have details on, the concern I have with storing the keys on disk on a build server vs on a HSM attached to that server is that while compromising the build machine lets you make a build in either case. In the no HSM scenario you can run exfiltrate the key much more easily and then have a better shot at hiding the compromise.
Assuming that HSMs are perfect and nothing will ever go wrong is a bad idea, but if you assume they make compromise harder or more expensive then they still add security.
I don't find a lack of updatable firmware to be a particular issue on a device inexpensive enough to replace, especially when that replacement also means key rotation. Replacement isn't cheap, but it's also not that bad at least for a corporate environment.