Finally!
After ~12 hours more testing and debugging, I got CP/M 3 fully booted and working.
It took me quite a while to determine the proper wait states and timings for accessing the bank registers.
Now it boots up without any issue and I couldn't find any errors so far.

