It’s not a big deal. My SO and I are very different heights (like a foot/35cm), and it takes all of 10 seconds:
We do it like 2-3x/week, and as the taller person, I’m totally capable of doing the adjustments while driving out of the neighborhood. It’s really a non-issue.
My wife and I are just the right height difference that the little flicky switch on the internal mirror will swap between the angles each of us need.
One day we got a car that has some auto light filtering for night driving and it doesn’t have the switch. Can’t wait to sell it.