@[email protected] I initially wrote a different reply aimed for a non-musician but on re-read that's not you 🙂
I'd say if the conductor takes you out of the performance, then try your best to focus on the music without the visual component (e.g. closing your eyes, etc.)
As a musician it's also my job to watch the conductor but I like to watch purely to see if my interpretation differs from that of the conductor!
Your post reminded me of a group I was in several decades ago. Every Christmas season we would play Handel’s Hallelujah Chorus and we would have a member of the audience come up and conduct it. I can’t remember how the audience member was chosen. Fun times though. 😊
Anyway, back to your point—I understand. Enjoyed your post.