This is a known Safari bug. The trick is to lift your finger off the screen for a second and the issue will fix itself.

Although it is a safari bug, it’s still enough to keep me from using voyager. It’s too annoying and happens too frequently. Native apps just perform better.

Which native app are you using? I have a hard time finding anything that looks and works just as good as voyager (aside from this one annoying bug and a few more that got introduced in Sonoma).