I'm with you - I was on Voat back when it was still Whoaverse. I bailed pretty early once I saw the direction it was heading in.

That said - the nice thing about the Fediverse is that there are multiple communities, with different moderation styles. Lemmy has a sizable communist community, for example, but you don't have to associate with them. In fact, some of the larger Lemmy instances (like Beehaw) don't even federate with them; you have to go out of your way to subscribe to them if your home instance is Beehaw.

So if what you're saying comes to pass, and the Alt-Right comes to Lemmy (I don't think they will, but let's imagine) - you can easily pack up shop and migrate to another Lemmy instance that has a better moderation style, or one that won't federate with those instances. It's just like Mastodon in that respect.