the trouble with anarchism is all that liberty
The part where that random guy with a bigger gun than mine will start making decisions for me.
You mean what literally happens today where the US does whatever it wants? And the states with their guns makes the citizens follow its laws?
Yes, something like that. But in case of governments we have a few sources of threat, while without the governments we have millions sources of threat, half of which are completely crazy.

What extra sources of threats do you imagine with a people led system vs a ruling class led system?

The exact same threats exist.