What? I dont get the comparison is this a reference to baseball?

The second part of the metaphor, yes. It’s criticizing people born into extreme privilege for ignoring their privilege and thinking they earned what was actually given to them without effort or merit on their part, and then those people criticize others for not working hard enough to attain what they themselves didn’t have to work for.