Deviation from social expectations can be attractive as can novelty. Fetishization of a group is often associated with the group being perceived as exotic and possibly even taboo. And there are a lot of people who when they feel shame about their desires lash out against those they desire. The sort of guy who feels the need to tell everyone he’s not into fat chicks? Yeah he’s getting rejected by them in secret.

Your example is someone you personally hate because Gary fucking sucks. But if there were a group of Garys who were just a bit different that you were always told not to interact with because they’re bad despite the fact that they all seem perfectly nice if a bit odd. Well a certain portion of your community would find themselves torn between the excitement of the exotic (hell you don’t see a Gary every day, and you usually have to go to a certain part of town to see one) and the expectations of not interacting with them, and this would lead to shame and anger, because to an unexamined and immature person these Garys are seducing them, tempting them away from what they know to be the rightful separation of good normal people and Garys. This is even worse if anyone could turn out to have always been a Gary. Maybe some of those people are afraid they might be a Gary but are afraid to even think about it enough to discount it. So they may start calling to ensure there’s no chance of them accidentally running into a Gary and making sure nobody becomes a Gary hoping that that will keep them safe and maybe even punish the Garys who did this to them.