I see this type of an argumentation very often and I strongly disagree.

You're removing all responsibility from an actor that is a part of a bigger thing. Imagine if you slapped someone on his hand for doing something wrong, and he or someone else argued what you did is wrong because it wasn't that hand that has offended.

I'm an antitheist but the Bible (gospels) put it well "The student is not above his master" [translation mine] - which means if you follow said master you have to share responsibility for his doings or the doings of the gang as a whole.

From the perspective of the effect, if you make life of an employee miserable, the employee is more likely to resign or ask for a raise, this does apply some pressure.

Moreover, consider what happens if your argument convinces too many people: malevolent actors can just wall themselves with "innocent" people and get away with pretty much anything.