Being a community admin isn't easy. You'll feel bad sometimes. It feels much easier, at least in the short term, to let trolls and arseholes and people who are making themselves miserable just kinda stick around, and hope they'll leave.
You've pretty much got to deal with occasionally feeling awful. If it helps, remember this: although to you, handing down a ban can feel like giving a death sentence, to the person you're banning, it just means they'll have to look at a different website.