You're allowed to say no to (and protect yourself from) people who cause harm and refuse to make genuine changes. As you evolve, some people will stay stuck where they are. They don't want to put in the effort to join you where you're going, and instead want to keep you small, because they've benefitted from that.

Remember: they may not want to ride your train, but they don't get to pull it off the tracks. Focus on where you're going. Don't let them derail your progress.