Nobody will care if the conference isn’t held in Philly. Holding it elsewhere will probably make it a little easier and possibly a little cheaper for people to attend. I doubt mathematicians are part of the 1%, so cash and travel hassle should matter. And given today’s Internet, there’s going to be remote attendance which can happen most anywhere.

While it’s still convenient to gather together to discuss a field, it’s not crucial as it was in past times. Easier to do what’s best for the largest number of people.