Those who say this are usually frustrated by the increasing number of misunderstandings/miscommunications due to increasing English illiteracy. Its become more difficult to communicate to those around you.

It’s utterly unacceptable behaviour but I believe the issue lies deeper than simple “racism”. I also sometimes find it frustrating from the sheer volume of people that can’t speak English, from coworkers, customers, fellow students, etc. I don’t even live near the border, where the problem is much worse.