There is more than enough land for everyone, and rich people aren't really competing for the kind of housing that poor people are competing for, e.g. smaller plots with smaller homes. The demand of the rich does not eliminate demand of the poor, so the market produces different kinds of housing for different clientele.

Think about it this way: assume you supply all the housing to all the rich people. Then there still remains untapped demand of others that can be fulfilled by further production of homes for those specific people.

This story fails when land becomes restricted, which is exactly what zoning laws cause. Zoning is a big harm to the poor.