However on social networks like GNU Social, diaspora, etc I agree - they're just small servers. Right now tough to setup and maintain. In the future probably someone could make it as easy as click and install.
Also concerns about privacy apply here - say if you have a server and your friends are in it, what would prevent you from spying on them and violating their privacy? (a question on this regard was asked by a friend).

But is end to end social networking possible?