The SSH port can be set to just accept connections from within the VPN.
However, what I meant is: VPN does allow for more than SSH. Let’s assume something like you allowed your girlfriends phone to use your wifi, but she uses an app with a Chinese backdoor. The Chinese hacked your network printer which is available to all using the wifi. Your linux CUPS printing service talks to the printer and gets infected with a worm, but being linux it’s confined within the things the cups user can access.
At that point the attacker/worm has no access to your personal files yet, except for what you print. Nor does the attacker/worm know about your server.
Now when you use just SSH it will likely stay that way.
If you use VPN though, it will allow the worm/attacker to find out about the existence of the server and send network traffic to your server. Hopefully, that doesn’t get them far, but it’s an additional attack vector they get.