If you don't use a VPN, then ask yourself the following questions:
- How would the connection from your client to your file server be encrypted?
- How would the client and the server mutually authenticate each other?
- How would the keys used to build this encrypted connection be managed on the client and the server?
- How strong is the encryption, and how strong are the keys?
VPN's are used, because they provide reasonably good solutions to all of the above problems.
If your public-facing file server provides acceptable solutions to all of the problems above, then maybe you can consider accessing the server directly, without using a VPN. For example, using key-based authentication with an SSH server may be an acceptable solution in your case.
But even in this case, if the VPN is integrated with a hardware firewall protecting the server, this provides an additional layer of security - because the hardware firewall blocks stray packets from random bots, etc. from even reaching your server. This way, in the event that a vulnerability is discovered in the file server (which would otherwise be public-facing without the VPN/firewall), the server is still protected by the firewall.