So, to start, let's look at the architecture. Many VPS providers aren't good about this, but if you can communicate between your VPS' on an internal network using private IPs, that's preferable. Digital Ocean will let you do private networking and accomplish a lot of the inter-connectivity without traversing the public Internet. If you can do this, you'll remove a ton of attack surface by limiting potential MitM attacks and making IP spoofing untenable (or at least less-tenable) from the Internet.
Next you would want to avoid having to expose the API server to the public Internet (or at least I would). The kubernetes API server is fairly robust and uses reasonable authentication, but it presents a large attack surface and potentially large damage if it's compromised. So I'd recommend having a Bastion Host or VPN server that allows you to connect into the kubernetes environment where you can administer your cluster. Just don't store any authentication mechanisms on this bastion hosts, make it a dumb jump-box in case it's compromised.
To avoid exposing all of the nodeport services, you can use a Load Balancer, if your VPS provider has one, probably you should use that, otherwise a dedicated nginx box would be best. However, for tighter integration, you might expose 443 on each of your worker nodes and use the nginx-ingress controller on the cluster itself, this way you can use a lot of the built-in integrations, but still control what services are externally exposed.
Once you have those two things taken care of, you could probably get away with just allowing all traffic between your kubernetes nodes within the cluster. But it would be better to batten the hatches as it were.
Let's define the servers/VPS into the following groups
- Etcd nodes
- API Servers
- Control plane (all of the Etc nodes, schedulers, controllers, api servers)
- Worker nodes
- Bastion Host
- Load Balancer
you would want to implement the following rules using UFW and (if it exists) your VPS' security groups/cloud firewalls/etc. These rules assume a stateful firewall like UFW or AWS Security Groups. If you're using AWS Security groups, you can have named security groups for each of the above groups and have them reference each other and not specify IP addresses at all. Using UFW, you will have to list IP addresses.
- Control Plane and Bastion Host -> API servers over TCP/6443
- Etcd and API Servers -> Etcd Nodes over TCP/2379-2380
- Load Balancer and Worker nodes and Control Plane -> Worker nodes over TCP/30000-32767
- Control Plane -> Control Plane and Worker nodes over TCP/10250
- Control Plane -> Control Plane over TCP/10251-10252
This could be broken down further, but I think this is a reasonable configuration. I don't think the Control Plane needs access to the nodeport nodes, I don't think the etcd cluster needs access to the kubelet, and so on. Proceed down the rabbit hole as far as you like. To lighten up the security posture you can expose the nodeports and the API server to the world, just make sure you understand the ramifications of doing so.
Attacks against this architecture
As you mention, if your cluster is talking over the public Internet, anyone can identify themself as a whitelisted IP address. Suppose an attacker sends a request to Etcd at 120.88.41.177 and says that their IP is 120.88.41.176, but their real IP is 120.88.41.175. The request will bypass UFW just fine, but the server will also send the reply to 120.88.41.176 and not to the attacker. Unless the attacker can intercept the traffic, the attacker never can see any of the replies.
Also, for this to work, the IP addresses of your servers would have to be known to the attacker (which may or may not be a tall order).
Next, the cluster-level communications (Between the Kubelets, API servers, Etcd cluster, controllers, and schedulers) all run over HTTPS by default, using client-side TLS authentication. (EDIT It looks like Etcd with authentication is not support by default as of k8s 1.17, so limiting network access to Etcd from ONLY the API server(s) is very important). So in order for the attacker to communicate with anything within your cluster, they also have to compromise your CA or a client key somewhere in your infrastructure. At which point, if an attacker compromises a client key, they probably have some access to one of your nodes and the network security is moot.
This is why I would suggest some variation of the above suggestions for your network, and then focus your efforts on good RBAC controls to limit what actions each pod can do, and limit what actions each node can do. Or add some network policies within kubernetes to limit the access of the pods. At the host/VPS level, the networking is pretty straightforward, the API server has to be reachable by all of your nodes, so you can't really protect against a compromised node/pod using UFW.
