Is HTTPS required for local network server to server communication

Question

I am building web applications for my customer's company. At the server side, there will be 2 kinds of server to server network communication.

1. Separated REST API servers making requests among each other.
2. Communication from application load balancers (AWS ALB specifically) to their auto-scaling EC2 instances.

Currently all of these communications use HTTP protocol. Only the user-facing nodes (such as the load balancer or the web server reverse proxy) will serve HTTPS with valid certificates.

The customer ask us to change them all to HTTPS as thet believe that it is the modern best practice to always use HTTPS instead of HTTP anywhere.

I would like to dispute with the customer but I am no security expert. Please help review my understanding below and correct me if I am wrong.

---

In my view, I think the purpose of HTTPS protocol is for being a trusted channel in an untrusted environment (such as the Internet). So I cannot see any benefit of changing the already trusted channel to HTTPS. Further more, having to install certificates to all servers make it difficult to maintain, chances are, the customer will find their application servers broken someday in the future because some server has certificate expired and no one know.

Another problem, if we have to config all the application server, apache for example, behind the load balance to serve HTTPS, then what is the ServerName to put inside the VirtualHost? Currently we have no problem using the domain name such as my-website.example.com for HTTP VirtualHost. But if it were to be HTTPS we have to install certificate of my-website.example.com to all instances behind the load-balancer? I think it's weird because then we have many servers claiming to be my-website.example.com.

Answer 1

The answer to your question comes down to threat modeling. Using cryptographic protocols like HTTPS is a security mechanism to protect against certain threats. If those threats are relevant for you, must be analyzed:

• Are there potential threat actors in your internal network? Based on your question you seem to assume that the internal network can be fully trusted. This is often a misconception, because there are several ways your internal network can be compromised (e.g. valid users with access to this network are turning malicious, a systems in this network gets compromised, a misconfiguration opens up the network segment, etc.).
• Will the architecture be subject to change? It is likely that the system will change over time and prior security assumptions (e.g. my internal network is trusted) no longer hold. If that's a reasonable scenario, it might be a good idea to build the necessary security mechanism in in advance. That's what security best-practices are for. Providing security in an area of uncertainty.
• Is there a regulatory, legal or compliance requirement that must be fulfilled? You said that your customer considers HTTPS to be state-of-the-art / modern best-practice. The source of this friendly worded statement might actually be an externally driven requirement, that must be fulfilled. Non-compliance is a threat that should also be covered in a threat analysis.

Those are important topics worth analyzing. When I design system architectures and I am in doubt, I prefer to err on the side of security. In this case the best-practice approach is indeed using HTTPS for communication, no matter the circumstances, as long as there are isn't a considerable impact on the application (e.g. performance impact).

Difficulty to maintain server certificates shouldn't be a problem nowadays, as this is common practice. This should be part of normal scheduled operations activity.

Having said all this, there is of course additional effort required to use HTTPS instead of HTTP and it is your right to charge the customer for this additional effort. I suggest you calculate what this will cost during development and over time during operation and let the customer decide if the cost is worth the benefit.

Answer 2

Mixing and matching HTTP and HTTPS is not a good idea - you will constantly be juggling configurations.

Usually adding a component into a system should only be done if there is a very specific reason for it - just because someone thought it a good idea is not a specific reason.

I'm not saying that HTTPS is a bad idea - quite the opposite - but you have a lot of learning to do. The model you propose undermines the trust relationship that is the primary reason for using TLS in the first place. You also don't seem to have thought about how to plan your PKI.

servers broken someday in the future because some server has certificate expired and no one know

If you are providing the service, then you should be configuring monitoring for the service, including certificate expiry.

It sounds like you are looking for reasons to argue with the approach of rolling out certificates. Reading between the lines here, it seems you are currently lacking the skills and planning you need to implement this.

Yes, it's a lot of work, but that's the business model - you assess the amount of work, the skills you need to acquire and those you can buy in and you charge the customer for that. (Serge highlights the cost of the certificates - but that is the smallest cost in this whole exercise).

Answer 3

Internal networks are not secure

In general, internal networks are more secure than public-facing systems, but they should not be considered as completely secure. A significant portion of attacks come from the inside - spearphishing, social engineering and insider attacks are all popular vectors which start with a foothold inside of your network.

So there's no good reason for unencrypted traffic of secret or private information even over your internal networks. You don't necessarily need public names or CA hierarchy - if you have well-defined bilateral communications channels, then it may be simpler to have an explicit trust relationship where your load-balancers are configured to trust a particular self-signed certificate of your backend servers and nothing else.

Answer 4

As a professional, you owe advises to your client, but should not take the decision yourself.

The arguments to present to your client are:

• what is the gain in using HTTPS inside the server network? If this network is isolated from any other system and only sysadmins can access it, you may argue the the gain can be neglected because it is just protecting a system against someone which already has admin priviledges. If other staff members with no admin priviledges can access it, the gain is not null, neither it is is systems for other clients can.
• what is the risk to do it? The disgression on certificates is mainly... a disgression. But the fact is that HTTPS is a more complex protocol that HTTP is, and any added complexity adds risks for implementation errors. If the previous step concluded that the gain is neglectable, that is enough to advise the client not to do.
• what would be the added cost? Here you have to considere direct and indirect costs. Direct costs could include the price of additional certificates if you use external ones, or the time for creation of private certificates if you use a private PKI. They would also include the time for the configuration of the system. And they should include the maintenance times as recurrent costs, including a programmed renewal of certificates - this part is in you responsability domain, but you can charge your client for the time. Indirect costs are harder to establish but you should use your own experience to evaluate the risk of errors due the the added complexity and their possible consequences. And IMHO you may charge your client for that if they insist in not following your advices.

But when you have said that all, the client is responsable for the decision.

Answer 5

Encryption is cheap. Data leak or data loss are not.

Use encryption between servers (and it is even better to use TLS auth between servers).

And when I say cheap, it is cheap even considering the management of the keys and the certificates. It may be reasonable to issue a self-signed and long-lived certs to both servers.

There are a few exceptions to the rule:

1. Either the client or the server are legacy having known SSL/TLS vulnerabilities. It is always better to update the vulnerable code, but we all know it is not always possible. It is sometimes better (still not good, but better) to disable the vulnerable code entirely, run plaintext and mitigate the risks in one way or another.

2. You are exchanging an insane amount of data and/or need an insanely low latency. The encryption may become a bottleneck and/or a resource hog. You may opt to no encryption and also do something else to secure the thing.

Answer 6

"https" not only secures the communications as it passes over the network, but also verifies the certificate that is presented by the server. This enables you to know that you really are talking to the correct site. And this, in my opinion, is the real advantage of "https."

"letsencrypt.org," which issues signed certificates for free, has a lot of material on their web-site which discusses these benefits. They argue ... quite rightly, I think ... that "everything should be https, whether the material is actually sensitive or not."

(mod_ssl et al are also capable of enforcing certificate possession on the client side, too, although this is rarely done. For a secure internal application, however, you might want to do such a thing. The server is then able to restrict which computers are capable of connecting to it at all, restricting them by the credentials they must possess.)