2

I'm dealing with an SSL issue that's been plaguing me for several days. I know relatively little about security, so if anyone here could lend a helping hand I would be extremely grateful.

Ultimately what I'm trying to do is to add a particular Helm repository to a Kubernetes deployment. This has worked fine in my testing environment (AWS), but it's failing in my prod environment (an on-prem cluster provided by a client) due to SSL issues. Specifically, the command I'm running is:

helm repo add polyaxon https://charts.polyaxon.com

And the error I get back is:

Error: Looks like "https://charts.polyaxon.com" is not a valid chart repository 
or cannot be reached: Get https://charts.polyaxon.com/index.yaml: 
x509: certificate signed by unknown authority

This is confusing to me because:

  1. I do not get this error when working locally or on AWS.
  2. When I go to the above url in my browser, it appears to have a valid cert issued by COMODO.
  3. Polyaxon is a fairly popular project, and this is a basic step in using it. And yet I can't find a single instance of this happening to anyone else.

Attempting to access the repo via wget yields a more helpful error message:

:~$ wget https://charts.polyaxon.com
--2019-02-22 22:02:53--  https://charts.polyaxon.com/
Resolving charts.polyaxon.com (charts.polyaxon.com)... 104.27.149.134, 104.27.148.134, 2606:4700:30::681b:9586, ...
Connecting to charts.polyaxon.com (charts.polyaxon.com)|104.27.149.134|:443... connected.
ERROR: cannot verify charts.polyaxon.com's certificate, issued by 
\[email protected],CN=FGT37D4615801045,OU=Certificate Authority,O=Fortinet,L=Sunnyvale,ST=California,C=US\u2019:
  Self-signed certificate encountered.
To connect to charts.polyaxon.com insecurely, use `--no-check-certificate'.

What's interesting/confusing about this is that, as best I can tell, it's saying that the cert is issued by "Fortinet" and not by COMODO. I had never heard of Fortinet, so I googled them and discovered that they're not in the business of issuing SSL certs but instead of providing heavy-duty firewalls and the like. I also came across links like this and this, which seemed like they might be related.

So at this point my working theory is that the client has installed some Fortinet product on the network and that this product is causing an opaque SSL error that I wouldn't see elsewhere. But again, I'm way out of my depth here. Good chance I'm wrong, and even if I'm right, I still wouldn't know how to fix the issue.

So with that said, my questions are:

  1. What do you think is going on?
  2. How do I fix it?

Note that, if it's helpful, I can explicitly pass certs to helm via the options here.

Improve this question

3 Answers 3

Reset to default
2

The first thing is to communicate with your client: ask if they have a Fortinet appliance that is configured for SSL inspection on purpose. This is the only way to distinguish this from a genuine man-in-the-middle (MITM) attack, as anyone could make a self-signed CA that appears as a Fortinet appliance.

After that, you have the following choices, but this should be discussed closely with your client, comparing pros & cons regarding both security and maintainability.

  1. Add an exception for your entire server to bypass the transparent HTTPS proxy. While the SSL inspection protects the users inside the organization while they surf freely on the Internet, a server should only make connections to trusted web sites.

  2. Add exceptions for the sites the server needs. This is more restrictive: it may add some security, but also needs more maintenance as you might occasionally need to ask for more exceptions.

  3. Ask your client for the public CA certificate of the Fortinet appliance and install it to this server's trusted root certificate authorities. This has its risks it the private key for the appliance gets compromised. Also, you need to do this again if the client upgrades or replaces the appliance with another. At that point you must remember to remove the old CA certificate, too.

They should know if they are performing SSL inspection, as they must have done the same for all the client computers. In some countries they would also need a consent from their employees.

Improve this answer
2
  • Thanks very much for this helpful comment. One thing that's still not clear to me is why this would happen selectively to the Polyaxon url. I'm able to wget other resources and even add other helm repos without an issue. If I'm understanding SSL Inspection properly, it would seem that the Fortinet cert would be generated and thus cause the error in each case. So is it possible/common to perform SSL Inspection selectively, and if so, what would trigger it for one site and not another?
    – JJL
    Commented Feb 23, 2019 at 13:20
  • It might well be that all the other repositories are retrieved using plain HTTP. Transparent proxy for plain HTTP is... transparent, while an HTTPS proxy is always intercepting the TLS connection. My guess is that's why it's the only connection that fails for this reason. I'm not that familiar with Helm, but e.g. Debian repositories don't necessarily need TLS encrypted connections, because the packages are PGP signed: if they are tampered during the download, the signatures won't match.
    – Esa Jokinen
    Commented Feb 23, 2019 at 15:34
0

Some Fortinet appliance is performing what amounts to a Man-in-the-Middle (MITM) attack or, in the context of firewalls and network-based intrusion detection systems, TLS inspection. This is done to allow the firewall to inspect encrypted traffic that would otherwise be completely opaque to it. Because the certificate used by the Fortinet device is not trusted by the client, the client (rightfully) refuses to accept the certificate and throws an error during the TLS handshake. You have two solutions to this issue:

  1. Disable TLS inspection on the firewall device.
  2. Set the firewall's root certificate as trusted on all relevant clients.

If the firewall is unmaintained by your client which very well may be possible if they did not even know it was stripping encrypted connections, then I would be weary with installing and trusting the device's root certificate. If the firewall is ever compromised or the certificate was generated incorrectly, which is more likely if the firewall's software is outdated or the device is not actively maintained, it could become very possible for an attacker to perform genuine MITM attacks against any device on the network.

Improve this answer
0

Fortinet is a vendor of firewalls and these firewalls also support TLS interception, i.e. analyzing the traffic by decrypting it and encrypting it again. This essentially makes of one single end-to-end TLS connections from client to server one TLS connection from firewall to server and another from client to firewall. Due to how TLS works the connection from client to firewall cannot have the original certificate from the server but has a new one which got dynamically issued by a certificate authority (CA) on the firewall. For more details see How does SSL Proxy server in company work? for more information.

In order to fix this you need to either have an exception at the firewall so that no interception is done for specific targets. Or you would need to import the CA of the firewall as trusted into your infrastructure. In both cases you need to contact the local system administrators managing the firewall, i.e. either to get the exception or to get the CA certificate which you can then import.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .