I could distribute the RADIUS' server certificate to all clients. But this would counteract the whole reason I want to implement option B.
Let me start by saying it isn't the RADIUS server's certificates you need to distribute, it is the certificate of the CA that signed the RADIUS server certificate. I will get back to this a bit later in the answer.
if I use a self-signed certificate, the user has to manually accept this self signed certificate
Some supplicants won't even accept a self signed server certificate, unless certificate validation is disabled.
But what if I use a certificate signed by a CA (like Let's Encrypt, or any other CA which can be found on most devices) for the RADIUS server? Will the client accept it automatically?
Depends on the supplicant. Some will, others won't. Without some sort of additional configuration (manual, GPO, MDM, etc), I would even go out on a limb to say that vast majority of them will still prompt the user to accept the certificate.
What stops an attacker to go to the same public CA and obtain a valid certificate for himself? (Is there a way to specify to accept only certificates for a certain domain?)
Same public CA? Pick one that does a reasonable attempt to validate you are the domain owner before generating a certificate.
However, that is beside the point as a valid certificate from any major CA will often provide the same "assurance" (if the EAP supplicant uses any known valid CA) and it may be easier for an attacker to get a valid certificate from a different CA from the one where you get yours (if your CA already has legitimate contacts for your organization, it is likely they will use those to validate the request).
Are the answers to the previous questions valid for all 802.1x methods that provide server authentication?
It sounds like a brief primer on what is taking place with 802.1X could be helpful.
Generally speaking, the EAP supplicant (no matter the EAP method chosen) must make a decision on whether it will send your username/password or other credentials to the authentication server without being connected to the network. This means it is only able to work with information it has prior to the connection and information it is given by the authentication server. The EAP supplicant can't go out on the Internet to check another source until after it has authenticated and it is too late to withhold your credentials.
The EAP supplicant does this primarily by checking two things (with a couple options present in many EAP supplicants like the Windows and OSX native supplicants, but often not on mobile devices like tablets and phones):
- Is the certificate provided by the authentication server a valid certificate issued from a CA trusted by the client device?
- (Optional) Is the certificate issued from the designated CA(s) in the EAP supplicant?
- Does the hostname listed on the certificate match the hostname given by the authentication server?
- (Optional) Is the hostname given by the authentication server one of the hostnames allowed in the configuration of the EAP supplicant?
The supplicant may also prompt the user to approve the certificate if it passes 1 and 3 above. Or the EAP supplicant may skip the above and have the OS prompt the user to "check and approve" the certificate (bad idea), or it may be configured to not validate the certificate at all (worse idea) before sending your credentials.
Since a) requires distributing a certificate to the users, I want to go with option B.
Distributing some sort of certificate may be unavoidable. Often it is simply a matter of if you are distributing client certificates as well. (If you do choose an EAP method with client certificates, such as EAP-TLS, I highly recommend individual client certificates per user/device rather than a shared client certificate.)
Many organizations will set up their own CA to generate their RADIUS server certificates (and client certificates where applicable). This does require installing the CA certificate on the clients as well.
Why would they do this? Here are a few reasons:
- Cost - getting certificates that provide for authentication service (not just HTTPS web traffic) could be expensive, especially if the organization runs a large number of authentication servers (as generally you need a certificate per server).
- Reduced end user impact - most CAs will only issue certificates for a maximum of three years (and some now only two). When the certificate on a wireless authentication server changes, at the very least the end user will be notified of the change and prompted to accept the new certificate. In some cases, the EAP supplicant will simply fail to connect to the wireless network until reconfigured. Organizations that run their own CA can choose to issue certificates with expiration further out than 3 years, reducing how often their end users are impacted by a certificate change.
- Increased security - if the organization uses a public CA to issue their authentication certificates, then an "attacker" will find it easier to "impersonate" the organization's network. It is trivial (and in some cases entirely free) for an attacker to get a valid certificate from a public CA that can match the hostname of the attacker's own "authentication server." If specific hostnames for the authentication servers are not configured in the EAP supplicant (and they often are not or can't be), then even if the designated CA is configured and is a public CA, the attack just needs to get their certificate from the same public CA. If only the organization's CA is configured, the attacker would have to compromise the CA and generate a certificate there as no other certificate would be accepted.
how can I mitigate the rouge AP attack?
No matter how your approach it, while 802.1X adds a higher level of security to your wireless connections, it does so at the cost of making secure configuration of the client devices more difficult.
If you are only concerned about Windows in an Active Directory environment, then you can simply use group policies to enforce the correct configuration and install any certificates.
Otherwise (and who worries only about Windows today?), the best methods for approaching this issue are to use one of many 802.1X on-boarding solutions that can configure the majority of your clients for the best level of security. These do however cost to acquire and/or operate and most on the products on the market are now bundled as part of a CA, NAC and/or RADIUS solution.
Solutions such as Cisco ISE, SecureW2, Ruckus Cloudpath, Aruba Clearpath/QuickConnect, and Impulse SafeConnect are some of the solutions I have seen deployed for this purpose. Or simply use your search engine of choice to look for "802.1X onboarding" or "BYOD onboarding."
Note: portions of this answer were derived from my own answers elsewhere on this or other sites.