Is it possible to create a CA certificate (even unsigned), which is only allowed to sign certificates for specific limited domain(s), so that it can't be misused for other domains?
(I assume you are talking about certificates for SSL servers.)
Technically no. What would be closest to that would be the
Name Constraints extension (see section 22.214.171.124 of RFC 5280) (OID 126.96.36.199), which theoretically allows for restricting a complete PKI subtree to an explicit set of domains (and subdomains thereof). The extension supports both whitelist and blacklist semantics (in your case, you would like a whitelist). In practice, however, this fails for two reasons:
Name Constraintsextension is mostly unsupported by existing implementations of SSL. They are likely to ignore the extension.
When a SSL client connects to a server, it looks for the server name in the server certificate, as specified in RFC 2818, section 3.1. It will look for names of type
Subject Alt Nameextension, and these names are covered (theoretically) by the
Name Constraints. However, if the server certificate lacks a
Subject Alt Nameextension, clients will fall back on the Common Name (in the
subjectDN). The Common Name is not in scope of the
Name Constraints. This means that a certificate could evade the name constraints by omitting the
Subject Alt Nameextension and putting an arbitrary server name in its Common Name.
(This is the whole story of X.509: lots of hooks and provisions for many useful features, which don't work because of lack of support from implementation and lack of coordination between specification bodies.)
I've found that OpenSSL 1.0.1k and Windows 7 support the extension.
Using XCA, I created a self-signed CA certificate, and added a critical
Name Constraints extension for
.lab.example.com, by adding the following line on the "Advanced" tab during certificate creation:
Note: the constraint should not have a leading dot. It's technically incorrect, but support for this is expanding: https://github.com/golang/go/commit/e4dafa32620e80e4e39937d8e2033fb2ee6085f8
Then, I used that CA certificate to sign two other certificates for HTTPS servers:
bad.google.com- Clearly invalid
Next, after setting up DNS entries accordingly, I used this modified
simple-https-server.py to run an HTTPS server, once with each of the generated certificates:
./simple-https-server --certfile test.lab.example.com.pem --hostname test.lab.example.com
./simple-https-server --certfile bad.google.com.pem --hostname bad.google.com
After installing the CA certificate into the OS trust, I then tried to visit each site with several clients.
OpenSSL 1.0.1k seems to support this.
curl gave me the following error when I tried to visit
curl: (60) The Certifying Authority for this certificate is not permitted to issue a certificate with this name.
Chrome on Windows 7 also does the right thing. Chrome gives a fairly generic
net::ERR_CERT_INVALID, but Windows certificate viewer is quite explicit:
The certificate has an invalid name. The name is not included in the permitted list or is explicitly excluded.
I also tried signing a certificate that did not specify a Subject Alternative Name, instead relying on the old common-name only.
OpenSSL / curl still refused to accept the certificate.
Both Chrome and IE11 on Windows refused to accept the certificate on Windows, even though windows itself (when viewing the server certificate) didn't complain about it. To me, that means that the browsers are doing more than simply asking the OS to verify the certificate, which is a good thing.
However, it appears that name constraints are not supported on OSX.
I feel secure in asking others to install my root CA certificate, without putting them at any risk.