Self-signing client certificates and distributing them, is the following a sound procedure?

Question

I have a case at hand as follows:

1. There is a number of clients in Internet (i.e. untrusted channel), initially in hundreds but growing in numbers.
2. There is a server doing processing related to these clients. This relation is established prior to this elsewhere, say a web portal.
3. The clients should be able to connect to the server via a piece of software (non-human operated, e.g. not a browser) over the internet to load the material so that channel is encrypted and that the server knows which client is connected.
4. There should not be an issue the clients could claim they have not requested the material when a connection has been made and the material has been transmitted. As an edit related to Thomas's question about not receiving material when they did: if I were to use plain password here, would there be a danger the password leaking more easily than the cert and someone else using it to obtain the material to be transferred? Though, I've thought about mitigatiging against such a situation by using this "sequence id" (see point 6 in the following list), which probably helps with or without certificates.
5. The client never accepts incoming connections. It only initiates them to this well-known server.
6. The ones operating the clients usually do not have their own certificates issued by some certificate authority.
7. The server processor program has a certificate issued by a certificate authority (e.g. Thawte).

Question: What would be a good way to accomplish this in a secure and a maintainable way?

As far as I know, there isn’t any PKI software that I can use. I've planned the following:

1. Let the clients download the piece of transfer software, say from the portal. The portal can include a settings file having a client id of some sort in it (and a, say, one-time password see point 4).
2. When the client installs the transfer software on the server it generates a self-signed certificate with the client ID in the CN. For instance makecert -r -n "CN=SomeClientId" -pe -ss my.
3. This software sends the thumbprint of the generated certificate to the server along with the client id and uses a one-time password it has obtained elsewhere (see point 1).
4. The server stores the thumbprint along with the client ID.
5. Now when transfer software connects to the server, it can check to which client it belongs via the transmitted thumbprint and the channel is secured.
6. For added security the server maintains a sequence number that needs to match with that of a respective transfer client. If they get out of sync, a breach is suspected and the very least the client needs both to go to the portal to reset the counter and reset in the transfer software (e.g. set 0 in the settings file).

Does this look like a sound arrangement? Would there be a better one? It looks like that if access of certain client needs to denied, all that is needed is to set a flag in the server database to indicate a certain thumbprint is not allowed anymore. Whilst browsing through SE, I came across How to distribute client certificates without exposing private key?. Taking the cue from Thomas Pornin's answer, I see I could alter this process so that upon generating the certificate. As for an example:

1. Create someclient.inf file with appropriate parameters (in Windows, otherwise in other OSes, e.g. using openssl).
2. Do certreq -new someclient.inf request.csr to create the request file.
3. Send the csr to the server using the one-time password.
4. The server receives and csr file and creates the certificate, saves the thumbprint and sends the certificate back.
5. The client optionally installs the certificate with certreq -accept someclient.cer.

Another question: Would this bring additional benefits beyond the procedure I outlined initially (and is this a sound procedure)?

Additiona questions: It's unclear me as to how the server should do the signing. Which program? With which parameters? But maybe these should be another questions if this procedure is a usable one.

As a note: it could be I don't have access to all of the software on the command line or writing the code gets compplicated on the client end, so perhaps I'll need to resort to something like a BouncyCastle.

Answer 1

Your point 4 is a bit unclear: is it a problem if clients start to claim that they did not receive the "material", while in fact they did ?

If it is not a problem, then what you basically need is:

• A SSL connection to your server. The server's certificate must be recognizable as valid by the clients. Since the client software is under your control, you don't need a certificate from an external commercial CA (like Thawte); what you need is a certificate that will be accepted by the client software. You could make your own self-signed certificate for the server, as long as you embed a copy of that certificate in the client code, and the client checks that the certificate sent by the server is bit-to-bit equal to the expected one.

• Some client-specific secret value S stored on the client side. This value can be thought as a kind of password, except that since no human brain is involved in the operation, it can be strong -- say, an arbitrary 128-bit string, generated randomly and uniformly. The client stores S and the server stores h(S) (for some hash function h()).

• When the client connects, it sends S to the server through the SSL connection. The server computes h(S) and looks it up in its database, to know which client is connected.

There is no need to add a client-side certificate in this model. You can, but that does not bring extra benefits to the security model. One good reason to use client-side certificates is to interoperate with hardware elements designed for certificates (smart cards); as long as you remain on the software side of things, a simple secret value will do the trick (in any case, the client has to store some secret value, e.g. its private key if it owns a certificate).