Risks with OpenSSL verifying a signature with un-trusted PEM encoded public key

Question

If a website user wants to use WebAuthn, they will start by creating a credential, where their authentication device provides a public key.

This key is encoded, and sent back to the server to store against their account.

Later, when the user needs to be verified, the website provides a challenge, their authentication device signs it, and that's sent back to the server.

Assuming PEM encoding of the public key (originally sent to the server from a potentially hostile user), are there any risks with this?

• Is it possible to cause a Denial of Service?
• Could it cause OpenSSL/PHP/Apache to crash?
• Use up too much memory, or take a long time to process?
• Provide a key format that OpenSSL does not understand, or get confused by, and return an unexpected result?

Some example questions I'm unsure about:

• If it's an Elliptic Curve (e.g. prime256v1), could it include excessively large x/y values?
• Is there a problem if a DSA key was provided?
• Is there a problem with an RSA key using PKCS1v1.5 padding?
• While RSA is normally 2048-bits, what happens if it's a 65,536 bit key?
• What about invalid DER encoding (e.g. wrong field lengths)?
• How about an invalid DER Object Identifier?
• Could invalid base64 encoding of the PEM data cause issues for OpenSSL?

I know a normal user would not do any of these things, and I accept that anyone who does provide a flawed public key would be affecting their own account, but could it cause other problems?

---

This is a basic implementation in PHP:

<?php

// PEM encoded public key, from hostile user
$key = '-----BEGIN PUBLIC KEY----- [...] -----END PUBLIC KEY-----';

// Other checks

$verify  = base64_decode($response['authenticatorData']);
$verify .= hash('sha256', base64_decode($response['clientDataJSON']), true);

$signature = base64_decode($response['signature']);

if (openssl_verify($verify, $signature, $key, OPENSSL_ALGO_SHA256) === 1) {
    // Success
}

?>

Note how openssl_verify() takes 3 values that came from the user.

And I could use openssl_pkey_get_details() to check the type, curve_name/oid, and x/y values.

---

In short, should the server be doing any additional checks on the public key?

Answer 1

It seems like you're question boils down to two concerns: Can a logic flaw in your code be exploited? And Is the process efficient enough to not fall over under mild load? The concept itself doesn't seem unreasonable for a web server to accomplish, but if you're worried about the processing load, you can have the web server and php processor reside on different servers (or even put them on clusters of servers, with load balancers, queues, and caching layers, there's a real bottomless pit in the pursuit of high availability). But bear in mind that avoiding DoSs and app server crashes is a hard problem. The best we can do is make it hard/expensive for attackers to knock your server(s) over.

Limit the size of POST requests

Have the web server limit the size of post requests to limit the impact a single request can make. Here's an example using httpd .htaccess file

Check return codes

I see you're checking openssl for errors, you should also check the return code for base64 decode and make sure that operation succeeds, before passing garbage further on in the pipeline.

String size

I'm less familiar with how to do this in php, but setting an explicit max size and throwing errors if the input is larger or only handling a truncated version of long inputs. But if a malicious user sends a 1GB block of text, at a high rate, and you are running expensive hashing operations, your server might get knocked over. But if you curtail it to a max size of 100kb before sending it to be hashed, you'll be doing better.

Regular Expressions

I'm a proponent of running user input against a regular expression before processing it (i.e. making sure the supposed base64 data only contains [A-Za-z0-9-/\=], though you may have to URL-decode it first. A regex expects to to be passed arbitrary data and most regex libraries can handle binary data, however, a base64 decoder might not expect characters outside of the 64 character set. However, a regex engine could also be compromised, so your mileage may vary. Also, worth considering, regexes are expensive in terms of memory and CPU, so it may actually make it so that your application can handle less requests.

I wouldn't bother doing much validation of the blobs that come out of the base64 decode. Openssl probably has a safer and more correct implementation that anything you homebrew, so long as you...

Patch your dependencies

The openssl library, the base64 library, and anything else in the critical path of parsing user input should be on an aggressive patch schedule (balancing the risk of downtime if you deploy without adequate testing). This includes the php engine, web server, and the operating system kernel. Because most known vulnerabilities will be patch within days, you'll only have to worry about zero-days, which is largely out of your control.

Infrastructure Layer redundancy

External to your question, you could also use a CDN, load balancer, or a WAF to protect against certain attacks or floods. Having a dedicated SSL Terminator can reduce the load on a webserver. Also, rearchitecting your application to operate in independent tiers (or tiers of clusters) can limit the blast radius of a crash and can help you address load. Using a geolocation-based sharding/routing algorithm in a distributed web app can make it so an attacker from a single IP could only DoS other users routed to that shard, so you can ensure, for example, that 67% of users will be up if you have three shards. Botnets can sometimes be addressed by WAFs. This is a huge rabbit hole that probably spans beyond the scope of your question, but hopefully gives you some ideas of directions to look.