I am reviewing the security of an embedded system, specifically how it uses the TLS or DTLS protocol to communicate securely. The system implements as few features of the protocol as it can get away with. It doesn't deviate from the protocol as such, but it may not always be strictly compliant with all applicable RFC. My objective is to determine whether the choice of security parameters are safe.
I will have to do this again for other similar systems, hence my question is somewhat general rather than being about a specific system.
The security objectives of using (D)TLS are the usual one: an attacker could intercept network traffic (active man-in-the-middle). We want to ensure that the client is talking to the server it thinks it's talking to and that a third party cannot obtain or modify transmitted data (confidentiality and integrity of traffic on the connection). Client authentication is sometimes desired as well.
These are embedded (IoT) systems with very few resources, and usually interacting in controlled ecosystems. At least, we control the legitimate peers, so compatibility with diverse antique clients and servers is not an issue — the concerns are very different from using TLS on the web. A typical endpoint client only speaks one or two ciphersuites (“classic” signature-based server authentication, pre-shared key, or one of each). Depending on the scenario, the PKI may or may not be a closed system: some devices live in a world where we can trust CAs to only deliver certificates of a certain form, while others may trust a “public web” CA.
These are devices with stringent constraints on available computing power, RAM size and code size. So they're generally running stripped down TLS libraries with as few features as possible. I'm talking microcontroller with 128–512kB of RAM, not higher-end devices running Linux.
Are there (D)TLS or X.509 extensions that are important? I'm not concerned with the choice of ciphersuites (nobody uses DES, RC4 or MD5 in my world, or ciphersuites without server authentication) or with key sizes. What I'm worried about is, for example, an implementation that skips checks on certificates (e.g. key usage restriction) which would allow a certificate to be misused. Or a missing check during the handshake that allow some kind of downgrade attack (an endpoint device usually only speaks a single protocol version, but the servers it talks to may support multiple versions). Or a missing extension that allows an active MitM to convince the two parties to settle on mismatched security parameters, or to renegotiate insecurely.
What should I be looking for when reviewing the TLS configuration on a small embedded device?