I'll answer as developer and designer of some additions to Crypto-API's.
First of all, Poncho is correct that no specification of AES or mode of operation (such as the insecure ECB) will indicate what to do in this case. They will almost certainly not propose #2 or #3, if just because that would severely upset any security proof of the algorithm.
1. returning an error ?
The only correct answer in my opinion. AES only accepts 128, 192 or 256 byte keys. This is called a pre-condition. Failing to comply with a pre-condition should result in a failure. In Java I would quickly upgrade it to a
RuntimeException as providing fewer bits for a symmetric key is always a programming error (you don't expect a user to directly enter a key, and if a hexadecimal key does need to be entered, you need to verify it before offering it to the crypto API).
The only place where this might not be the best answer is very low level API's where just a pointer to a key with known size is returned. But users should never call or even have access to that low level API.
Returning an error (or exception, for higher level languages) is also fail-fast, which is a good security design practice.
2. truncating the key to 128-bit ?
No that's horrible. You should not perform a cryptographic function with less security relevant data than the user provided. You're would be deciding for the user that less security is acceptable, something that obviously is not the role of an API designer.
Extending the key with zero's is equally horrible. You're telling your user that they are using AES-256 while the actual security offered is lower than that (and possibly even next to zero).
Just like (3) this will also fail the principle of least surprise.
The PHP mcrypt based functions take this route and there is a never ending stream of questions on SO trying to replicate mcrypt functionality in well behaved crypto API's.
3. treat the key as password and use some KDF function?
No. If the user wants to use a KDF, then offer them a KDF. Performing a KDF en a mode of encryption for AES violates all kind of design principles, including the principle of least surprise. An AES API should just do that: offer a cipher interface to obtain confidentiality. An API could make it easy to use a KDF, but it should not itself choose a KDF and set of parameters and make that a default (which at that time seems secure, and can never be changed afterwards).
There are many users of the CryptoJS API that mistake the encryption routine that uses a password with the one that uses a key, just because they have the same name.
- Some Crypto API's like Java do not directly accept a binary key. The key must first be generated from the bytes. One reason to not directly use bytes as key is that hardware devices (HSM's, smart cards) may not even expose the bytes to the software in the first place. In that case the error may be thrown in advance, depending on the implementation.