I am generating RSA keys with the openssl library. I have heard that it is possible to recognise the method used to encrypt a string with AES or DES but is it possible to identify whether a string is a RSA public key?
Yes. RSA public keys are typically generated as ASN.1 structures as per the definition in RFC3447, encoded in DER format.
The format can be validated as correct DER / ASN.1, and then decoded to validate that the structure and contained values match what is expected from an RSA public key. The values within the structure are simply the public semiprime (n) and the exponent (e).
By validating that the value of e is prime, and that the size of n matches an expected RSA key size (e.g. 1024 bits) then you can assume that the provided data was in fact an RSA public key.
For the case of OpenSSH, public keys start with a string identifier (e.g. "ssh-rsa") which makes detection easy. You can also extract and validate the n and e values in the same way as above, albeit using the OpenSSH key format specified in RFC4253 instead of the DER / ASN.1 format.