I have cryptographic code used in the Windows platform, which uses the Crypto API functions and need to convert this to using Common Crypto on OS X.

Essentially the original code is this, with error checking removed for brevity: -

CryptCreateHash(m_hProv, CALG_MD5 ,0, 0, &hHash);
CryptHashData(hHash,(LPBYTE)pszInputData, lstrlen(pszInputData)*sizeof(TCHAR), 0);

CryptDeriveKey(m_hProv, CALG_RC4, hHash, CRYPT_EXPORTABLE | 0x00280000, &m_hKey);
CryptDecrypt(m_hKey, 0, bFinal, 0, pData, pdwDataSize);

As far as I understand, this is what's happening: -

CryptAcquireContext - Get an object to handle the cryptography

CryptCreateHash - Create an MD5 hashing object

CryptHashData - Hash the input data with MD5

CryptDeriveKey, CryptDecrypt - Decode pData with RC4, using the key m_hKey

The size of pszInputData is 12 bytes and the output array of the MD5 hashed object is the same on both platforms.

To decode with RC4, I'm doing the following with Common Crypto: -

CCCryptorRef cryptor = NULL;
CCCryptorCreate(kCCDecrypt, kCCAlgorithmRC4, 0,
                      (void*)m_hKey.data(), m_hKey.length(), NULL, &cryptor);

char outBuffer[12];
size_t outBytes;
CCCryptorUpdate(cryptor, (void*)pData, *pdwDataSize, outBuffer, 12, &outBytes);

Testing the output (outBuffer array) from Common Crypto with an online RC4 decoder matches, so this is decoding correctly.

However, the final output from the Windows code in pData doesn't match the RC4 decoded in Common Crypto.

Are there some steps I'm missing or not understanding with the Windows Crypto API calls here; why do the outputs differ?

(Please note, I'm not looking for comments on the security or flaws in using RC4)

  • Is bFinal being set to TRUE at some point? – RoraΖ Jun 19 '15 at 15:46
  • @raz, yes it is; bFinal == TRUE – TheDarkKnight Jun 19 '15 at 15:48

According to CryptDeriveKey() documentation, upper WORD of dwFlags parameter specifies desired key size in bits. In your case it should generate 40-bit (0x28) keys, effectively discarding all but first five bytes of the MD5 output.

To achieve similar behavior with CommonCrypto you can try this:

CCCryptorCreate(kCCDecrypt, kCCAlgorithmRC4, 0,
                      (void*)m_hKey.data(), 5, NULL, &cryptor);


Actually, the above code is not compatible with CryptDeriveKey(). It turns out (from test vectors published in comments) that Windows still uses 16-byte key but last 11 bytes are set to zero (thus, effective key length is still 5 bytes or 40 bits). Python code that produces same result as Windows code:

from Crypto.Cipher import ARC4
import binascii

KEY = "cfba3dd2ed"
DATA = "eb b9 bd 6f 5d 46 8d 17 c1 88 db 78"

key = binascii.unhexlify(KEY) + "\x00" * 11
rc4 = ARC4.new(key)
print binascii.hexlify(rc4.encrypt(binascii.unhexlify(DATA.replace(" ", "").strip())))

Outputs for me:


This CommonCrypto code should work:

if (m_hKey.length() > 5) {    
    memset(m_hKey.data() + 5, 0, m_hKey.length() - 5);

CCCryptorCreate(kCCDecrypt, kCCAlgorithmRC4, 0,
                          (void*)m_hKey.data(), m_hKey.length(), NULL, &cryptor);
  • Thanks @Andrey, I tried using just 5 bytes and it still doesn't work. The input key bytes being used is "cfba3dd2ed", the data to decrypt is "eb b9 bd 6f 5d 46 8d 17 c1 88 db 78" and the Windows API returns "04 02 1f 02 73 01 01 00 00 00 ff 9b". However, the OS X API results in the following bytes: "7d b1 76 3a 87 99 a0 9a cc ef 3f 61 – TheDarkKnight Jun 19 '15 at 15:33
  • Yep, you're right. Please see edits to the answer, hope that solves the issue :) – Andrey Jun 19 '15 at 16:14
  • I just worked that out, after seeing this thread. Shame the Windows documentation doesn't state this. Thanks for the help. – TheDarkKnight Jun 19 '15 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.