Can I compare 2 same encrypted files by checking it on binary mode?

Question

I have been working on testing whether same encrypted files are same without decrypting it. I have a test file provided to me one with normal data and another encrypted version of it. I need to check whether that test file(without encryption) which goes through an api and encrypts it will be the same as encrypted file that I got. Here, is the python script for checking these files:

def compare_binaries(self,f1, f2):
    '''
    Using ittertools zip_longest to check each lines of both files
    f1 - binary file of source file
    f2- binary file of target file

    Returns True if all lines are same else False if any line is not same
    '''
    for line1, line2 in zip_longest(f1, f2, fillvalue=None):
        if line1 == line2:
            continue
        else:
            return False
    return True 

Is this process okay for checking the encrypted files are same or are there better alternatives for it?

Answer 1

It depends on how the API encrypts the file, and there are a LOT of possible variants here: symmetric (block cipher and stream cipher), or asymmetric, and different implementations of each one.

Some encryption modes employ something called random IV, and in practice it means that encrypting the same file multiple times with the same key will output different files each time. So even if the API is rightfully encrypting the file, the result from the API and the file you just encrypted will be different.

You will have to check what encryption mode the API is using. AES with a deterministic IV would work (like using the truncated hash of the file as the IV, for example). The vital part is to never encrypt two different files with the same IV, or it could leak information on the key.

Only if the API returns the same output every time you submit the same file, then it is possible to compare the hashes of the files.

Answer 2

"I have been working on testing whether same encrypted files are same without decrypting it"

In general, it will not work. The more secure encryption schemes explicitly use a "non-deterministic" (actually they endeavour to be "truly random", not just use a crypto-level PRNG, by collecting entropy through various means) approach, and will never encrypt the same data in the same way twice.

It is sometimes possible to add a (either signed or unprotected) "hash signature" that represents the unencrypted content of the file.

This is usually not done because determining that an encrypted document is identical to a known plaintext one effectively allows "decrypting" the document, or being sure of its content, defeating the cryptography.

The more robust cryptography, therefore, is designed to prevent exactly what you seek to accomplish.

But

If you are dealing with a deterministic encryption, and you're (e.g.) writing a test case to verify that the encryption is working, then this is doable in principle.

For safety, I would not check the files line by line (depending on the platform this might cause unexpected behaviour, or even false positives).

Verify that the size is the same. If not, they files are obviously different.

Otherwise, compare them byte by byte, or use a library function to extract a reliable enough hash (sha256, but in practice, for testing purposes, MD5 is usually enough - chances of false positives are negligible). Then you can compare the two hash strings.

Answer 3

You did not mention the type of encryption is g. There are various ways to achieve comparison on the encrypted data and here some of them;

ECB

To have equality you may need to use ECB mode of operation with a block cipher like AES. This is insecure and there are better and more secure ways than this.

Hash

Hash can be used to check equality, too. However, there are two issues here

1. The hash must be calculated with the plaintext.
2. hashing is free therefore one can calculate hashes of all files found on the internet and may decide some of your files.

HMAC

HMAC ( hash-based message authentication code) requires a key, and we can consider it as a keyed hash. This doesn't have the second problem of the hashes since the attackers don't have the key. Use HMAC-SHA512 to reduce the collision change to negligible. As in hash, the HMAC-SHA512 must be applied to plaintext. Note that, once calculated, the comparison is very fast since it has a 512-bit output.

SIV

If you use the same IV and key for each file with AES-GCM ( internally uses CTR and this valid for any stream cipher) then you can have the equality on the ciphertext, too. Again this is dangerous and can remove the confidentiality and even more, the attacker can forge messages too ( this is catastrophic). This is called nonce misuse, that is the (key, nonce) pair uses more than once. To mitigate this there is Synthetic IV mode (SIV). SIV mode is nonce misuse-resistant authenticated encryption. It simple has 3 properties;

1. The IV is modified with the message itself ( requires double pass on the data)
2. The ciphertexts are equal if the messages are the same.
3. If the messages are different, with a great probability that they are different, too.

Therefore with a fixed nonce and key, you can test the equality.

FHE

Although slow, one can use Fully-Homomorphic-Encryption for the equality of the ciphertext. This will be 100% true, there are no false positives. Microsoft uses this to securely check that you are pwned or not.

Answer 4

The question is probably more simplified by saying "How can I check if 2 binary file are the same?"

md5sum f1 f2

If you get the same value, they are the same. It will even help you detect duplicates in a large set of files.

Answer 5

Comparing the hashes of the files, as @camp0 suggested, is considered to be best practice. This script should point you in the right direction:

import hashlib

def sha256file(fname):
    BLOCKSIZE = 65536
    hasher = hashlib.sha256()
    with open(fname, 'rb') as afile:
        while True:
            buf = afile.read(BLOCKSIZE)
            if not buf: break
            hasher.update(buf)
    return hasher.hexdigest()

file1="/path/to/file1"
file2="/path/to/file2"    
file1hash=sha256file(file1)
file2hash=sha256file(file2)    
print(file1, file1hash)
print(file2, file2hash)

if(file1hash==file2hash):
    print('files are the same')
else:   
    print('files are not the same')

Answer 6

Lots of people pointing out the general flaws in the idea of "encrypt X, see if that matches Y". However, there's a much bigger problem with your specific implementation of the checking code:

Unless the encryption that the API uses works line-by-line, instead of on the whole file, the idea of comparing line lengths is meaningless!

In a normal text file, a line ends when the program parsing the file finds a newline character (\n), or sometimes a combination carriage-return+newline (\r\n). In an encrypted file, those characters have been encrypted and will no longer be recognizable. Sure, in an encrypted file (or any other binary blob of effectively random noise) approximately 1 in every 256 bytes will be a newline character, but it won't in any way be correlated with where newlines occurred in the original file.

Furthermore, checking for matching lengths (of lines or even the whole file) is unlikely to work. Encryption can, and often does, increase the length. Block ciphers usually need padding, which will extend each encrypted message out to a multiple of the "block size" (typically 16 bytes for modern ciphers, some older ones use 8 bytes). Stream ciphers don't have this property, but both block and stream ciphers can (and if even half-competently written, do) have a random "Initialization Vector" (IV) or "NONCE" (Number used ONCE) that is used to ensure each encrypted message (ciphertext) is unique even if the plaintext was the same. The IV/NONCE is not technically part of the ciphertext, but is usually prepended or appended to the ciphertext because the decrypting party needs to know it. Additionally, good cryptography uses an integrity check (something to ensure the ciphertext wasn't messed with before decrypting), such as an HMAC, digital (asymmetric) signature, or the authentication tag of an authenticated encryption cipher. In any case, this requires sending additional data (typically appended) with each ciphertext.

If the API does, in fact, encrypt line by line... stop worrying about whether you've matched the file and start worrying about why you're using a site whose encryption scheme makes WEP look secure.