I have asked a question regarding the benefits (if any) of the following scheme: AES(HomeBrew(plaintext)). There are many good answers for that, but many people say that it's theoretically possible that your algorithm will reduce security of the whole scheme, so it will be possible to break it without breaking AES itself.

Assuming this is correct, what if we now invert the transformations to HomeBrew(AES(plaintext)). It seems there is no way of doing it wrong, because if you really compromise the security of the scheme in your algorithm that will really mean you have found a vulnerability in the AES itself, which can probably make you famous, but does not seem very likely to happen.

Even though security by obscurity is known to be bad, it can still require a lot of resources for an attacker to break. The famous Enigma example proving security by obscurity is bad still proves my point as well: even though it was broken it did cost enormous efforts to do so. So I have few questions with regards to all this:

  1. Is it possible to compromise security by doing AES(HomeBrew(plaintext)) [Provided we do not mess with keys between algorithms]?

  2. Is it possible to compromise security by doing HomeBrew(AES(plaintext)) [Again provided that we don't mess with keys or open text in the out algorithm]?

  3. Will the scheme from point number 2 raise cost for an attacker, perhaps proportionally to you ability to keep the scheme secret and you crypto skills? (So even if someone hacks it, they have to pay the cost and bump into AES after that).

  • openText = plaintext ​ ? ​ ​ ​ ​
    – user49075
    Commented Jul 18, 2015 at 23:20
  • I am not a specialist, so I might be wrong in using the terms, but perhaps plain text is more correct term. By open text I mean the text that needs to be encrypted. Thanks for the note, I'll fix it. Commented Jul 18, 2015 at 23:22
  • possible duplicate of Double encryption with home brew algorithm
    – RoraΖ
    Commented Jul 20, 2015 at 12:06
  • That one is not a duplicate, that other question I have asked myself as well, and it's not exactly the same. Here I want to understand mostly if you can compromise security by doing it this way. Commented Jul 20, 2015 at 12:13
  • I'm not sure Enigma example is the best choice. Enigma has been precisely conceived in a way that, even when someone gets its hand on a working machine, he will not be able to use it in order to encrypt or decrypt any message. Enigma failed due to improper usage (known plain text and key reuse mostly), not because of a security through obscurity issue. Commented Jul 20, 2015 at 13:31

1 Answer 1


From a mathematical point of view: when you cascade two encryption functions f and g in that order (you encrypt with f, then encrypt the output of f with g), then g cannot make the data less confidential than with f alone, provided that an important condition is fulfilled: the secret key used for g must be unrelated to the secret key used for f.

It is not that easy to make a formal and correct definition of what "unrelated" means here. Basically, suppose that f is applied by Alice, who then hands the encrypted data to Bob, who applies g; decryption would go in the reverse direction (Bob, then Alice). Alice knows the key for f, Bob knows the key for g. By definition, Bob only works on what Alice produced, and Alice uses a key that Bob does not know. Thus, regardless of who much incompetent Bob is, he cannot unravel the encryption done by Alice: if Bob could through using a really poor algorithm, then any attacker could do as well, and that would mean that Bob's incompetence would count as a break on f (the encryption performed by Alice).

This points out that the "unrelatedness" of the keys for f and g can be obtained by either generating both independently of each other, or by producing the two keys from a Key Derivation Function. For instance, there is a master key K, and you run it through a KDF to obtain 256 bits of output; the first 128 bits will be the key for f, the other 128 bits will be the key for g. If the KDF is secure as a KDF, then the two keys will be sufficiently "unrelated".

Using the same key for both algorithms ("AES" and "HomeBrew") could prove deadly. Deriving one key from the other could also be problematic, depending on how you derive them. Deriving both keys from the same source with a KDF that is sufficiently one-way ought to be safe.

The above is the "mathematical" answer. But cryptography does not happen in the ethereal world of mathematics. At one point, you run it on actual hardware. Even with "unrelated" keys, the combination HomeBrew(AES) can still be harmful in the following ways:

  • Software is secure only insofar as it is reviewed and reviewed again and fixed. This is a time-consuming process. By implementing two algorithms instead of one, you doubled the quantity of code to review, and thus, mechanically, halved the available workforce for the review of either algorithm.

  • Computing "HomeBrew" might be fast, but certainly not as much as not computing "HomeBrew". By cascading the two algorithms, you increased the computational cost of doing encryption. This may imply increased hardware costs, or higher latency, and possibly degraded user experience. If the user experience is too much degraded, then the user may want to actually skip the encryption step, and then security has been seriously harmed.

  • "HomeBrew" being, well, home-brewed, it is not compatible with anything else on Earth. When you use standardized algorithms, you may have the choice between several implementations. For instance, is using normal SSL/TLS, you can rely on the existing OpenSSL library, but if at some point you get fed up with the hole-of-the-month feature of OpenSSL, you can switch to another implementation like GnuTLS, which implements the same protocol -- you can do that on one side without impacting other systems, since they all run the same standard protocol. By using a home-brewed algorithm, you squander these advantages. You are now married with a single implementation and must stick with it until the End of Times.

  • If you use AES(HomeBrew) instead of HomeBrew(AES), you get all the disadvantages above, and you get a brand new one as well, which is that "HomeBrew" now processes the actual plaintext, and may leak information about it through side-channels. In that configuration, you lose all the mathematical security that I was explaining above: incompetent Bob cannot reveal anything about the plaintext when he only ever sees data encrypted by Alice; but if he gets the real plaintext, then anything goes.

proportionally to your ability to keep the scheme secret and your crypto skills

Neither algorithm secrecy nor crypto skills can be measured or even quantified, so your "proportionally" is meaningless.

This is a fundamental point: as per Kerckhoffs's principle, we use public algorithms because we do not know "how much secret" a given algorithm can be, especially since it is implemented as software, and also exists as source code, and in the head of one or several developers. When we want security, we do not want just security; we also want to know how much of the stuff we have. And we cannot do that with a secret algorithm.

  • So if we decide to use a known simple algorithm instead of the HomeBrew (e.g. Rot13, or rather something a bit more sophisticated), and then apply it on top of AES, that will make it much harder for less advanced attacker still, won't it? If your plain text is a result of AES which should be an array of random bytes hopefully that does not give any clue to an attacker, doing simple transformation on top of it will produce additional challenge at almost no cost? Or am I wrong in assuming it's not that easy to find a simple algorithm applied to AES on top? Commented Jul 22, 2015 at 7:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .