Everyone says that home brew encryption is not a good idea because one day the algorithm will be known to attacker which sounds reasonable. So that seems it's main problem, on the other hand known algorithms are known to everyone and are most-likely safe to use. Would it make sense to some paranoid developer to create a scheme like that AES(HomeBrew(openText)) where HomeBrew will either provide an encryption or some other kind of transformation to make it harder for the attacker as they will have to guess those things even if they manage to break AES somehow (provided they do not have the algorithm). So won't we get best of both approaches in this case?

  1. Standard encryption that is proven to be good
  2. Unknown algorithm "just in case" that even in case of leaking will not provide much to attacker unless they know how to break AES
  • 9
    If somebody breaks AES, you better know you're screwed. – rev Jul 18 '15 at 12:20
  • Seems like this is actually a very good point: if AES is broken then you should assume that your whole encryption scheme is broken instead of relying on unverified scheme which is probably a question of "when", not "if". So according to this logic there is just no point in adding that layer because 1. If AES is broken your are screwed anyway 2. If AES is not broken: no need in another part. Though perhaps depending on the case it might help to postpone the attacker e.g. by a week which can be a good thing if your transmitted data is valid for less than a week. – Ilya Chernomordik Jul 18 '15 at 22:12
  • 2
    You might as well do AES(ROT13()). – bishop Jul 19 '15 at 3:31
  • 5
    Moreover, aside from security considerations, it's worth noting that AES is also verified for correctness, which guarantees you that you can decrypt everything that you have encrypted. In your HomeBrew solution you might find yourself unable to decrypt your own data because of bug in your coder/decoder. – el.pescado Jul 19 '15 at 8:03
  • This related discussion may contain relevant information: can multiple encryption of data with multiple keys increase the security?. It does not deal with the home brew algorithm specificities, but covers security implication of cascading several crypto algorithms with different keys. – WhiteWinterWolf Jul 20 '15 at 13:21

Kerckhoffs's principle states:

A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.

Therefore if AES is broken, your homebrew algorithm is likely to be much easier and would have been broken already.

I realise part of your original question stated:

provided they do not have the algorithm

However, as this violates Kerckhoffs's principle your home brew approach is in itself flawed and insecure.

The main reason this is a problem is that it is unquantifiable how much more security it adds to your system. I refer you to Thomas Pornin's answer to a similar question here:

An additional twist is that algorithm obscurity can harm security. What I explain above is that obscurity cannot be trusted for security: it might increase security, but not by much (and you cannot really know "how much"). It turns out that it can also decrease security. The problem is the following: it is very hard to make a secure cryptographic algorithm. The only known method is to publish the algorithm and wait for the collective wisdom of cryptographers around the world to gnaw at it and reach a conclusion which can be expressed as either "can be broken that way" or "apparently robust". An algorithm is declared "good" only if it resisted the onslaught of dozens or hundreds of competent cryptographers for at least three or four years.

So therefore you should use another algorithm if you are worried that AES may soon be broken (that or don't use AES at all). I'm assuming the rest of your system isn't 100% secure - so instead of wasting time, effort, energy and resource on creating a home brew algorithm this effort should be spent elsewhere, where it can be quantifiable that it is actually increasing security. All code has a cost, and it is not just the salary of the junior developer that is creating it. It needs to be maintained and understood by everybody - and each extra person that learns how it works is an extra avenue for the algorithm to be leaked. Don't waste your time.

| improve this answer | |
  • You are saying the approach is insecure, but the whole point is that it is covered by a secure approach, so it does not make it worse, but can possible make it better if you are lucky to keep the algorithm in secret, so it makes life of an attacker harder even a "little bit" compared to AES and can give you some time, isn't it? – Ilya Chernomordik Jul 18 '15 at 18:59
  • 1
    You're adding extra complexity, and complexity doesn't sit well with security. Also if AES is broken you can never be sure that your algorithm is secret. If that was ever transmitted by AES then it could be known. – SilverlightFox Jul 18 '15 at 19:04
  • 3
    Why not use another established algorithm rather than home brew? – SilverlightFox Jul 18 '15 at 19:26
  • 7
    AES(k_1, HB(k_2, pt)) can easily be completely broken if HB is designed by an amateur. For example, if HB uses loop bounds or lookup tables based on the plaintext, it can trivially leak enough information for an attacker to recover it without having to break AES. Leave cryptography to professional cryptographers. – Stephen Touset Jul 18 '15 at 20:49
  • 1
    @PatrickM In addition to probably not providing much extra security, you open yourself up to a whole variety of side channel attacks that were (hopefully) already considered and mitigated in the standardized encryption you're using. – Chris Hayes Jul 19 '15 at 3:50

It could work, and would be security-in-depth instead of security-by-obscurity, but there are a few ways to mess this up catastrophically:

  • Using a HomeBrew algorithm that is vulnerable to side-channel attacks. Now the attacker can do a simple timing or cache analysis, for example, and bypass the AES part entirely. Are you that secure in your implementation?

  • Reusing material between the algorithms. If you reuse keys and nonces, for example AES(HomeBrew(openText, "secret"), "secret"), it might be vulnerable to cryptanalysis. This is a distant threat, but necessary if you want to keep the same security standard of just AES(openText).

  • Using a brittle HomeBrew implementation. If it might break when the text contains nulls, or are above a certain size, or are below a certain size, or look like this, or contains valid unicode, or contains invalid encodings, etc. This is a programming concern, but opens your software to anything from denials of service to improper authentication or worse.

All in all, I find this a fun idea, but not a practical one. Maybe using something that is not homebrew would work better; there are safer ways to add depth to your security.

| improve this answer | |
  • How can an attacker bypass the AES part completely? Provided I got your answer correctly, they can get the result of homebrew encryption they can as well get the open text itself I guess so it's too late to save anything. – Ilya Chernomordik Jul 18 '15 at 19:02
  • 1
    If HomeBrew leaks the plaintext through side channels (timing, cache, external requests, etc), there's nothing that AES or further ciphers can do to re-secure it. You don't need to see the ciphertext of the homebrew encryption, just its side-effects. Maybe doing HomeBrew(AES(text)) fixes this, because it would only leak the AES ciphertext, but it's still risky. – BoppreH Jul 18 '15 at 19:42
  • 1
    Actually, if we do it the way you say HomeBrew(AES(text)) and suddenly this exposes AES in some way, we will probably be famous by breaking AES as we have essentially found a transformation making AES vulnerable, meaning HomeBrew(AES(text)) cannot be worse than AES(text), isn't it? :) (Provided we don't mess with key used for AES or with the open text after AES) – Ilya Chernomordik Jul 18 '15 at 22:20
  • 1
    @IlyaChernomordik: suppose that "HomeBrew" is actually AES but used in the decryption direction; this is a perfectly fine block cipher, and is as secure as normal AES (since any weakness in the decryption direction would still count as a weakness). However, in that case, AES(HomeBrew(x, k), k) = x for all plaintexts x and all keys k; i.e. no encryption at all. You cannot get weaker than that. Therefore, the combination of AES and HomeBrew with the same key (or related keys) can be utterly weak even if AES is perfectly secure. – Thomas Pornin Jul 18 '15 at 22:45
  • That's why I wrote: provided we don't mess with key used for AES or the open text. Guess you have to be really "smart" to apply the reversed one :) Is it still possible then? I have asked a broader question for it here: security.stackexchange.com/questions/94338/… – Ilya Chernomordik Jul 18 '15 at 22:47

By itself, your home-brew algorithm is a form of security through obscurity. However, when applied in combination with a known good encryption, your home-brew algorithm may be considered a defense in depth strategy. Quoting directly from Wikipedia,

A system may use security through obscurity as a defense in depth measure; while all known security vulnerabilities would be mitigated through other measures, public disclosure of products and versions in use makes them early targets for newly discovered vulnerabilities in those products and versions. An attacker's first step is usually information gathering; this step may be delayed by security through obscurity.

If a vulnerability in AES is publicly disclosed today, it would still take some time for an attacker to figure out how to crack your home-brew algorithm, giving you time to switch to another algorithm. As the saying goes, "don't put all your eggs in one basket".

For the really paranoid, you can adopt two layers of different standard encryption such as AES+DES. Having multiple layers of different encryption is not cost-free though. The trade-off is that your ciphertext becomes chunkier.

| improve this answer | |
  • 1
    "it would still take some time for an attacker to figure out how to crack your home-brew algorithm" - There's no guarantee of that. You should see some of the homebrew algorithms that people have come up with.... – D.W. Jul 18 '15 at 23:20
  • Homebrew algorithms may mitigate AES's security. What if they accidentally reinvent the AES decryption algorithm? – PyRulez Jul 19 '15 at 19:40

Dieter Vandenbroeck wrote an article on when security through obscurity makes sense:

Usually, the principle of security through obscurity is just utterly wrong. The idea behind this principle is to have obscurity in your solution as the principal means of security. Of course, this is not a valid substitute for real security: once your obscurity schema is broken, your security would be broken. It might form a defense against automated scripts, but an attacker with sufficient determination will always be able to fight through this defense.

In your case you won't have an automated script and by the time AES can be cracked in an automated way, homebrew will probably be even more flawed and as easy to crack. So no I do not think homebrew would give you an additional security benefit compared to plain AES.

| improve this answer | |
  • 1
    Yes, but is a home-brew algorithm classical security through obscurity? I mean, if you family name is Shneier, El Gamal, or KoreK you might in fact be on the safe side with such a scheme. – Konrad Gajewski Jul 18 '15 at 11:08
  • It does make sense, even if you use ROT13, since you will add a layer of security. AES(HomeBrew(openText)) will always be more secure than AES(openText)...

  • ...unless this algorithm of yours adds some weakness to AES that will make cryptanalysis easier. Which I doubt.

  • But I'd personally stick with known algorithms and use something like AES(Threefish(Plaintext)).

  • However, where is the fun in that?

  • And how do you intend to keep your algorithm secure, if you intend to use it?

I generally wouldn't bother.

| improve this answer | |

I agree with abligh.

Undertaking "Homebrew_Crypto(AES(plaintext))" will likely not add a great deal to the security of the system -- but what it WILL do -- even against a state-level attacker -- is simply slow them down. (It will slow down unsophisticated attackers a lot; for No Such Agencies it will force them to use a few more cycles of supercomputer time, which may get you brownie points in their "Enemies Of Our Beloved National Security State" list.)

One really funny variation of this would be :

"Homebrew_Crypto(Twofish_Or_Some_Other_Good_Crypto_Algorithm(AES(plaintext)))" -- at the risk of significantly slowing I/O, of course. Kind of like those Russian dolls that nest one inside the other.

Note that the latter option was available in Truecrypt and I believe in its successors like Veracrypt.

| improve this answer | |

There is another reason why this might help. Rather than doing AES(Homebrew(Plaintext)), let's assume you do Homebrew(AES(Plaintext)).

Let's also assume the worst case, and though you think you are crypto genius, your Homebrew algorithm is no more useful than ROT-13.

In this case, today you have a combined scheme which is no worse than AES alone (save that it might be a bit slower), but no better. If your Homebrew is better than ROT-13, who knows the combined scheme might be slightly better than AES alone.

However, let's now assume that some time in the future, AES gets broken. A zero day exploit becomes available, and an exploit is widely publicised. Script kiddies galore attempt to find AES encrypted material and decrypt it. Assuming they aren't targeting you specifically, they are unlikely to be able to decrypt your data using the hack, as they won't think to apply some other decryption first, and there are low-hanging fruit elsewhere. This at the very least gives you a little more time to change your underlying encryption to something that isn't broken.

Whether that's worth the speed and maintenance penalty is another question entirely.

| improve this answer | |
  1. Standard encryption that is proven to be good
  2. Unknown algorithm "just in case" that even in case of leaking will not provide much to attacker unless they know how to break AES

Except you now have TWO standards to maintain, and a weakness in any ONE of them will weaken the second, because two ciphers in tandem effectively produce a third cipher which is subject to each one's weaknesses. The homebrew is probably going to spoil it for everyone, at the end of the day. :)

If you have one cipher which is KNOWN to be stronger than the other, then effort would be better spent in ensuring that it is properly deployed.

Even established protocols with provable security should only be combined in certain cases; cascading unsuitable algorithms can result in underlying patterns emerging when the individual ciphers effectively "undo" each other's work, simply because their methodologies are counter productive.

It is far more likely that a homebrew cipher is going to weaken the result rather than strengthen it. In my opinion, it's best not to knit you own crypto.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.