Security BY obscurity is horrible. Is security AND obscurity good?

Question

Normally I preach that rolling your own custom crypto algorithm is a bad idea. But will it really hurt if it's the outermost layer though? Or will it make security worse?

AES -> CipherText -> CustomEncryptionAlgorithm-> CipherText

I'm thinking that the extra layer will help. Let's say even if CustomEncryptionAlgorithm is bug ridden mess, it can't possibly make things worse. That's because AES output is already indistinguishable from random noise.

On the other hand, something tells me that the following is problematic

CustomEncryptionAlgorithm -> CipherText -> AES -> CipherText

Is it bad? and why?

Please don't comment on company resources spent on security vs obscurity etc (I agree security comes out ahead) I am more interested in understanding the cryptographic theory behind vulnerabilities in this approach.

Answer 1

Don't roll your own crypto!

From a purely cryptographic point of view, any length-preserving bijective function cannot reduce security. In fact, even the identity function, defined as f(x) = x, will not reduce security, assuming the keys used for the standard cipher and your homebrew cipher are mutually independent. The only possible way it could reduce security is if your homebrew function does not use a different, independent key and leaks the key in the ciphertext, for example with f_k(x) = x ⊕ k done on each individual block of input x, a classic XOR cipher vulnerable to known-plaintext attacks.

From a practical standpoint, there are gotchas that can matter. I mentioned length-preserving above for good reason. A compression function is still a function, and sometimes compression and encryption can lead to very bad results. This is partially why your second example, with your custom algorithm applied before standard encryption, is indeed worse. It can leak information about the plaintext through length. There can also be bugs in your implementation that result in other security vulnerabilities. From a purely theoretical point of view, they are out-of-scope.

---

It was pointed out to me in a comment that I may not be sufficiently emphasizing just how bad of an idea this is. While it may be fine in theory, the real world works differently. Actually using your own homebrew crypto is a very bad idea, no matter how you use it. The only time you should ever actually do this is if you are a professional cryptographer. Bernstein can do this. Rivest can do this. Rijmen can do this. You cannot. Don't shoot yourself in the foot and instead use proper algorithms.

Answer 2

There is a risk involved when you apply your custom encryption algorithm first.
This is based on the fact that an encryption like AES does leak information about the length of the plaintext.

Suppose the extremely hypothetical (and unpractical) example where you custom algorithm 'encrypts' a single byte plaintext like 0x40 into 64 zeros and a two byte plaintext like 0x02 0x00 into 512 zeros.

When you encrypt this using AES an attacker will still know the length of the custom encrypted result just by looking at the length of the AES encrypted text.
With this information the attacker can 'decrypt' this into the original plaintext without having the AES encryption key.

In summary: a very bad custom algorithm can indeed harm the security of a following AES encryption.

Answer 3

The first question to ask when assessing any sort of security system is "Who is the attacker, and what can they do?" At the low end, you're against an attacker who can just see the encrypted output of one of your messages. At the high end you're up against an attacker who knows the plaintext and ciphertext pairs of hundreds of other messages, who can coerce your computer into encrypting other messages with the same key, who can monitor the power lines to your building and measure the tiny variations in power draw as the encryption process proceeds, and all kinds of similar attacks. Standard encryption algorithms are vetted to make sure they're resistent even to these powerful adversaries.

If you use your own encryption first, even if the second layer is military grade encryption, you risk decreasing your security compared to just using the standard encryption.

For example, suppose that you are encrypting text. You first use a simple substitution cipher and then use your choice of heavy hitting algorithm. Unfortunately, substituion ciphers are inherently reliant on doing array lookups from private information (i.e. your plaintext and your key). This makes them vulnerable to timing attacks: because of the way CPU caches work if two letters in the plaintext are close together they will likely encrypt faster than two letters that are further appart. That doesn't sound like much information, but if an attacker can watch enough encryptions they can potentially work out what the plaintext is without having to crack AES or whatever you're using on top.

If your custom encryption layer does not interact at all with secret information: i.e. it works with data that has already been securely enough encrypted for transmission and doesn't touch the keys of the standard encryption, then it may be safe. At least, it won't be possible to bypass the standard encryption. There are still risks, though, because every extra piece of software carries additional risks. Perhaps a bug in your custom layer lets a remote adversary run arbitrary commands on your machine; then they could just email themselves the plaintext!

The take home message, then, is that having one lump of excellent security code that is well vetted and bug free like a standard encryption function can still be undermined if you put a buggy home grown program (of any sort, encryption or otherwise) onto the same system.

Answer 4

Simple. Don't do this.

First of all, Encryption algorithms are designed so that you, me, and everyone around us can look at it and try to break it. You can quite literally look a the math for AES. A lot of smart people have invest time in validating AES. This is a tenet of good encryption and why we can trust it.

Second, Obscurity is never a "Security" option. It doesn't provide any benefit. If you actually believe this, than I refer you to "Schneier's Law".

Anyone can invent a security system that he himself cannot break. I've said this so often that Cory Doctorow has named it "Schneier's Law": When someone hands you a security system and says, "I believe this is secure," the first thing you have to ask is, "Who the hell are you?" Show me what you've broken to demonstrate that your assertion of the system's security means something. - Bruce Schneier, 2016

All you've done is:

• Introduce unnecessary risk into your application
• Increased debug time for issues
• Added CPU time for a worthless function

The most basic principal in Security. Keep It Simple Stupid (KISS).

Answer 5

Security by obscurity is good, sometimes even great, as long as it's not the only layer of security you rely on. But in your case I'm afraid it's not worth it.

Take GPG for example. If you use GPG to encrypt a file with AES, it will have a header that allows an attacker to recognize it as a GPG file encrypted with AES. If you then use custom encryption to encrypt the GPG file, you will get a file that might be unrecognizable. An attacker might wrongly suppose you are using whatever kind of encryption (and lose a lot of time for nothing), or might suppose you are using your custom encryption and then decide to use cryptanalysis techniques to analyze your file (and manage to decrypt it if he's good enough or your algorithm sucks enough). On the other hand, if you do it the other way around (first encrypt with custom algorithm and then with AES), the scenario doesn't change significantly after all.

That said, I'm afraid it is not worth it because custom encryption has a huge con anyway: you will have to store your encryption software somewhere, and your software is totally custom, meaning that if for whatever reason you lose it (lost or stolen backups, etc.) you are screwed and you won't be able to recover any of your data.

A better solution might be to use several layers of encryption using algorithms that are already known and available. For example GPG seems to support twofish, camellia, etc. I have no experience to know which algorithms should be considered the safest though, apart from AES. Anyway, just as an example, AES -> TWOFISH -> CAMELLIA is probably a much better solution that your AES -> CUSTOM_ALGO. Of course provided you use different strong passwords for each encryption layer!

Answer 6

OPSEC?...

Putting additional barriers between critical information and an adversary is a good idea in general, provided it is done effectively.

Obscurity as a meaningful security measure is addressed by the principle of OPSEC (operational security). In short, this entails depriving an adversary of any information which could help them compromise your organization via either social or technical methods.

With good crypto, however, keeping the keys secret is sufficient to protect your data. Any additional technical layers must be justified on their own technical merits.

...Or bust

Cryptography is a very complicated mathematical discipline, and small mistakes can have enormous consequences. When dealing with cryptography, assume the value of an algorithm is virtually zero unless a trustworthy expert indicates otherwise.

In the US, the NSA and NIST are the official assessors of cryptographic algorithms and implementations, respectively. If you don't know where to look for opinions or want a reasonable basis for internal policies, start there.

In practice, wrapping good crypto is pointless. Once an attacker encounters good crypto, they will generally pivot and attack endpoints where the unencrypted data is exposed.

This could be the web server where users supply information, the SCADA system that feeds into your database, or the workstations where employees analyze or monitor the information. If you have a data pump that pulls information from a database and converts it to another format for delivery to a vendor/customer, that's another big target. Alternatively, they could go searching for your keys.

On the balance

Adding features to an application has a non-zero chance of introducing bugs and complicating troubleshooting. This can lead to extended downtime when something goes wrong or slower responses to security vulnerabilities in the application. Due to these factors, a security feature of little-to-no value is a detriment rather than an improvement.

If you wish to protect yourself against a potential attack on AES, then you should simply use another crypto standard based on a vetted algorithm rather than developing your own. You will get better security with less effort.

Answer 7

There are two main issues with the security and obscurity mode.

1. Obscurity can interfere with security. If you roll your own crypto it is very likely to be flawed. Even the best systems have flaws, despite the leading experts having worked on them for decades. The chances of you being able to do better than the security community is low.

2. Few things are really obscure. There aren't many new ideas in crypto and whatever you do is likely to be a variation on something that has been done before, so it's not actually that obscure. Minor changes are something attackers are used to handling.

Answer 8

It's easy to see that the first variant (custom algorithm last) can't compromise the security of a well-known encryption, since if it could, it would be a method for cracking the well-known encryption. After all, attackers could also run the custom algorithm on a cyphertext encrypted solely with the well-known encryption if it helped them cracking it.

The other direction (custom algorithm first) could indeed compromise the security of the well-known algorithm by introducing redundancy into the plaintext input of the well-known encryption, which could potentially weaken that algorithm.

For example, take the following utterly flawed custom “encryption” scheme: repeat the input plaintext twice to produce a “cyphertext”. This leads to an extreme amount of added redundancy in the input of the second algorithm, which could weaken some real encryptions. (Whether it weakens AES or not is a different question.) This has actually happened with the Enigma, where operators repeated a three-letter sequence in its plaintext that made its cyphertext vulnerable to certain kinds of attacks. Quoting from the Cryptanalysis of the Enigma Wikipedia page:

The second problem was the repetition of message key within the indicator, which was a serious security flaw. The message setting was encoded twice, resulting in a relation between first and fourth, second and fifth, and third and sixth character. This security problem enabled the Polish Cipher Bureau to break into the pre-war Enigma system as early as 1932. On 1 May 1940 the Germans changed the procedures to encipher the message key only once.

Answer 9

Solution 1 could potentially decrease security if you use the same secret key for encrypting AES and CustomEncryptionAlgorithm.

CustomEncryptionAlgorithm might allow the deduction of the secret key and therefore compromise AES as well.

If you create a unique key for CustomEncryptionAlgorithm you should be fine again.

Something like KEY = SHA-2(AES-CipherText)

Answer 10

The problem is that it is possible your custom encryption function has a bug and it somehow reduces the security. The only way for you to be sure that your CustomEncryptionAlgorithm isn't reducing your security is to have researchers who are smarter than you, me, and the rest of your team combined to pour over it and check your work. But then it isn't any way obscure at all.

To make an extreme example:

public string CustomEncryptionAlgorithm(string plaintext) {
    var ciphertext = plaintext.shuffle();
    return "password";
    return ciphertext; // ya ya. I know. You'd never make a bug that obvious.
                       // tell it to the guy who did the 'goto bug' at apple
}

You'd get something like this as a result if you used this scheme:

• AES -> CipherText -> CustomEncryptionAlgorithm-> CipherText :

  'abcABC123!@#' -> 'password'
  'correct_horse_staple_battery' -> 'password'
  'password' -> 'password'
  

On the other hand, if you used the other scheme:

• CustomEncryptionAlgorithm -> CipherText -> AES -> CipherText

  'abcABC123!@#' -> '8ZnO44trPK48kqr3rDxkdQ=='
  'correct_horse_staple_battery' -> '8ZnO44trPK48kqr3rDxkdQ=='
  'password' -> '8ZnO44trPK48kqr3rDxkdQ=='
  

Slightly better, maybe. But still not any good in the slightest.

Answer 11

As has been said: Who is the attacker?

This is really what you must find out and define first!

Is it your technophobic grandma who has problems with the TV remote control? Your little (and very nosy) sister? Your dad, who actually programs and administrates computers --- and knows how to remove a hard drive? The local school bully who works you over? Organized crime? A company which can spend millions on renting cloud computers for a cracking attack? A third world dictator? FBI, NSA or equivalent agencies in your own country? Some world power that has an excellent spy service and will not mind body bags to get you?

Second: What extra, real security would your 'encryption'1 give you against your attackers? Unless you can make a very good point why encrypting the ciphertext again[2] is needed or very helpful, you should not do so. More complexity does not mean more security, it means more bugs and more chances of things not working properly?

And why won't attackers not break or sneak into your house and place some spy software or key logger on your computer? Are you sure your encryption will be any help against rubber hose cryptoanalysis[3], or if there is no chance of that, who is powerful enough to break AES but unable to reach you or your dearest and nearest?

---

1. Let's just say your chance of getting the encryption strong and secure is about as high as my chance to walk on the moon; with encryption, not only must everything work with the right key, but nothing may work without it ...

   I do not know of any cryptosystem built by someone outside the field that did not crack when being checked by experts, and how many strong encryption systems invented by the best experts have been broken?

2. One example would be the communication in WWII from Germany to the submarines. Some very secret communication was "officer only", which was encrypted, meta-information was prepended and both were then encrypted by the regular key and sent. On receiving the radio/Enigma operator would decrypt the message and pass the meta information and still encrypted block to the officer. (see here for example)

3. "in which a rubber hose is applied forcefully and frequently to the soles of the feet until the key to the cryptosystem is discovered, a process that can take a surprisingly short time and is quite computationally inexpensive.”

Answer 12

Short answer: It can be, but not in your example.

Long answer: Other answers have adequately covered why your example is not all that helpful and can hurt. That being said, obscurity can be very helpful. Case in point: Don't put SSH on port 22. If you change from 22 to 2222, you will have a tiny fraction of the brute force attempts you would otherwise. If you change to port 10000+rand(1,55535) they'll likely disappear completely. Another example is port knocking. Basically, obscurity is mostly useful in the connection process, not in encryption or passwords.

Answer 13

You refer to the second algorithm as obscurity, so you seem to assume it does not add any security.

If it actually does not add security, why use it? Use the second password in addition to the first one for the aes encryption. A password twice as long is much more secure than two passwords (example: 2^2+2^2 = 4 vs. 2^4 = 16), especially when the second algorithm is insecure anyway.

Answer 14

If done properly: sure. The question is how do you know it's proper? You certainly can write something that leaks information through various layers. Most obvious one is length and repetitions. Also using the same key might be problematic.

It's just too hard to tell what's proper and whatnot. Sure, you could argue that using ROT13 before AES makes it a little bit secure and my intuition would say... well it at least can't hurt but I don't actually know.

My intuition would say that if you apply an algorithm that produces output indistinguishable from randomness while not changing the length and then apply AES it at least shouldn't reduce the security of AES.

Answer 15

Obscurity is a part of a security (general), and is orthogonal to the security (AES encryption in your case). Obscurity defends against different threats than crypto. Take passwords for example - crypto hashes are required to prevent leaking plaintexts from server, TLS prevents from eavesdropping network traffic, while obscurity being a password form filling with [*******] prevents from peeking your plaintext by random spectator. So you cannot do security by obscurity (only), but security with obscurity is usually the way to go. In real world the obscurity means hiding the metadata, e.g. mailing with BCC instead of CC/To or blocking 3rd party cookies.

Answer 16

It's bad, bad, bad.

There is an anecdote from some Far East mobile protocol for SIM initialization or something similar, that used a combination of RSA (good, secure, and strong) and something homebrewed.

The decisive weakness was not only the homebrewed "additional" encryption. But the problem was that in fact there was a special case when the additional method was associative with RSA, which caused partial key bleeding. I don't remember details, as I was told them as a lecture anecdote many years ago.

But the point stands: with an improper homebrew addition to an otherwise secure protocol you might make the protocol weaker than its original version. So, that's a major no go.