99

Normally I preach that rolling your own custom crypto algorithm is a bad idea. But will it really hurt if it's the outermost layer though? Or will it make security worse?

AES -> CipherText -> CustomEncryptionAlgorithm-> CipherText

I'm thinking that the extra layer will help. Let's say even if CustomEncryptionAlgorithm is bug ridden mess, it can't possibly make things worse. That's because AES output is already indistinguishable from random noise.

On the other hand, something tells me that the following is problematic

CustomEncryptionAlgorithm -> CipherText -> AES -> CipherText

Is it bad? and why?

Please don't comment on company resources spent on security vs obscurity etc (I agree security comes out ahead) I am more interested in understanding the cryptographic theory behind vulnerabilities in this approach.

  • 3
    I'm confused by the way that you've listed the workflows. Is the first workflow var x = plaintext; var y = aes(x); var z = custom(y); send(z) and the second is var x = plaintext; var y = custom(x); var z = aes(y); send(z); - or do I have it backwards? And could you elaborate on why you feel the two are different in terms of cryptographic strength because it's not clear why you feel that. – corsiKa May 31 '18 at 15:24
  • 2
    Cheap answer: as long as there's security and it's not bothered in any way, there's security. – EKons Jun 1 '18 at 18:17
  • 2
    I would say that the plain term explanation is that "Security from obscurity is always short lived". So if your application is still secure after the obscure part is reviled to the world, it is by definition still secure. Of course the real problem with home-brew is unwittingly leaking some piece of information; thus, unknowingly making it less secure. – Tezra Jun 1 '18 at 19:07
  • 6
    A custom encryption algorithm isn't "security through obscurity". It is that if you rely on the fact of the algorithm's secret status for security. "Nobody has heard of this algorithm, therefore it is secure" is security through obscurity. – Kaz Jun 2 '18 at 2:14
  • 3
    The biggest problem with obscurity is that it prevent peer review which is immensely useful in this field. – Thomas Jun 4 '18 at 22:47

16 Answers 16

87

Don't roll your own crypto!

From a purely cryptographic point of view, any length-preserving bijective function cannot reduce security. In fact, even the identity function, defined as f(x) = x, will not reduce security, assuming the keys used for the standard cipher and your homebrew cipher are mutually independent. The only possible way it could reduce security is if your homebrew function does not use a different, independent key and leaks the key in the ciphertext, for example with fk(x) = x ⊕ k done on each individual block of input x, a classic XOR cipher vulnerable to known-plaintext attacks.

From a practical standpoint, there are gotchas that can matter. I mentioned length-preserving above for good reason. A compression function is still a function, and sometimes compression and encryption can lead to very bad results. This is partially why your second example, with your custom algorithm applied before standard encryption, is indeed worse. It can leak information about the plaintext through length. There can also be bugs in your implementation that result in other security vulnerabilities. From a purely theoretical point of view, they are out-of-scope.


It was pointed out to me in a comment that I may not be sufficiently emphasizing just how bad of an idea this is. While it may be fine in theory, the real world works differently. Actually using your own homebrew crypto is a very bad idea, no matter how you use it. The only time you should ever actually do this is if you are a professional cryptographer. Bernstein can do this. Rivest can do this. Rijmen can do this. You cannot. Don't shoot yourself in the foot and instead use proper algorithms.

  • 4
    I agree. If there was an function such that f(AES(x)) leaked any info about x, an attacker with access to cipher text could simply apply f herself. Indded, the existance of such a function would make AES unsafe. – Anders May 31 '18 at 13:41
  • 9
    I'm not sure I'm understanding accidentally invent AES and set it to decrypt with the same key correctly. If you're saying that the same key is going to be used for AES and CustomEncryptionAlgorithm, there's a decent chance CustomEncryptionAlgorithm might compromise the key itself, even if it's used to encrypt the AES ciphertext. – Dennis May 31 '18 at 15:23
  • 5
    Introducing additional complexity or obscurity is bad for maintenance. Someone (maybe you) will have to maintain this code / process in the future. Anything which hinders understanding is detrimental. Since it doesn't increase security in any meaningful cryptographic way, I'd argue complexity or obscurity for its own sake reduces maintainability, and hence indirectly, could be considered to reduce security in increased maintenance costs. – JesseM May 31 '18 at 17:44
  • 2
    @Dennis It was intended to be a non-sequitur to point out the absurd unlikelihood of creating a homebrew function that happened to be the identical of AES decryption with the same key as used for encryption hardcoded in it. – forest May 31 '18 at 23:22
  • 2
    @Anders, I think the concern is not f(AES(x)) leaking information about x (which would indeed point to a mathematical vulnerability in AES), but f(AES(x, key), key) leaking information about the key; if the key was reused for both f and AES encryption rounds. – sehrgut Jun 1 '18 at 13:15
48

There is a risk involved when you apply your custom encryption algorithm first.
This is based on the fact that an encryption like AES does leak information about the length of the plaintext.

Suppose the extremely hypothetical (and unpractical) example where you custom algorithm 'encrypts' a single byte plaintext like 0x40 into 64 zeros and a two byte plaintext like 0x02 0x00 into 512 zeros.

When you encrypt this using AES an attacker will still know the length of the custom encrypted result just by looking at the length of the AES encrypted text.
With this information the attacker can 'decrypt' this into the original plaintext without having the AES encryption key.

In summary: a very bad custom algorithm can indeed harm the security of a following AES encryption.

  • 3
    That's an interesting example. Like you said "extremely hypothetical" – user3280964 May 30 '18 at 20:16
  • 34
    Is it, really? – forest May 31 '18 at 1:53
  • 8
    You are right. This proves that if you apply the custom encryption first it's going to end badly. This doesn't answer the second (more important) part of the question. Do similar vulnerabilities exist when custom encryption is applied at the end? – user3280964 May 31 '18 at 5:13
  • 3
    @user3280964 Nope. From a purely cryptographic point of view, the worst that could happen with an independent custom encryption algorithm at the end is that it could be inverted and the attacker could obtain the AES ciphertext. – forest May 31 '18 at 23:23
  • 4
    @user3280964 Bear in mind that more code means more security risk, no matter what that code is. You can't cryptographically weaken soundly encrypted data by running your custom algorithm on top, but you can practically weaken the system as a whole if you have any bugs in all the code that implements your custom algorithm. – mtraceur Jun 5 '18 at 4:12
32

The first question to ask when assessing any sort of security system is "Who is the attacker, and what can they do?" At the low end, you're against an attacker who can just see the encrypted output of one of your messages. At the high end you're up against an attacker who knows the plaintext and ciphertext pairs of hundreds of other messages, who can coerce your computer into encrypting other messages with the same key, who can monitor the power lines to your building and measure the tiny variations in power draw as the encryption process proceeds, and all kinds of similar attacks. Standard encryption algorithms are vetted to make sure they're resistent even to these powerful adversaries.

If you use your own encryption first, even if the second layer is military grade encryption, you risk decreasing your security compared to just using the standard encryption.

For example, suppose that you are encrypting text. You first use a simple substitution cipher and then use your choice of heavy hitting algorithm. Unfortunately, substituion ciphers are inherently reliant on doing array lookups from private information (i.e. your plaintext and your key). This makes them vulnerable to timing attacks: because of the way CPU caches work if two letters in the plaintext are close together they will likely encrypt faster than two letters that are further appart. That doesn't sound like much information, but if an attacker can watch enough encryptions they can potentially work out what the plaintext is without having to crack AES or whatever you're using on top.

If your custom encryption layer does not interact at all with secret information: i.e. it works with data that has already been securely enough encrypted for transmission and doesn't touch the keys of the standard encryption, then it may be safe. At least, it won't be possible to bypass the standard encryption. There are still risks, though, because every extra piece of software carries additional risks. Perhaps a bug in your custom layer lets a remote adversary run arbitrary commands on your machine; then they could just email themselves the plaintext!

The take home message, then, is that having one lump of excellent security code that is well vetted and bug free like a standard encryption function can still be undermined if you put a buggy home grown program (of any sort, encryption or otherwise) onto the same system.

  • 2
    Well done. Keep em coming. – Anthony Jun 1 '18 at 6:22
14

Simple. Don't do this.

First of all, Encryption algorithms are designed so that you, me, and everyone around us can look at it and try to break it. You can quite literally look a the math for AES. A lot of smart people have invest time in validating AES. This is a tenet of good encryption and why we can trust it.

Second, Obscurity is never a "Security" option. It doesn't provide any benefit. If you actually believe this, than I refer you to "Schneier's Law".

Anyone can invent a security system that he himself cannot break. I've said this so often that Cory Doctorow has named it "Schneier's Law": When someone hands you a security system and says, "I believe this is secure," the first thing you have to ask is, "Who the hell are you?" Show me what you've broken to demonstrate that your assertion of the system's security means something. - Bruce Schneier, 2016

All you've done is:

  • Introduce unnecessary risk into your application
  • Increased debug time for issues
  • Added CPU time for a worthless function

The most basic principal in Security. Keep It Simple Stupid (KISS).

  • 5
    The reason why I'm considering this extra layer is to cover additional threat models. Let's say we have the following threat models: A) Bad actor with fully automated decryption/cracking capability of AES. B) Bad actor with capability A plus resources to manually decrypt things that fully automated method can't. For A I would argue that the extra obfuscation fully defeats that threat. By definition as soon as human eye balls and fingers have to intervene we are now dealing with threat model B. In threat model B you are likely to be ignored because of shortage of qualified cryptographers. – user3280964 May 30 '18 at 21:24
  • 8
    In other words AES defeats manual decryption efforts. Obscurity defeats mass-automated decryption. Combining the two covers both threat models. – user3280964 May 30 '18 at 21:28
  • 20
    Obscurity can keep the opportunist attacker away (one that does not attack because he has a target in mind, but is looking for the easiest targets). – rackandboneman May 30 '18 at 23:56
  • 2
    @rackandboneman Obscurity is not keeping them away. Good configuration management is. – Shane Andrie Jun 1 '18 at 14:09
  • 4
    Depends on what you mean by "obscurity" and what you're trying to protect against. Tinted windows in your car don't protect you from bullets, but they do protect you from random passersby seeing you pick your nose. – barbecue Jun 1 '18 at 18:42
9

Security by obscurity is good, sometimes even great, as long as it's not the only layer of security you rely on. But in your case I'm afraid it's not worth it.

Take GPG for example. If you use GPG to encrypt a file with AES, it will have a header that allows an attacker to recognize it as a GPG file encrypted with AES. If you then use custom encryption to encrypt the GPG file, you will get a file that might be unrecognizable. An attacker might wrongly suppose you are using whatever kind of encryption (and lose a lot of time for nothing), or might suppose you are using your custom encryption and then decide to use cryptanalysis techniques to analyze your file (and manage to decrypt it if he's good enough or your algorithm sucks enough). On the other hand, if you do it the other way around (first encrypt with custom algorithm and then with AES), the scenario doesn't change significantly after all.

That said, I'm afraid it is not worth it because custom encryption has a huge con anyway: you will have to store your encryption software somewhere, and your software is totally custom, meaning that if for whatever reason you lose it (lost or stolen backups, etc.) you are screwed and you won't be able to recover any of your data.

A better solution might be to use several layers of encryption using algorithms that are already known and available. For example GPG seems to support twofish, camellia, etc. I have no experience to know which algorithms should be considered the safest though, apart from AES. Anyway, just as an example, AES -> TWOFISH -> CAMELLIA is probably a much better solution that your AES -> CUSTOM_ALGO. Of course provided you use different strong passwords for each encryption layer!

  • 5
    Or... They'll exploit a bug in your program and get a local shell or tamper with the file due to your custom crypto not using any authentication... – forest May 31 '18 at 1:39
  • 3
    @forest, how does the attacker know how to successfully exploit a bug in a custom program that they are not even aware of, have no chance to see it, and maybe have no chance to provide any input into it? Your logic could apply to any custom program then, or even any program at all. I think that if an attacker has the ability to exploit the custom program, chances are they are able to do it because of other very serious vulnerabilities that might allow him to steal the AES key anyway, regardless of custom encryption. So I think what you depict is an unrelated attack scenario. – reed May 31 '18 at 9:10
  • 3
    @reed What if, for example, the attacker is an ex-employee of the company that wrote this software? Or they are able to obtain and reverse-engineer a binary of the software? Whatever cryptographic algorithms you are using you should assume the algorithms themselves are public knowledge and only the key data is private, because it's much easier to keep key data private than it is to keep code private (that people have to be able to view, by necessity, to actually write it)... – Sean Burton May 31 '18 at 13:28
  • @SeanBurton, sorry, I was assuming the software had to be written and used by a single person, for private use. It isn't really clear to me where and when exactly the OP wants to use custom crypto. If custom crypto to be used by a whole company then yeah, it's a totally bad idea for even more reasons (like the one you mentioned). – reed May 31 '18 at 15:19
5

OPSEC?...

Putting additional barriers between critical information and an adversary is a good idea in general, provided it is done effectively.

Obscurity as a meaningful security measure is addressed by the principle of OPSEC (operational security). In short, this entails depriving an adversary of any information which could help them compromise your organization via either social or technical methods.

With good crypto, however, keeping the keys secret is sufficient to protect your data. Any additional technical layers must be justified on their own technical merits.

...Or bust

Cryptography is a very complicated mathematical discipline, and small mistakes can have enormous consequences. When dealing with cryptography, assume the value of an algorithm is virtually zero unless a trustworthy expert indicates otherwise.

In the US, the NSA and NIST are the official assessors of cryptographic algorithms and implementations, respectively. If you don't know where to look for opinions or want a reasonable basis for internal policies, start there.

In practice, wrapping good crypto is pointless. Once an attacker encounters good crypto, they will generally pivot and attack endpoints where the unencrypted data is exposed.

This could be the web server where users supply information, the SCADA system that feeds into your database, or the workstations where employees analyze or monitor the information. If you have a data pump that pulls information from a database and converts it to another format for delivery to a vendor/customer, that's another big target. Alternatively, they could go searching for your keys.

On the balance

Adding features to an application has a non-zero chance of introducing bugs and complicating troubleshooting. This can lead to extended downtime when something goes wrong or slower responses to security vulnerabilities in the application. Due to these factors, a security feature of little-to-no value is a detriment rather than an improvement.

If you wish to protect yourself against a potential attack on AES, then you should simply use another crypto standard based on a vetted algorithm rather than developing your own. You will get better security with less effort.

  • It might be helpful to describe "justified on their own technical merits" as serving a useful purpose which has nothing to do with cryptographic strength. For example, compressing files prior to encrypting them should not affect the difficulty of an attacker decrypting them unless the encryption is really bad or the compression process enables side-channel attacks, but may have the benefit of reducing the amount of data that needs to be transmitted. – supercat May 31 '18 at 17:34
3

There are two main issues with the security and obscurity mode.

  1. Obscurity can interfere with security. If you roll your own crypto it is very likely to be flawed. Even the best systems have flaws, despite the leading experts having worked on them for decades. The chances of you being able to do better than the security community is low.

  2. Few things are really obscure. There aren't many new ideas in crypto and whatever you do is likely to be a variation on something that has been done before, so it's not actually that obscure. Minor changes are something attackers are used to handling.

  • 2
    There may not be many good ideas in crypto, but it's easy to come up with a bad encryption scheme that's completely different from existing ideas. Even if it's crackable by a human in five minutes, as long as it can't be cracked by any existing automated attacks it still adds a bit of security if no human ever bothers looking at it manually. – leftaroundabout Jun 1 '18 at 16:14
2

It's easy to see that the first variant (custom algorithm last) can't compromise the security of a well-known encryption, since if it could, it would be a method for cracking the well-known encryption. After all, attackers could also run the custom algorithm on a cyphertext encrypted solely with the well-known encryption if it helped them cracking it.

The other direction (custom algorithm first) could indeed compromise the security of the well-known algorithm by introducing redundancy into the plaintext input of the well-known encryption, which could potentially weaken that algorithm.

For example, take the following utterly flawed custom “encryption” scheme: repeat the input plaintext twice to produce a “cyphertext”. This leads to an extreme amount of added redundancy in the input of the second algorithm, which could weaken some real encryptions. (Whether it weakens AES or not is a different question.) This has actually happened with the Enigma, where operators repeated a three-letter sequence in its plaintext that made its cyphertext vulnerable to certain kinds of attacks. Quoting from the Cryptanalysis of the Enigma Wikipedia page:

The second problem was the repetition of message key within the indicator, which was a serious security flaw. The message setting was encoded twice, resulting in a relation between first and fourth, second and fifth, and third and sixth character. This security problem enabled the Polish Cipher Bureau to break into the pre-war Enigma system as early as 1932. On 1 May 1940 the Germans changed the procedures to encipher the message key only once.

1

Solution 1 could potentially decrease security if you use the same secret key for encrypting AES and CustomEncryptionAlgorithm.

CustomEncryptionAlgorithm might allow the deduction of the secret key and therefore compromise AES as well.

If you create a unique key for CustomEncryptionAlgorithm you should be fine again.

Something like KEY = SHA-2(AES-CipherText)

  • Should never use the hash of data an attacker has - like the ciphertext - as key material. Even if the attacker is only given the CustomEncryptionAlgorithm output and not the ciphertext, the application of the SHA-2 to the data and its subsequent use as a key can be used to attack the cipher. Use a proper KDF. – davenpcj Jun 14 '18 at 19:36
1

The problem is that it is possible your custom encryption function has a bug and it somehow reduces the security. The only way for you to be sure that your CustomEncryptionAlgorithm isn't reducing your security is to have researchers who are smarter than you, me, and the rest of your team combined to pour over it and check your work. But then it isn't any way obscure at all.

To make an extreme example:

public string CustomEncryptionAlgorithm(string plaintext) {
    var ciphertext = plaintext.shuffle();
    return "password";
    return ciphertext; // ya ya. I know. You'd never make a bug that obvious.
                       // tell it to the guy who did the 'goto bug' at apple
}

You'd get something like this as a result if you used this scheme:

  • AES -> CipherText -> CustomEncryptionAlgorithm-> CipherText :

    'abcABC123!@#' -> 'password'
    'correct_horse_staple_battery' -> 'password'
    'password' -> 'password'
    

On the other hand, if you used the other scheme:

  • CustomEncryptionAlgorithm -> CipherText -> AES -> CipherText

    'abcABC123!@#' -> '8ZnO44trPK48kqr3rDxkdQ=='
    'correct_horse_staple_battery' -> '8ZnO44trPK48kqr3rDxkdQ=='
    'password' -> '8ZnO44trPK48kqr3rDxkdQ=='
    

Slightly better, maybe. But still not any good in the slightest.

  • I disagree. The 2 concerns are: (1) OP implements a non-invertible encryption algorithm which leads to data loss; and (2) the encryption algorithm uses data other than its plain text input. For this purpose, ROT(k) is OK because it adds an extremely small positive amount of security. – emory May 31 '18 at 18:19
1

As has been said: Who is the attacker?

This is really what you must find out and define first!

Is it your technophobic grandma who has problems with the TV remote control? Your little (and very nosy) sister? Your dad, who actually programs and administrates computers --- and knows how to remove a hard drive? The local school bully who works you over? Organized crime? A company which can spend millions on renting cloud computers for a cracking attack? A third world dictator? FBI, NSA or equivalent agencies in your own country? Some world power that has an excellent spy service and will not mind body bags to get you?

Second: What extra, real security would your 'encryption'1 give you against your attackers? Unless you can make a very good point why encrypting the ciphertext again[2] is needed or very helpful, you should not do so. More complexity does not mean more security, it means more bugs and more chances of things not working properly?

And why won't attackers not break or sneak into your house and place some spy software or key logger on your computer? Are you sure your encryption will be any help against rubber hose cryptoanalysis[3], or if there is no chance of that, who is powerful enough to break AES but unable to reach you or your dearest and nearest?


  1. Let's just say your chance of getting the encryption strong and secure is about as high as my chance to walk on the moon; with encryption, not only must everything work with the right key, but nothing may work without it ...

    I do not know of any cryptosystem built by someone outside the field that did not crack when being checked by experts, and how many strong encryption systems invented by the best experts have been broken?

  2. One example would be the communication in WWII from Germany to the submarines. Some very secret communication was "officer only", which was encrypted, meta-information was prepended and both were then encrypted by the regular key and sent. On receiving the radio/Enigma operator would decrypt the message and pass the meta information and still encrypted block to the officer. (see here for example)

  3. "in which a rubber hose is applied forcefully and frequently to the soles of the feet until the key to the cryptosystem is discovered, a process that can take a surprisingly short time and is quite computationally inexpensive.”

0

Short answer: It can be, but not in your example.

Long answer: Other answers have adequately covered why your example is not all that helpful and can hurt. That being said, obscurity can be very helpful. Case in point: Don't put SSH on port 22. If you change from 22 to 2222, you will have a tiny fraction of the brute force attempts you would otherwise. If you change to port 10000+rand(1,55535) they'll likely disappear completely. Another example is port knocking. Basically, obscurity is mostly useful in the connection process, not in encryption or passwords.

  • Do not put SSH SSH on 2222 or any port above 1024. Doing so will bind it to an unprivileged high port. – forest May 31 '18 at 23:30
  • 2
    @forest So? I know of no ill effects of that. – Duncan X Simpson Jun 1 '18 at 1:14
  • It means that an unprivileged user can bind to it, which allows a local DoS (crash, etc) against the SSH daemon to enable the attacker to bind their own daemon and exploit the client. See this for an example of why it's a bad idea. The same applies to other daemons like the Apache httpd. That's why all these privileged daemons bind to low ports. – forest Jun 1 '18 at 1:18
  • 2
    In principle this is right, but usually ssh does very strict key checking alerting you if something is wrong. And I never saw a crashed ssh daemon. And from experience I can tell that even port 2222 (which may be common for this purpose) decreases the number of attacks to absolutely zero (since many years). But you can use another privileged port which will not be used by anything you want to run. – allo Jun 4 '18 at 8:00
0

You refer to the second algorithm as obscurity, so you seem to assume it does not add any security.

If it actually does not add security, why use it? Use the second password in addition to the first one for the aes encryption. A password twice as long is much more secure than two passwords (example: 2^2+2^2 = 4 vs. 2^4 = 16), especially when the second algorithm is insecure anyway.

0

If done properly: sure. The question is how do you know it's proper? You certainly can write something that leaks information through various layers. Most obvious one is length and repetitions. Also using the same key might be problematic.

It's just too hard to tell what's proper and whatnot. Sure, you could argue that using ROT13 before AES makes it a little bit secure and my intuition would say... well it at least can't hurt but I don't actually know.

My intuition would say that if you apply an algorithm that produces output indistinguishable from randomness while not changing the length and then apply AES it at least shouldn't reduce the security of AES.

0

Obscurity is a part of a security (general), and is orthogonal to the security (AES encryption in your case). Obscurity defends against different threats than crypto. Take passwords for example - crypto hashes are required to prevent leaking plaintexts from server, TLS prevents from eavesdropping network traffic, while obscurity being a password form filling with [*******] prevents from peeking your plaintext by random spectator. So you cannot do security by obscurity (only), but security with obscurity is usually the way to go. In real world the obscurity means hiding the metadata, e.g. mailing with BCC instead of CC/To or blocking 3rd party cookies.

0

It's bad, bad, bad.

There is an anecdote from some Far East mobile protocol for SIM initialization or something similar, that used a combination of RSA (good, secure, and strong) and something homebrewed.

The decisive weakness was not only the homebrewed "additional" encryption. But the problem was that in fact there was a special case when the additional method was associative with RSA, which caused partial key bleeding. I don't remember details, as I was told them as a lecture anecdote many years ago.

But the point stands: with an improper homebrew addition to an otherwise secure protocol you might make the protocol weaker than its original version. So, that's a major no go.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.