I know this is usually frowned upon, but supposing you were to roll your own encryption cipher (completely on your own or with a small group of friends, with no peer review possible), what could you do to make your custom algorithm as safe as possible (or rather, to keep the danger of using it to a minimum)? Similarly, what would you do for something like a hash function to make the probability of it having some trivial collision or preimage attack as low as possible?

I understand that there would be no way to approach the security of algorithms like AES, and I have no plans to use custom algorithms for anything important, but I thought it would be informative to think of what some best practices would be for rolling your own cryptosystem if you were to do so.

closed as too broad by ThoriumBR, forest, Tom K., multithr3at3d, Serge Ballesta May 9 '18 at 9:57

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    "keep the danger of using it to a minimum" is a phrase that could be interpreted a few different ways. One way to answer that is to make it, then shred all evidence of its existence. Then there is no danger of it being used. I think "use it safely" might work as an alternative? – schroeder May 2 '18 at 15:36
  • 1
    @schroeder Well, sending it to /dev/null is the correct way to use your own crypto. But I think what he wrote makes sense. He explained he wants to use it and is asking how to minimize the danger created by that. – Peter Harmann May 2 '18 at 15:38
  • 5
    @DEP do not cross-post on StackExchange: crypto.stackexchange.com/questions/58876/… – schroeder May 2 '18 at 15:43
  • 1
    I think the question contradicts itself. On the one hand the OP knows to better not create own cryptography unless one has a considerable expertise in this field - which the OP does not have. On the other hand the OP wants to to it anyway and asks how to do this with the fewest problems. Only, in order to really understand the problems and to make sure you don't run into these when designing your own cryptography you would already need to have some serious understanding of cryptography, which you don't have. Therefore I propose to close the question as unclear what you are actually asking. – Steffen Ullrich May 2 '18 at 16:52
  • 2
    Possible duplicate of Why shouldn't we roll our own? – jknappen May 4 '18 at 9:24

Revised answer

In general, you may want to start by basing your cipher (mostly relevant for asymmetric ciphers) on a known hard problem. You may want to make mathematical proofs verifying your cipher has the desired properties and breaking it indeed requires solving the known hard problem. What kind of problems to choose was explained more on Crypto SE. By creating these mathematical proofs, you will make sure your cipher is not easily breakable directly.

The next step may be to look at side channels. Most notably timing attacks. If your cipher takes different amount of time for different inputs and keys, you may want to at least add code to either make it take the same amount of time always or just add random amount of time by active waiting.

There are many other side-channels and considerations a widely used cipher has to take into account, but this should get you started. Of course, making all these mathematical proofs is very difficult and even then, it does not give you any guarantee the cipher can't be broken. Even if the cipher is secure on its own, in combination with other cryptographic primitives, it may turn out to be easily breakable. That is why rolling your own crypto is such a big taboo. Getting it right is very very difficult and you will not find out you did something wrong until someone breaks it, or if you are unlucky, not even after that.

  • 2
    This is a quibble, but I am not aware of many practical asymetric cipher that actually use an NP-hard problem (lattice-based cryptography being a notable exception). – cody May 2 '18 at 18:37
  • @cody Right, for some reason I thought integer factorization is NP hard, but it is not known whether it is. – Peter Harmann May 2 '18 at 20:10
  • 1
    @PeterHarmann Even for symmetric ciphers, the proof is not that it's hard, but that the difficulty is reduced to at least the difficulty of a hardness problem. Like the Rabin cryptosystem is proven to be as hard as integer factorization. For symmetric ciphers you don't need to do any reductions like that, although you can create proofs for other things (such as certain properties holding in certain conditions, like having a flat keyspace or being IND-CCA secure, or having desirable properties if an underlying core function is a good PRP). – forest May 5 '18 at 22:40
  • 1
    Crypto is extremely complex. With very few exceptions (such as the OTP for confidentiality or poly1305 for integrity), there is no way at all to prove that a cipher is information theoretic secure, only that certain properties hold given certain assumptions. – forest May 5 '18 at 22:43
  • 1
    Er, I meant "even for asymmetric ciphers". Curse the 5 minute editing limit... – forest May 5 '18 at 22:50

Obligatory disclaimer.

You really shouldn't roll your own.

It's not like normal programming, where you can reinvent your own wheel with a different shape or color.

Any small detail can introduce a weakness you don't even know about, and which can completely break your scheme.

As your skill in designing crypto is (understandably) low, the likelihood is high. The more sensitive are the data you entrust to your scheme, or the longer you use it, the higher is the impact.

High likelihood + high impact -> high risk.

Not even professional cryptographers invent schemes and use them straight away.

Plus, even good algorithms can be implemented badly. Again, implementing algorithms (even good algorithms) is something you shouldn't do, unless it's really necessary. If anything, inventing your own algorithms is even (by far) less necessary.

Do not put this stuff in production, period.

Do not use this stuff for any information you care about, period.

That said, if you want to design something for learning purposes only, @PeterHarmann's advice is valid.

I'm just going to expand a bit on symmetric algorithms.

Most block ciphers are based on either of two approaches, SPN or Feistel networks. Choose one.

If you go with SPN, the round number can be relatively low, but remember you need a quite large P-box.

You also need to design a way to obtain the round key.

Now comes the fun part... break it!

I don't design crypto, but if I had to, for whatever unlikely reason, I would check that the scheme is reasonably secure at the very least against linear and differential cryptanalysis.

It's not a piece of cake, but you're doing it for learning purposes, right?


There are certain attributes that ciphers need. The Ciphertext indistinguishability property says that pairs of output of a cipher can't be distinguishable from each other. So perform tests on the outputs, comparing them.

Another property is that of diffusion: a one bit change to the input of a block cipher should have a ripple effect that results in an average of 50% change of bits to the output. That should be testable.

Use fuzz testing on the inputs to provide a variety of samples to compare.

Basically, what you're looking for are all the attributes that make ciphers secure, and trying to find ways to write tests against those.

Next, follow that up with attacks. What are the classic attacks on ciphers similar to yours? Start with reduced round attacks and learn how many rounds it's going to take to adequately diffuse your ciphertext. Understand how your diffusion works, and structure your rounds and mixing accordingly.

And like everyone else here, I do feel compelled to end with Shneier's Law: "Anyone, from the most clueless amateur to the best cryptographer, can create an algorithm that he himself can't break." This isn't just a bunch of web amateurs telling you not to roll your own; this is one of the most preeminent cryptographers on the planet reminding other talented cryptographers not to roll their own. Heed his warning.

Not the answer you're looking for? Browse other questions tagged or ask your own question.