How to test security of an encryption algorithm?

Question

What are the standards methods to test the quality of an encryption algorithms. Secondly if i am trying to implement an encryption algorithm then what practices should i adopt?

Answer 1

Yes, it is called writing research papers and getting them peer reviewed.

There is no reason you should ever write your own encryption algorithm unless you are in the field of academic research, and then you will still not "use" your algorithm for at least 4 years (that's how long it took to choose SHA-3 between the time when the algorithm was submitted to when it it was selected as the official standard) while other people look at it.

You will make mistakes that you will not even know are mistakes (do you know what a timing attack is? does your algorithm protect against it?). If someone with resources ever wanted to attack your algorithm do you think something you came up with could stand up to a team of PhD Crypto Analysts looking at it with a fine tooth comb for a week, a month, a year?

Use a existing library, please don't make your own.

Answer 2

I agree that do not go for a new encryption algorithm if you want to use it for practical purpose. Since you have not mention what type of cipher you are developing i.e stream cipher, block cipher as there are different classes of attacks. For a research prospective there are number of attacks through which you can evaluate the security of your encryption algorithm.

1. Linear cryptanalysis
2. Differential cryptanalysis
3. Chosen plaintext attack (attacker is given ciphertext of his choice of plaintext)
4. Known ciphertext attack
5. Correlation attack
6. Side channel attacks
7. Weak keys attack

As you see there different variation of attacks which not actually test your algorithms but actually test the its physical implementation i.e. power analysis