Normally I preach that rolling your own custom crypto algorithm is a bad idea. But will it really hurt if it's the outermost layer though? Or will it make security worse?
AES -> CipherText -> CustomEncryptionAlgorithm-> CipherText
I'm thinking that the extra layer will help. Let's say even if CustomEncryptionAlgorithm is bug ridden mess, it can't possibly make things worse. That's because AES output is already indistinguishable from random noise.
On the other hand, something tells me that the following is problematic
CustomEncryptionAlgorithm -> CipherText -> AES -> CipherText
Is it bad? and why?
Please don't comment on company resources spent on security vs obscurity etc (I agree security comes out ahead) I am more interested in understanding the cryptographic theory behind vulnerabilities in this approach.