I'm a newcomer to cryptography and was wondering how often systems are attacked by cracking cryptographic systems in the real world. Is it practical to study cryptography as a hacker?
Crypto systems, if properly done, are usually airtight in terms of mathematics. This means that in most cases a polynomial efficient adversary (that is an adversary that can run for say 100 years) only has a negligible probability of success (that we don't care about).
Unfortunately, while a cipher may be strong on paper, it can be often wrongly implemented so you can gain info about the key or plaintext by observing the device or attacking the random number generator or any other side channel.
Other than that, organizations and vendors who respect themselves will update their software when ciphers previously thought to be safe are proven to be anything but.
I am in no way an expert in cryptography but I have studied it for a bit and it opened my eyes to the beauty and subtlety of mathematics, something I previously thought to be unecessary. It's also very interesting to see how the things we rely upon for our safety and security work, and that's more or less what hacking is all about. My answer would be, hacker or not, do study crypto for its own merits, but it can certainly aid you in your work as a pentester or programmer.
Unless it's a known-weak algorithm, probably never -- crypto is specifically designed to withstand direct attacks, so trying to break crypto by brute force is like trying to incapacitate a tank by shooting at the place where the armour is the strongest.
You would normally attack communication channels and seek ways to defeat cryptography by exploiting implementation weaknesses (side-channel attacks, social engineering, key exchange or key derivation mechanisms, etc).
From my experience it depends where and what the crypto is and at what point you're attacking.
If your attacking the cryptography itself for flaws in the implementation:
- If its cryptography being used to secure SSL/TLS communications, then generally, there are misconfigurations in place which significantly reduce the amount of time required to crack any intercepted exchanges.
But if your attacking the result of poor quality encryption and weak passwords:
If its file encryption, its usually possible to brute force the password assuming you have some knowledge of the company.
If its cracking user accounts encrypted/hashed passwords stored in a database of some kind (using a weak algorithm like unsalted MD5) then its usually very easy as most people choose weak passwords.
Actually finding flaws in things like AES, MD5, TLS/SSL etc is quite hard (lots of experts have spent a lot of time reviewing them) however looking for vulnerabilities in implementations of them is a bit easier. Furthermore, looking for vulnerabilities caused by human issues (like poor password strength) is the easiest option available.