I take it that your question is:
How do people determine if a cryptosystem is weak?
Well there are two types of people who need to do this as you pointed out: the ones who made the system and those who want to break the system.
If the process is symmetric, then all we need is that the process is not linear. That is why AES won - it is long and extensive, and it is not linear at all (linear meaning it can't be mapped by matrices).
Every system can be broken somehow, and people need to determine how long that process takes. If the process is symmetric, then we know how to crack it - it just takes a while.
If computers get too fast though, then we need to switch to a new cryptosystem. That's why we left DES - we always new it would be fairly weak to fast computation and when faster computers came out - we had to switch.
So we tried to find a cryptosystem that took much longer to break.
The ones who make the cryptosystem have to know how to break it in every way possible too.
For cryptosystems like RSA and ECC, it might be fairly hard to know how to crack it. The security off of these are based off of what is called: NP-hard problems.
These are problems that are really tough to solve and computers don't do squat to help crack them.
Yes computers can factorize numbers faster than humans, but the algorithms of doing so are VERY slow.
There are many NP-hard problems out there though.
You can take any one of these and make a cryptosystem, and it will likely be secure! That's how they made ECC - they picked a random NP-hard problem, found out that all the computations were fast and easy and that was it!