I think this is a good way to look at it:
You have a secret, which may be a 256-bit key, or a password from which you derive that key, or either of those plus other information like which encryption algorithm you used.
The attacker wants to guess your secret. They do this by trying various possibilities until they find the right one or they run out of time, money, or motivation.
You have no idea what possibilities they are trying. In your question, you say "what if all the years he was using the wrong algorithm?" and the only answer to that is "what if he wasn't?" You have no control over that. If you knew which possibilities the attacker was going to try, you could just pick anything not on their list as your secret, and the security problem would be trivially solved.
What you can do, though, is roughly estimate how many possibilities they can try before running out of time and/or money, based on the state of computing technology. This assumes that they don't secretly have access to technology that the rest of the world doesn't, such as quantum computing or a backdoor in AES - which is probably a safe assumption since they would have better things to do in that case than try to crack your password. (Cf Cut Lex Luthor a Check, though see also this rebuttal.)
You can also prove the following result: if you choose your secret uniformly at random (using a high quality RNG) from n possibilities, and the attacker tries k possibilities, no matter what they are, the chance that they'll guess your secret is at most k/n.
The nice thing is that n grows exponentially with the amount of information you have to store/remember, whereas k grows only linearly with the amount of time/money they spend, so it's not hard to make k/n very small.
So, you should choose your secret uniformly at random from a large set of possibilities. A random 256-bit symmetric key is chosen uniformly from a set of size 2256, which is (far more than) large enough.
You can pick randomly from a bag of (algorithm,key) pairs as well, but it's pointless because any single algorithm already offers plenty of choices.
You can pick an obscure algorithm and hope that the attacker won't try it, but that's not picking at random any more, and therefore you can't prove that it helps at all. If there were no other options then this would be better than nothing, but there are other options.
This is the fundamental reason that cryptographers advise you to treat only the key as your secret: there are plenty of keys and keys are the easiest thing to choose at random. You don't need anything else.