In the context of security, everything is a trade off. More on that in a second....
What are the attack options for breaking into your encrypted volume? They are, basically:
Brute force search of the keyspace for the key utilized by the symmetric encryption primitive.
Brute force the passphrase by trying every possible password utilizing your volume's specific salt and key derivation parameters.
Evil maid, rubber hose, key logger, or other "non crypto" means of securing your keymat.
You should worry about number 3. Because its the obvious attack vector.
No one is going to succeed in brute forcing the key used for the block encryption primitive.
From a basic security perspective, I remain convinced that even 128 bit keys are fully secure if the implementation is not broken. A brute force search in that case has to exhaust on average half of a key space that has 340,282,366,920,938,463,463,374,607,431,768,211,456 possibilities.
For a 256 bit keyed primitive, you're talking about 115,792,089,237,316,195,423,570,985,008,687,907,853,269,984,665,640,564,039,457,584,007,913,129,639,936 possibilities.
So even with 256 bits of key security you're already reaching numbers that approach the number of atoms in the known universe.
So that's out.
What you're worried about is someone trying every possible pass phrase.
Now, remember that LUKS uses some salt + PBKDF2 to stretch your pass phrase to the size of the key space. What this means is that even a password of "a" can map to any possible spot in the entire key space. So attackers aren't going to have a rainbow table that will help here.
If they're going to brute force the pass phrase, they will have try all the possible inputs, running each through the key derivation function with the parameters used on your volume (salt & iterations). That takes time. And, if memory serves, LUKS actually sets the iterations to take a specific amount of time per "try."
What this means is that you simply need a pass phrase length that has enough entropy to make it unfeasible for an attacker to try a sufficient number of possible phrases to locate your phrase. The attacker can of course use GPUs or specialized ASIC hardware to speed this process up (its why folks are experimenting with things like Scrypt to make that harder), but you will be more than protected with 20 to 30 characters containing lowercase, uppercase, numbers and keyboard symbols (including symbols from shifted numbers and other symbols on the keyboard).
This brings me back to my initial comment. Its important to keep in mind the trade offs. You can try to come up with a pass phrase that is equivalent to the difficulty of your 512 bit key space, but it will IMPACT your behavior. You might find yourself not dismounting / logging off when you get up to get a cup of coffee because its such a pain to get that pass phrase right when its time to log back in. You might store the long pass phrase on a phone or piece of paper. If you store it in a password manager, you will be limited by the security of the password manager and not the security of the length of your pass phrase.
Meanwhile, once an attacker has to try 20 to 30 characters of pass phrase utilizing 93 possible characters, the attacker is facing (effectively) 95 to 150 bits of entropy which puts it outside of the reasonable range of brute force.
And, all of which leads back to that third possibility of getting into your volume. Compared to brute forcing the key to the encryption primitive or trying all the possible pass phrases, the third scenario "COMPROMISE" is really what you should be worrying about. An attacker that gets you to point a compromised web browser at an exploit that gives escalated privileges on your running system can scan the memory for your password. An attacker that gets a rootkit on your system can do the same. If you plug in a compromised USB or firewire device, you're potentially compromising security. And these are all much easier for the attacker than brute forcing the keyspace or pass phrase space.