Do I need to be a "mathematical whiz" to be a hacker?
Do you need it to just run hacking attacks or simple social engineering? No math is needed.
However, if you want to be an expert and really understand modern cryptography, you'll need to learn some rather advanced/obscure math like modular arithmetic, Fermat little's theorem, discrete logarithms, etc. You don't need to understand cryptography to use it, but if you want to implement it, or search an implementation for flaws/vulnerabilities, you'll need to understand it. Have a look at Handbook of Applied Cryptography and you'll see there's a fair amount of math, that likely is unfamiliar to you.
Say you know that all passwords in a system are comprised of 12 random characters (case-sensitive including special characters) (and some salt) and you have the salt+hash, and want to break it with a GPU that generates a billion hashes per second. Is it worth trying to break? What about if passwords were 6 random characters? The answer from basic probability/counting/conversions is don't try with the first one (should take million years); but the second should be trivial to crack (done in under 5 minutes)? Some security algorithms (like Microsoft's LM-HASH ) didn't recognize this basic fact that two N/2-length passwords could be cracked in ~10 minutes while one N-length password may take a million years.
Its hard to say a priori what you don't need to know; as sometimes understanding something can help tremendously, see relevant (and today's) xkcd. Math is also great at just teaching you to think very logically; sort of like how doing a push-up isn't very useful on its own -- but being in shape/strong from doing push-ups is quite useful.
I would say that, rather than mathematical skills, you require logical skills.
Maths is important, of course, but not in the sense that you could apply mathematical concepts to hacking (examples: derivatives, integrals), but meaning that it could give you the tools and the training needed to help you with your problem-solving skills.
Computer science can be divided in several sub-classes. I'd say security is not the most mathematically challenging field among those, in contrary to fields that research numeric methods to solve problems, for example.
That doesn't mean that mathematics are unimportant at all from the perspective of a security researcher (or hacker, if you will), but like user1301428 said: you have more need for logical skills as well as creativity and imagination.