First of all, if you had an easy time with the Security+ certification, it sounds like you probably wouldn't have many issues with understanding the basics, and therefore I don't think you should worry too much about your background.
Now, that said, certain branches of mathematics do have direct practical applications in IT/IA:
- Any background in statistical analysis is likely to be invaluable, especially for IA. Risk assessments are a really big part of IT work, not just security related stuff, but even deciding when to replace old hardware, how much redundancy is needed in a given service configuration, and similar things. It also helps with understanding performance data from software (actually understanding how the data was produced is big).
- Basic set theory (more specifically the algebra of sets) is often very useful in reasoning about failure domains, both in hardware, and in software, which is helpful when doing risk assessments.
- Information theory is, somewhat obviously, really helpful if you're doing low-level design of software. It still has practical uses beyond that in designing security policies, but is not as widely used as you might think.
Many other branches of mathematics have much more specific applications, examples include:
- Lambda calculus is rather critical if you're writing code in certain programming languages.
- Much of high-level computer networking is really just graph theory with some extra stuff on top of it. In fact, there's such a strong correlation that many networking specialists would have very little difficulty taking a course on graph theory as they already know a decent percentage of it.
Even without any of that though, the foundations in logical analysis and problem solving that studying mathematics gives you are useful even without any other background.