As others have said, reading/printing is your major risk area for buffer overflows.
However, it is possible you could use the bignum library to spray the heap. For example, MPIR/GMP use a bignum structure which looks a little like this:
off_t describes the number of limbs and its sign describes the sign of the overall number. Then each
mpn_limb_t is typedef'd to an appropriate size for your platform.
Clearly, this cannot be and is not stack allocated, so you're basically writing all of these numbers (
mpn_limb_t) to the heap, allowing you to heap spray which won't directly exploit the library, but it may facilitate another exploit.
The sanitization process for this is to not trust a binary serialization of your number, since every bit is used to store the number in raw form and put that into memory that gives you the full range of opcodes. You really need to read the number from a human-readable format, which whilst less efficient, does make it somewhat more difficult to produce useful shellcode (unless you can write something with the opcodes represented by the ascii of 0-9, which I doubt).
Addition, multiplication etc "cannot" overflow, in the sense that gmp/mpir at least will fail to allocate sufficient memory for the target
mpz_t and stop there.
There should be (there definitely is a print function) a function for reading integers from text representation. I would ensure you use this, and use the corresponding print function for output. You will then be able to handle numbers up to the limits imposed by your memory while not writing arbitrary nonsense to memory. You may wish to check the documentation for these functions to check their behaviour on non-integer/decimal characters.
Floating point exceptions shouldn't be caused by any mp float library, because the internal size/representation of the float in question can't be handled by the processor anyway. If you have a look at the algorithms (see algorithms for APFPA used in this kind of code, you'll see they use normal precision floating point operations. Any decent library really should ensure they do not cause a floating point exception when working on "limbs".