As always, it depends. Potentially every piece of code operating on user-controlled data is exploitable if it contains bugs in implementation (it does). However common pdf attacks are targeted at popular viewer applications, not automated converters, so your potential adversary is a targeted and capable hacker. They are less common, much more dangerous and are attracted to financial organizations ;)
Can the process of converting a potentially malicious PDF file be exploitable?
Yes. Here your security depends on the security of your chosen library. For example, image processing library Imagemagic and video processing library ffmpeg have 535 and 314 CVEs respectively. Some of those allow the attacker to execute arbitrary code on your machine.
The most popular (AFAIK) pdf processing library (Apache PDFBox) is written in Java, so it's relatively hard to perform memory corruption attacks, however, it's not immune from all attacks and has 4 CVEs to show that. It's unfair to compare the numbers here since PDFBox is much less popular than the other two libraries.
The fact that PDF is one of the craziest formats ever invented (here, play some breakout.pdf) doesn't help with security.
The main takeaway is that just like a malicious image can exploit imagemagic and malicious video can exploit ffmpeg, malicious pdf can exploit your chosen library.
Or convert the PDF to PDF/A
I would imagine that converting to PDF/A allows you to save more useful information for later processing than converting to plain image, so that's a plus. As for security, this reduces the attack surface for the readers that would subsequently open the converted PDF/A and changes codepath inside pdf converter, but it's impossible to say whether this codepath is safer or not.
Either way, you need to care about the security of your converter. The principle of least privilege is very useful here. Extract your pdf converter into a separate process/container and allow it to take pdfs from one dir, convert them, and store them in another dir. No process spawning, no network access, execution timeouts, etc.
Is the presence of a network anti-malware product going to help?
I'm skeptical. Despite how much AV vendors push "behavioral analysis" technologies, the bulk of AV job is just comparing files with signatures of common viruses. Attack on specific pdf parser will probably be quite uncommon, perhaps even 0-day, so you shouldn't rely on AV catching it.
Some kind of network activity analysis could make the attack harder to pull off unnoticed or even prevent it, but nothing can guarantee security here.
