There's not enough information here (or indeed, possibly anywhere) to answer your question, but I can help point the way to finding the answer.
is it possible for {anybody} to gain remote access to our server and steal files on our server via hacking
This is a question that depends almost entirely on the security of the server software, and to some extent on the permissions it runs with. What server is it? Who wrote it? How much security review has it received? How often does it get updates? How many of those updates are security patches? How quickly do you apply them? What language is it written in? What third-party libraries does it depend on? What OS permissions does the process have? Is it sandboxed at all?
A firewall can be useful for restricting access to vulnerable software, but the buck ultimately stops with the security of the software itself (and, if it does get compromised, then the OS security determines what the attacker can access). If you can help it, don't run software that you aren't confident of the security of. If your software is sufficiently ironclad, a firewall is wholly unnecessary (although in practice it's a good idea to use one to reduce attack surface anyhow; perfect security does not, in practice, exist).
given that all ports are closed on our server except for port 5011 is it possible for a hacker who somehow finds out our IP address to do a port scan
Yep. That's standard network recon stuff, both the IP search and the port scan. Now, depending on the protocol used and the OS / firewall behavior, it might be difficult to determine the status of the port, but in general you should expect it to be possible to find open ports.
Mind you, rejecting all external traffic (either at the server software, or at the firewall) does mean that the port scan would need to come from (or look like it comes from) the forex broker. But you can't completely trust that e.g. their network doesn't host any backdoored or otherwise malicious machines / users.
find out if the trading platform software we are using via port 5011 has a vulnerability in it
Once again, we're back to the sort of questions in my second paragraph. Is your trading software externally known? Does it have any CVEs? Does it identify itself in any way (e.g. via a version string transmitted when you connect, or by consistently using a particular port, or so on)? Is your company known to use it? Is its source code public? Is it available publicly at all?
if so gain full remote access to our server and then steal our sensitive files
Same deal, but about the OS security again. What user account does the server software run as? What permissions does that account have? Does the software drop any permissions it doesn't need? Have you applied the principle of least privilege to the server software? Is there any way (e.g. through a broker process) you could gate access to the sensitive data? Does the server actually need to be on the same logical - or even physical - machine as your sensitive files? What security boundaries are in place protecting those files? Are they encrypted, and if so, where's the key? Do you have access to the files audited?
