Steffens Answer is pretty good but I want to get more in the detail in the part of:
How hard is to compromise that server in first place?
How can I secure a domestic server? (basic setup)
As Steffen mentioned it really dependes on your System but given your question you´re Pi is probably not really configured safely
Of course we can believe you already changed the default Credentials of your PI. Now First of all you should install OFW, an easy to use tool for making iptables-Firewall rules. Block everything except the ports you REALLY need.
Then you should install some IDS/IPS/AV Tools like rkhunter/chkrootkit/ossec.
Another neat tool is fail2ban which bans IPs via iptable-rules automaticly after serveral failed login attempts etc.
All of above said tools should use exim4 mail so that threat reports etc. get sent to you via email.
Also you definitely should deactivate password auth for SSH if it´s reachable from the Internet (not blocked by IPtables) and switch to certificate based authentication.
With this setup, a little bit of reading in google and common sense, your Pi should be considered "pretty safe" as "totally safe" just never happens :)
Tutorials on how to setup/configure said tool are widespread if you just search on google
EDIT: Since Raspian is just a Debian-Derivat, most Google Tutrorials on things like "how to setup a secure Debian server" are pretty much straight forward portable to the Pi