How to keep backup servers safe?

Question

I use duplicity on a few Ubuntu servers to encrypt backups and send them to a backup server, which then sends another copy to rsync.net, and then, once a week, give it or take, I download these backups to a local server.

The problem I have with this is that to send these encrypted archives to the backup server, each server has a password-less SSH key that allows them to connect to the backup server.

While each server has its own user on the backup server, file changes are monitored with OSSEC and the user only has permission to write to its own backup directory, I still fear that a compromised server — thinking ransomware, to be specific — could damage the backup server as well.

I thought about doing the inverse and having the backup server connect to the other servers, grab what it needs, and then shut itself down, but that seems worse, as a compromised backup server would have access to the entire server inventory.

So, I am wondering what is the best solution to keep backup servers safe? Is there a better software than duplicity to handle this?

Thanks in advance!

EDIT: Few details I forgot to include, might be worth something.

• Backup server is set up with a hardware RAID controller in RAID 10 with 15 drives
• Backup server uses Ubuntu 18.04 with EXT4 as the filesystem
• Backup server is a dedicated server with plenty of RAM and CPU power
• Backups stay inside /var/backups/SERVER-NAME/
• Client servers are all unprivileged LXD containers

Answer 1

Doing 'the inverse' seems like the more secure solution.

If:

• the backup function is the only function that the backup server performs
• all incoming connections to the backup server are blocked
• all outgoing connections (except the connection to the server being
  backed-up) are blocked
• the only process that the backup server performs is the backup process (by way of a cron job)
• all other standard best practices are applied

then the backup server has a very small attack surface.

Futhermore,

• if the backup server only has read-only access to the server being backed-up

then even if the backup server is somehow compromised, it can't tamper with files on the server being backed up, or copy malicious files to the server being backed up.

Finally,

• if the backup server makes incremental or differential backups

then even if the server being backed up were compromised, and files were tampered with, and backups of these files on the backup server were overwritten with the tampered files - 'pre tampered with' files would still be available on the backup server by way of the differential or incremental backups.

Edit July 26, 2020: There has been a fair amount of news over the past few days about Garmin suffering an outage that has lasted over three days to date. A number of news outlets are reporting that Garmin's network has been infected by the 'Wasted Locker' ransomware. According to MalwareBytes, Wasted Locker is specially designed for each victim that it targets, and one of its objectives is to find and destroy all backups. A backup solution like this one would be resilient to such an attack.