Consider a network for a web-application with 1 webserver and 1 MySQL database, both on physically different servers and both running Linux. The database stores mission critical data and this data is and has to be manipulated by the web-server.
Now, obviously, we want to make sure that only authorised changes are made to this database. This is largely implemented in the application and a part is handled by creating database users with limited amounts of authorisations.
There do remain gaps with this approach, however. For instance: the system administrator is not authorised to make changes to the database independently. Nor would any engineers with deployment rights. But, they obviously would be able to do so by grabbing the database credentials from the web-server and logging in from there.
Any changes made this way would basically be anonymous as they would be obscured by many other changes which would make it hard to detect.
It's (probably) not feasible or wise to implement something that will actually 100% prevent the sysadmin from being able to make changes this way. Sysadmin and others should be able to make data and schema changes using their own accounts as the application is actively developed. However, they shouldn't be able to do so without anyone noticing.
Here are some possible strategies to prevent insiders from changing data anonymously:
Implement audit logging on the system to report any programs executed. This would find any direct calls to
mysqlcli and maybe some sketchy looking scripts. Any successfully performed changes would still be hard to find as scripting would obscure the actual performed changes.
Use client-certificates on the database connection with pass-phrases on the private key. This means a sysadmin is no longer able to reboot a machine without the presence of an application manager. Other controls are probably needed to protect keys in memory, sysadm being root and all.
Implement audit logging in the database. The web-application causes a significant amount of database traffic, so any malicious changes with the application database user is unlikely to be detected. This means monitoring can only feasibly implemented for interactive database users.
Use IPTables to lock traffic to the database to the application user. Would at least create an audit-trail of sysadmin becoming the application user (sudo) or changing the firewall to allow traffic out. Not foolproof, since any changes made using application database user would still be hard to find.
Implement SELinux to limit database traffic (packet labeling through SECMARK) to the application domain and further securing the deployment and build chains to prevent/catch unauthorised changes. If sysadmin does need network access to database, this may be audited through
auditallowbut administrative access should be through another user which is audited. Any system changes or SELinux changes would be alerted upon. This is obviously a high impact change with a lot of extra requirements that requires a lot of knowledge.
Beyond just trusting the sysadmin, how would you approach this? Would the above things work? What other things could one do to make sure a system administrator or similarly privileged user can't access the database unnoticed?