I came across this practice, from a big company handling personal customer data, of not allowing developers/maintainers to look straight at the application logs (e.g. good old
tail | grep) by connecting to the application servers through SSH in the production environment. The logs do not contain any personal data.
Developers can still retrieve the logs, but they must do so through a web interface, which is exposed by the server through some kind of plugin. The interface exposes the logs as static files that the developer can download and then look up offline, for problem determination purposes. This is suboptimal from the maintainer point of view because
- it is impossible to look at real-time logs
- the download fails if the file gets bigger than a certain size, in which case you need to ask a sys admin to retrieve the logs manually (or set log rotation to smaller chunks, but then it's often harder to find the chunk you're looking for).
In the integration and/or UAT environments, maintainers are allowed to connect directly to the application servers through SSH, they are given personal users with read-only access to the machine, and can therefore read the logs straight e.g. from the
/logs dir. In the production environment, the company policy is "Only sys admins can SSH to the application server".
The purpose of this question is to better understand the good reasons behind this rule, and the possible alternatives around it i.e. TL;DR
- What could actually go wrong allowing maintainers to access production servers through SSH connection using a personal read-only user?
- Is there any best practice for allowing real-time access to production logs without exposing the system to security risks?