In an enterprise environment, should developers/maintainers be able to look at production logs by accessing the application servers?

Question

I came across this practice, from a big company handling personal customer data, of not allowing developers/maintainers to look straight at the application logs (e.g. good old tail | grep) by connecting to the application servers through SSH in the production environment. The logs do not contain any personal data.

Developers can still retrieve the logs, but they must do so through a web interface, which is exposed by the server through some kind of plugin. The interface exposes the logs as static files that the developer can download and then look up offline, for problem determination purposes. This is suboptimal from the maintainer point of view because

1. it is impossible to look at real-time logs
2. the download fails if the file gets bigger than a certain size, in which case you need to ask a sys admin to retrieve the logs manually (or set log rotation to smaller chunks, but then it's often harder to find the chunk you're looking for).

In the integration and/or UAT environments, maintainers are allowed to connect directly to the application servers through SSH, they are given personal users with read-only access to the machine, and can therefore read the logs straight e.g. from the /logs dir. In the production environment, the company policy is "Only sys admins can SSH to the application server".

The purpose of this question is to better understand the good reasons behind this rule, and the possible alternatives around it i.e. TL;DR

1. What could actually go wrong allowing maintainers to access production servers through SSH connection using a personal read-only user?
2. Is there any best practice for allowing real-time access to production logs without exposing the system to security risks?

Answer 1

Hidden Assumptions

I think you may have an underlying assumption here that is causing you to look in the wrong direction. Quoting you (emphasis mine)

they are given personal users with read-only access to the machine

This makes two assumptions: the application is run by a server that you can connect to, and there is only one of them. How do you propose to SSH into a machine and get the application logs if:

1. The application is actually running in a Lambda/cloud function and so there is no machine
2. The application is running in a auto-scaling cluster so no individual "runner" has all the logs anyway
3. The containers running the application are stateless and easily replaced, and so SSH access is not available anyway because it is an unnecessary risk.

I've managed services running under all the above situations, and in no case was it possible to SSH into a machine and fetch logs even if I wanted to! (which I didn't want to!)

Logs

Of course you still need logs, so what do you do? Naturally, the operation team would have configured logs so they get sent automatically to a separate system where they can be inspected as needed by those who need to.

Now granted, the systems you want to access may very well be distinct servers that someone can SSH into, and so it may be that the operations team really is just making your life unnecessarily difficult. However, there are also many, many situations where what you are requesting is simply impossible and for very good reasons as well!

An X-Y Problem

As a result, it sounds like you're approaching this as an X-Y problem. You need easy access to logs, but you are not being provided with that access, so are instead asking for direct SSH access yourself (which, in some contexts, is a legitimately risky thing to grant). Rather than asking to have your needs met in this particular way, you'll likely have better success if you present your current problem to the operations team:

I need to be able to easily look through logs, but the current log setup makes this impossible. Please fix this for me!

This isn't a security issue, but rather a basic operational one.

Answer 2

In general, I think there is recognition that operations engineers and sometimes developers (either internal or with the 3rd party vendor whose products you are using) will need to see production logs in order to debug issues. The number of hoops you need to jump through -- like type of authentication, audit logs, redacting sensitive data, etc -- will depend on context: is there customer data in the logs (specifically "personally identifying information (PII)"?) in which case GDPR may apply depending on where you, your servers, and your customers are. Is there health data in the logs? In which case health regulations like HIPAA or PHIPAA may apply. Is there financial data? In which case PCI-DSS may apply.

---

For example, quoting from PCI-DSS v3.2.1:

Requirement 8: Identify and authenticate access to system components

8.1.1 Assign all users a unique ID before allowing them to access system components or cardholder data.

8.3 Secure all individual non-console administrative access and all remote access to the Cardholder Data Environment using multi-factor authentication.

So under PCI-DSS, they recognize that people will need to access raw data for various reasons, but they want clear audit trails of who accessed what and when.

Whether or not ssh keys count as strong authentication, I think will depend on what you're doing to ensure that ssh keys are uniquely tied to a single human; ie how do you know that people don't have access to each other's ssh keys?

Answer 3

While Conor's answer remains the answer to this question, I'm dropping here the solutions I ended up considering for this matter, in case they could help someone else in the same situation. In my specific case the solution had to be implemented purely at system level (hence no awareness from the applications that their logs are being sent/streamed somewhere else).

The possible solutions I considered are:

1. use rsyslog in order to continuously send logs from the application server(s) to a central "logs server", this way operators can have easy and direct access to logs without connecting to the actual application servers; I implemented a small POC of this option on Docker;
2. use nfs and an exports configuration in order to export the log folders from each application server, and mount them on a central "logs server"; I haven't tried this (yet) but a POC can be built on Docker starting from here.