Sensitive information was placed in a publicly accessible folder. Who is responsible and how to proceed?

Question

Background

We have an IT staff who manages our server and a web developer who is not on the IT staff and has no root access to the server. All involved do very high quality work and I do not consider this lapse to be particularly grievous, given the relatively small consequences of not having this information secure. I am primarily interested in learning from this experience.

Last week, our web developer sent an email to me and the IT staff:

I did not realize that [project1] and [project2] had been put directly inside of /var/www on [my-server], but thought they had been put outside of publicly accessible space. I have corrected permissions to prevent access to anything sensitive at the moment, but they should not be in /var/www as only the 'public' folders need to be linked there.

Because of this all the files in the [project1] and [project2] projects have been accessible to the web since the start of the projects. From this people would have had access to all files in the projects, even access to database dumps and logs that have been stored there. It would be worth while to know if anyone has ever accessed any files in the 'path/to/project1/db_backups', 'path/to/project1/config', 'path/to/project1/db', 'rails/project2/db', 'rails/project2/config' folders as stored passwords could be found in these folders.

Questions

1. Was I at fault for assuming that either the IT staff or the web developer should have dealt with this?
2. Who is responsible for making sure that sensitive information is properly secured?
3. Are there steps beyond checking logs that I should take?

Answer 1

I would be reluctant to parcel out blame, particularly when it is clearly an honest mistake. Too much focus on blame on one incident can poison the well and make people more reluctant to report security incidents in the future. Even without any blame involved, it is already embarrassing enough to have to report that you screwed up and may have contributed to a security exposure. I like that you say "I am primarily interested in learning from this experience" -- I think that's a very healthy and constructive attitude, and I encourage you to repeat it widely so everyone involved knows it, and act in a corresponding way.

By the way, you can think of this as an opportunity to model how you react to security issues. Culture gets set not by the words company leaders say, but by the way they act -- especially the way they act when in a tough spot. Modeling the "this is not about blame, it's about learning from the experience" attitude seems like a nice way to make a contribution to a healthy culture that will serve you well in the future.

Back to your core question. I think the primary answer to this question is probably: communication. The first step is for the folks involved to talk about this, and agree on whose responsibility it is. Is it the responsibility of the web admin to confirm that all content which is put on the public web server, really is intended to be public? Is it the responsibility of the person who is posting the content? How would these people know whose responsibility it is?

Perhaps one possible lesson you could consider is: be cautious about what filesystems/mount points/shared directories you mount on your public webserver. You might want to have a rule of thumb that there is a separation: each filesystem/shared directory is either (a) public, or (b) internal, and label the filesystem/shared directory accordingly. Under this approach, you would avoid having a filesystem that contains some public files and some non-public files -- and when you add a new filesystem/shared directory to the public webserver, you might do a quick sanity-check to see whether that filesystem is indeed marked as public. This is just one idea, and of course, it may or may not be suitable for you. Don't get too caught up in the details of this particular idea; if it doesn't sound helpful, ignore it. The more important point is to think a little bit about processes that can help you avoid these errors -- without getting yourself bogged down in needless bureaucracy. You will know best what the culture and requirements of your company are like; you should be in a good position to brainstorm some ideas and start that conversation with your co-workers.

Answer 2

Was I at fault for assuming that either the IT staff or the web developer should have dealt with this?

Usually, you try to keep internal affairs unreachable from the outside world. Anything that goes into production must be checked and verified (at least by administrator). Well, and then double checked.

Who is responsible for making sure that sensitive information is properly secured?

Depends on the organisation's size and internal policy. This could be system administrator, software developer, chief information officer etc.

Are there steps beyond checking logs that I should take?

Yes! Never, never store passwords plain text or encrypted. Passwords must be hashed and salted before getting persisted into a database. Notify your customers about the leak and provide means to change their private data. This may get very serious if you have real customers, speak to a lawyer.

Change your sensitive configuration, if it has not been done before then encrypt its private sections.

Check search engines to see if your content has been cashed.

Answer 3

Are there steps beyond checking logs that I should take?

Although I'm sure you are asking what specific operational things should be done to try to ensure that the sensitive material has not been disclosed or altered, I'm going to answer this at a higher level of abstraction.

In cases like this, I like to ask myself the Five Whys. Work backwards from the incident and determine the root cause. It sounds like you've made some progress, but are not entirely done yet.

For example, if you are back-tracking through the incident and have determined that your development group inadvertently placed sensitive development material into a production environment, then this speaks to a lapse in security controls around your configuration mgmt system.

Institute a new (or alter your existing) configuration management system that ensures that your production environment cannot be altered by the development group. It is considered a best practice to have a development area and a production area (at a minimum; others such as a testing, integration or staging ground are also possible environments). The developers do not have access to production NOR do they have a tool that allows them to alter the production environment without going through some auditable process.

It also sounds like there is a lapse in auditing which could be a much larger security problem. You may want to understand why (again, 5 whys) you don't know who put it there. Perhaps it's because everyone uses the root account. Find another way to get the privileges you need (eg. sudo).

Oh. And change your passwords.

Answer 4

The first thing to understand is that even in the best of systems mistakes will happen. The good news here is that your web developer did a good job of identifying the problem and reporting it promptly.

"Was I at fault for assuming" - do you remember what they say about assume? ass_u_me....

Safeguarding corporate assets (of all types) is a responsibility of top level management and should be verified by your internal audit department.

"Who is responsible for making sure that sensitive information is properly secured?" The CEO and board of directors. This is a responsibility that can not be delegated. For publicly traded companies the days when a CEO could plead ignorance ended with Sarbanes-Oxley.

"Are there steps beyond checking logs that I should take?" Checking logs after the fact (anything after the fact for that matter) is too late. What you need to do is develop policies and procedures to safeguard your assets and the audit for compliance.