Best security practices for software development and release?

Question

I am trying to figure out a way to limit access and also to establish processes to avoid any and all of my team members to compromise our entire production system.

We have a team of 10 developers, 1 QA and 1 SA.

Our entire team has direct access to our entire production DB for development and for support. They also have access to all of our production web servers for the same reasons.

Our system is a high volume, critical system for 1000+ businesses across the U.S.

Team members need access to production DB to debug and correct problems with data. SA and some team members have access to web servers to deploy/release new versions of code.

The common problems or risks: Everyone in our organization can potentially: - Simply wipe out the entire DB. - Run stupid queries taking the system down. - Corrupt data. - Release new code at will. - Have access to see full clients list. - Have access to sensitive information.

Processes are not enough, I need to also be able to enforce it with technology, logging, auditing, etc.

So what I am looking for is: - Common practices for the several scenarios. - Ideas, tips and tools to help with this.

Thank you.

p.s Application is apache,php,mysql source control in git, on ubuntu servers 99% on owned and self managed hardware at a secure location.

Answer 1

Developers should never have direct access to the production environment. From an audit perspective this is a big no-no as this poses fraud risks. Also if one developer makes a mistake he can take down your critical systems which could have a high impact on your business.

The best practice is to have 4 separate environments, Development, Testing, Acceptance and Production. Developers can have access to testing and in some cases to acceptance, but NEVER to production. This is called the DTAP principle:

1. The program or component is developed on a Development system. This development environment might have no testing capabilities.

2. Once the developer thinks it is ready, the product is copied to a Test environment, to verify it works as expected. This test environment is supposedly standardized and in close alignment with the target environment.

3. If the test is successful, the product is copied to an Acceptance test environment. During the Acceptance test, the customer will test the product in this environment to verify whether it meets their expectations.

4. If the customer accepts the product, it is deployed to a Production environment, making it available to all users of the system.

As said, if an auditor sees or even gets the slightest hint that a developer can get access to a production environment you will almost certainly get failed. If you want to perform audits, it's best done by an independent person/organisation. If you want to setup more processes and regulations have a look at the COBIT framework.

Make sure you define KPIs and SLAs to have a form of power to enforce your regulations.

Answer 2

so I'd agree with what @lucaskauffman and @terrychia have said so far but here are some more tips that might be useful to you in improving the control over your dev environment. Ultimately you should try and get to the kind of control level that lucas mentions but it might be a hard sell to jump straight to that so there's some intermediate steps you could take.

• Start by moving devs from read-write access to production to read-only. Introduce a change control process so that changes to production data are staged and applied in a controlled way, not by individual developers. If developers need to try things out then introduce a pre-prod environment and try things out there.
• Remove developer access to the server. Change management for all code changes, developers shouldn't have direct write access to the production server, one of the nice things about git is that it's easy to have a copy of the code so hopefully there shouldn't be any circumstances where direct code changes would be made to prod.
• Once you've got people working that way, ideally you wouldn't have people working on live data at all, they should work on anonymised sample data which reduces your risks of losing customer information. There are tools to anonymise data but even something as simple as a script which replaces names with Mr M Mouse and the like could help reduce the impact of someone taking a copy of the database away with them.
• if you can get as far as this point, then look at taking things further (e.g. COBIT/ITIL style models)

The huge point about all this is it will require good levels of buy-in from your management. all these steps cost money and will slow down the development/test process in the short term (in the long term they'll likely save the whole company though).

Getting that buy-in may be tricky if management don't see the problem. I'd suggest searching for horror stories of uncontrolled dev. environments, of even better if there have been problems in the one you're working in, use these as examples of things that could be avoided by adding a bit more process to things.

Answer 3

For god's sake. Never perform testing or debugging on production systems and data.... Especially since you are a high volume and critical system...

Setup a proper development environment that duplicates a subset of your production data to work with... Any new releases should be thoroughly tested by QA before being pushed out on the production systems. This greatly reduces the risk of bad code corrupting important data.

Most team members do not need access to the production environment. You can and should setup a system to automatically deploy code after it has been checked into your VCS and tested...