Does it make sense to isolate each web service user on his own servers?

Question

I am planning an architecture of a web application for my company which is launching a financial service for its partners, and my main question is - does it make sense to put each partner on a different server (or even server groups web/db)?

The application is in some sense similar to https://www.wealthfront.com/ - partners trust us with their assets and provide us with data, which we use to automate investments. Each partner has an account with us, login/password pair and a url where he can log in. After login he gains access to a dashboard which contains all his data presented in charts in tables.

Partners and their data is completely independent from each other. We also have some admin interface to preprocess their data and manage their accounts, but there is a separate web/db/workers adm server and partners feed their data to main db via write-only services (they cannot query any arbitrary information).

Partners data is very sensitive, if any of it falls in wrong hands - the impact could be very serious up to us going out of business.

I have two ways in mind on how to organize an architecture of such service:

• Grouped, in which we have one server for each role, all partners are sitting all together on each server

• Isolated, in which we also have one server for each role, but each partners gets a set of servers for each role

I see two types of scenarios in which unauthorized access to data could happen:

• Attackers: somebody hacks some server and gains some kind of access to it. Could be anything from simple misconfiguration (ssh access left from public network without password, heartbleed-like vulnerability, ...). This is too vague and isolation surely cannot save from all scenarios, but I still have a feeling that with all other things being equal isolated architecture is more secure, because it creates more barriers.
• Human error: an engineer forgets to add some security checks or inadvertently removes some or gives everyone superadmin powers to see everything, it gets into production and everybody is unhappy. This surely should be prevented by all kinds of rules, reviews, constraints and processes but errors always find their to production. I have a feeling that physical isolation also helps mitigating if something likes happens (even when a junior partner employee gains access to partners senior-only information, it is much better if he gains access to senior-only information of other partner!)

Grouped pros/cons which I see:

• [+] more easily scaled
• [+] less expensive (less servers, can spread load more effectively)
• [+] less to configure (though we are using configuration management tools and host in cloud environment, so it is not a big problem anyway)
• [-] every access checks failure will result in exposure of all data data to user
• [-] hacking into web/db server gives you access to everything (e.g. if some partner decides to explore and somehow successfully finds his way inside our system)
• [-] if one partner creates a lot of load, it hurts everyone (though mitigated by somewhat easy scaling)
• [-] cannot effectively use ip-based restrictions (I think)

Isolation pros/cons which I see:

• [+] every access checks failure will result in exposure of user's own data to himself
• [+] hacking into web/db server gives you only this server data (e.g. if some partner decides to explore and somehow successfully finds his way inside our system)
• [+] separates one partner's load on system from other
• [+] can tune firewall rules more granularly (we can limit access from only partner ip which is not the case for grouped architecture)
• [-] less easily scaled (though this is definitely not a high-load project, so it's not that much of concern)
• [-] more expensive
• [-] more to configure (though we are using configuration management tools and host in cloud environment, so it is not a big problem)

So far I can see a lot of pros for isolation and some little cons, but there is a huge question: since we use configuration management tools, each server will have the same OS, same software and configuration, same codebase and everything. This way if we have any vulnerability, nothing can stop attacker to use them on all servers, isolated or not. Maybe there are also other pitfalls that I cannot see right now that undermine the whole idea.

So, given all that information:

• Do you think that isolated architecture provides more security than grouped?
• Are there any serious tradeoffs/showstoppers to isolated architecture that I haven't thought of?
• Do you think there's some common alternative solution to this problem (not grouping or isolating, but something else)?
• If isolated architecture is good, is there any sense splitting web and db servers (security-wise), since they are already isolated from everything else and splitting them doesn't add more security?

Answer 1

What I've seen in some similar cases is to have each client be given their own instance of the DB, with multiple DBs running on the same server. Each client can only get to their DB with their password, so there is very little risk of the data leakage. At worst, an encrypted DB ends up in the wind without the account credentials needed to unlock it.

The key of that working, however, is that the user account credentials must be a key component of getting the password for the DB that the user's data resides in and the DB must be encrypted at rest.

You are still exposed to leakage while the system is running, but your still significantly protected with a balance between isolating DBs and having completely stand alone infrastructure.

I've seen this scale to include relatively large systems.

Answer 2

I think the mistake here is in thinking that this is a black and white choice. Whether you group or isolate servers are shades of grayscale. If you use containers (e.g. Docker, LXC, Rkt) or virtual machines (e.g. IaaS), then you can run multiple application/database servers from different partners in the same machine, while benefiting from the isolation provided by the container/virtual machines and the economy of sharing the underlying resources.

In the lowest tier, you can have multiple partners share a single database cluster, the database server will enforce authorization by checking the partner_id. These can be served by a load balanced cluster of application servers.

In the mid low tier, same as previous but each partner have their own instance of database inside an application container.

In the middle tier, same as previous but each partner have a database server dedicated for themselves.

In the middle high tier, same as previous the partner can get their own dedicated load balancer.

The highest tier, partners who would pay more for security can get their own dedicated machines for both application and database server, with additional firewall rules to prevent their machines from being accessed by unauthorized server. Or possibly run the application inside their own corporate network.

Answer 3

Does it give more security? I could see a slight bit, namely in the scenario where a legitimate user (or an attacker that has gained control over a legitimate users network) finds they have a bit more access than they should. This would better prevent them from accessing one of your other customers data. The real question is the increase in security per cost and if there are any better options/is it worth while.

As to splitting web and db, I would still do this regardless of which plan because you don't want any outside connection, even from a whitelisted IP, to be connecting to your database unless there is a really strong business justification. I would also think that using an IPS/IDS would be easier when they sit between the web server and db server instead of trying to monitor activity on a single server.

Either way, you'll be able to use a whitelist, it is just a question of if it'll be many whitelists each on their own web server or the union of those whitelists on a single (or small number) of servers.

As for alternative, considering that this is a cloud based solution you are working with, you'll need to look to other answers on here.

Answer 4

Personally, I don't think the isolated architecture provides you much added security over the grouped architecture. If an adversary penetrates one server that mirrors another in design then, sure, you only have one problem for a moment. Effectively, though, you know that they have the key to the neighborhood and that all the locks need to be changed immediately.

Having served enterprise customers in the technology space for a long time now, there is one glaring thing I think you missed. Some technology decision makers are going to voice concerns that their data is not isolated "in the cloud" from their competitors' data, and you will need to provide a compelling answer. In some cases these leaders want to deploy all of their solutions in-house and avoid "the cloud" at any cost. In other cases, you might win them over with an isolated architecture like you've described.

To my prior point, one possible solution is to provide two options to your customers. Let them use your grouped (cloud) offering, or let them deploy the software in-house themselves. You will need to design the solution around those licensing options. Many vendors do this. Microsoft is probably the best example.

Answering your final question, there is a security advantage to separating the web servers from the database servers, but it depends on how you intend to make the databases accessible. If the only applications that will ever access the databases reside on the same environment, then you can just put them on the same environment and opt not to expose the database. But if you need to expose the database to multiple applications, many enterprises put their database servers in their LAN behind the DMZ and put their web servers in a perimeter network that is accessible to the internet. This way, adversaries are oblivious to the location (indeed, to the existence) of the servers on which the database resides, unless they gain access to the LAN.