Based on our AWS DevSecOps oriented experience a mixture of 1, 2 and 3 would probably be the most effective solution.
Have separate AWS accounts for each environment and use consolidated billing?
Absolutely agree this was the official rec we've gotten from the AWS support at early 2017. The multi-account architecture segregating user-mgmt, security-audit-logging and shared-resources-services and project-app accounts will reduce the blast radius to the maximum possible.
Consider the following AWS official links as reference since the AWS Organization structure will depend on your specific requirements.
For it's automation you'll have some alternatives like AWS python SDK + bash or terraform (consider that if your org has less than 10 accounts managing it from the AWS Web Console will be very straight forward). However keeping in id an IaC (InfraAsCode approach with Terraform or AWS Cloud Formation is almost a must do from our point of view).
Create separate VPCs per environment? - keep in mind we'll likely want multiple regions/availability zones for High Availability
Since we assume we're going to implement multi-account AWS Org approach then will have at least 1 VCP per account x environment. In our case having one VPC in a certain region it's enough since for High-Availability, Self-healing, Redundancy and Resilience we use a multi-availability-zone (multi-az) configuration (eg: us-east-1a, us-east-1b and us-east-1c, will be extended in 3). Keep in mind choosing your region evaluation both network performance (latency - https://www.cloudping.info/) and costs (N.Virginia-us-east-1, Ohio-us-east-2, Orego-us-west-2 - https://www.concurrencylabs.com/blog/choose-your-aws-region-wisely/).
Post steps to have in mind:
Let's consider the following info to evaluate if multi-region is also needed.
As defined in the official doc -> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of Regions and Availability Zones. Each Region is a separate geographic area. Each Region has multiple, isolated locations known as Availability Zones
So for as hosting our HA Apps in a region but within isolated locations (datacenters) it's enough referening to the per region AWS SLA of:
- AWS will use commercially reasonable efforts to make the Included Services each available for each AWS region with a Monthly Uptime Percentage of at least 99.99%
- availability: 99.99%
- durability: 99.999999999% (depends on the storage class)
If because of compliance frameworks you still need a Multi-Region active-active method: Then by architecting multi region applications and using DNS to balance between them in normal production status, you can adjust the DNS weighting and send all traffic to the AWS region that is available, this can even be performed automatically with Route53 or other DNS services that provide health check mechanisms as well as load balancing. For multi-region architecture please consider: https://aws.amazon.com/blogs/aws/latency-based-multi-region-routing-now-available-for-aws/
Separate subnets per environment. Or should we use separate subnets for web severs/database servers/management-administration type servers? To get the desired multi-az deploying for Production Grade environments at least 3 Private Subnets (through a redundant AWS Nat-GW - at least 2 of them in different az) and 3 Public Subnets (through Internet GW) of course each one in a separate AZ. And for Dev/Qa/Stg (can be consolidated in a single account for cost efficiency) and have at least 2 private and 2 public subnets in different AZs. This applies for EC2 & RDS (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html), for S3 versioning and multi-region replication it's recommended
Based on the before presented let's present our reference architecture.
AWS org multi-account architecture
- user-mgmt-acct (centralized iam roles based cross-org approach - iam roles per accout to be assumed still needed).
- security-audit-logging-acct (centralized account to have centralized logs like cloudtrail, vpc-flow-logs, alb and cloudfront access logs, among others, centralized firewall manager setup, or any other AWS service or sec&audit tool).
NOTE: To simplify the architecture the before mentioned accounts can be consolidated in only 1 account.
- shared-resources-services-acct 1 x VPC + 2 prv-sub-nw + 2 public-sub-nw (multi-az) + 1 x nat-gw + VPC-Peering with project-X-app-accts (here you'll orchestrate your Software Development shared tools like, VPN Solution, Monitoring tools like prometheus and graphana, CI/CD server, maybe jenkins, drone, spinnaker, etc, multi-purpose infra K8s cluster, centralized logging solution like ELK or self hosted Splunk).
- project-A-dev-stg-acct: 1 x VPC + 2 prv-sub-nw + 2 public-sub-nw (multi-az) + 1 x nat-gw + VPC-Peering with shared-resources-acct
- project-A-prod-acct: 1 x VPC + 3 prv-sub-nw + 3 public-sub-nw (multi-az) + 2 x nat-gw + VPC-Peering with shared-resources-acct
- project-B-dev-stg-acct: 1 x VPC + 2 prv-sub-nw + 2 public-sub-nw (multi-az) + 1 x nat-gw + VPC-Peering with shared-resources-acct
- project-B-prod-acct: 1 x VPC + 3 prv-sub-nw + 3 public-sub-nw (multi-az) + 2 x nat-gw + VPC-Peering with shared-resources-acct
SECURITY - Miscellaneous
NOTE1: Remember to encrypt every service storage at rest at least with AWS encryption - https://docs.aws.amazon.com/aws-technical-content/latest/efs-encrypted-file-systems/encryption-of-data-at-rest.html
NOTE2: The 5 Pillars of the AWS Well-Architected Framework - https://aws.amazon.com/blogs/apn/the-5-pillars-of-the-aws-well-architected-framework/
NOTE3: AWS Tagging Strategies - https://aws.amazon.com/answers/account-management/aws-tagging-strategies/
NOTE4: Prevent AWS Secrets to git repository - https://www.binbash.com.ar/archives/1038