An end-to-end guide to designing, deploying, and operating scalable software on AWS.
Saidul Islam RajibJul 22, 2026 · 40 min read · 17 views
The Lifecycle at a Glance
┌──────────────────────────────────────────────────────────────────────┐
│ 1. DISCOVER Problem, users, constraints, success metrics │
│ 2. DESIGN Domain model, architecture, ADRs, API contracts │
│ 3. BUILD Code, tests, reviews, local + ephemeral envs │
│ 4. INTEGRATE CI: build once, test, scan, produce one artifact │
│ 5. DEPLOY CD: promote that artifact Dev → QA → Staging → Prod│
│ 6. RELEASE DNS, CDN, TLS, WAF, progressive rollout │
│ 7. OPERATE Monitor, alert, on-call, incident response │
│ 8. EVOLVE Feedback, scale, refactor, pay down debt │
└──────────────────────────────────────────────────────────────────────┘
▲ │
└────────────────── continuous feedback ─────────────────┘
The loop matters more than the sequence. Mature teams run all eight phases continuously rather than as a one-time waterfall.
PHASE 1 — Discovery and Product Definition
Before any architecture, the team establishes what is actually being built and how success will be measured.
1.1 What Gets Produced
| Artifact | Purpose |
|---|---|
| Problem statement | The user pain, in one paragraph, without a proposed solution |
| User personas & journeys | Who uses it, what they are trying to accomplish |
| Functional requirements | What the system must do |
| Non-functional requirements (NFRs) | The numbers that drive architecture |
| Success metrics | How you know it worked (activation, retention, conversion) |
| Constraints | Budget, deadline, compliance, team size, existing systems |
1.2 Non-Functional Requirements Drive Everything
This is the step teams most often skip, and it is the one that determines the architecture. Get concrete numbers:
Scale
Expected users: 100k registered, 5k concurrent peak
Requests per second: 2,000 average / 12,000 peak
Data volume: 500 GB year 1, growing 40% annually
Traffic shape: 3x spike during promotional campaigns
Performance
API latency: p50 < 100ms, p95 < 300ms, p99 < 800ms
Page load (LCP): < 2.5s on 4G
Search results: < 500ms
Availability
Uptime target: 99.9% (≈43 min downtime/month)
RTO (recovery time): < 1 hour
RPO (data loss tolerance): < 5 minutes
Security & Compliance
PII handling, PCI-DSS for payments, GDPR right-to-erasure,
data residency requirements, audit retention period
Cost
Target infrastructure spend: $X/month at launch scale
Every one of these numbers maps to an AWS decision later. "99.9% uptime" means multi-AZ. "RPO < 5 minutes" means point-in-time recovery and cross-region replication. "3x campaign spikes" means autoscaling and a caching strategy. Without the numbers, architecture becomes guesswork or cargo-culting.
1.3 Deliberately Choosing Boring
A key industry discipline: innovation tokens are finite. Spend novelty on the parts that differentiate the product, and choose proven, well-understood technology everywhere else. A team that picks a novel database, a novel language, a novel deployment model, and a novel frontend framework simultaneously spends its entire first year fighting tooling rather than shipping.
PHASE 2 — Architecture and System Design
2.1 Domain Modelling First, Services Second
Microservice boundaries follow business capabilities, not technical layers. The standard approach is Domain-Driven Design:
- Event storming — map the business events end to end (
OrderPlaced,PaymentCaptured,ShipmentDispatched). - Identify bounded contexts — clusters of events and rules that change together and share a vocabulary.
- Define the ubiquitous language — the same word means one thing inside a context. "Customer" in Billing (a payment method holder) is not "Customer" in Support (a ticket author). Forcing them into one shared model is a classic source of coupling.
- Draw context maps — how contexts interact: upstream/downstream, published events, anti-corruption layers.
A correct boundary means a typical feature request touches one service. If most features require coordinated changes across five services, the boundaries are wrong — you have a distributed monolith, which combines the operational cost of microservices with the coupling of a monolith.
2.2 Monolith or Microservices?
The honest answer for most new products: start with a well-structured modular monolith.
Modular Monolith Microservices
────────────────── ──────────────
One deployable Many deployables
In-process calls (fast, reliable) Network calls (latency, partial failure)
One database, ACID transactions Per-service data, eventual consistency
Refactor boundaries cheaply Boundaries are expensive to move
One pipeline, one runtime to operate N pipelines, service mesh, tracing required
Team of 3–15 works fine Needs platform investment + team autonomy
Split into services when you have a concrete reason:
- Independent scaling — search needs 20 instances, billing needs 2
- Team autonomy — separate teams blocked by a shared release train
- Fault isolation — a failing recommendations engine must not take checkout down
- Divergent technology — one workload genuinely needs a different runtime
- Divergent compliance — payments needs a stricter, isolated environment
Never split because "microservices are best practice." Premature decomposition is one of the most expensive mistakes in software, because distributed boundaries are far harder to move than in-process ones.
The realistic path: modular monolith → extract the one or two services with real independent-scaling or isolation needs → extract further only as pain demands.
2.3 Reference Architecture on AWS
Users (browser / mobile)
│
Route 53 (DNS, health-based routing)
│
CloudFront (CDN, global edge)
┌───────────┴────────────┐
│ │
Static frontend Dynamic API
S3 (SPA bundle) │
AWS WAF
│
┌───────────────┴──────────────┐
│ API Gateway or ALB │
│ authn/z, rate limit, route │
└───────────────┬──────────────┘
│
┌──────────────┬────────────┬─────────┴────────┬────────────────┐
▼ ▼ ▼ ▼ ▼
Identity Svc Catalog Svc Order Svc Payment Svc Notification Svc
(ECS/EKS) (ECS/EKS) (ECS/EKS) (ECS/EKS) (Lambda)
│ │ │ │ │
│ ┌────┴────┐ │ │ │
│ │ElastiCache │ │ │
│ │ (Redis) │ │ │ │
│ └─────────┘ │ │ │
▼ ▼ ▼ ▼ ▼
RDS/Aurora RDS + OpenSearch Aurora Aurora (stateless)
(Postgres) + DynamoDB (encrypted)
│
▼
┌──────────────────────────────────────┐
│ EventBridge / SNS / SQS / Kinesis │ ← async backbone
└──────────────────────────────────────┘
│
┌──────────────────┴───────────────────┐
│ S3 (objects, data lake) │
│ Secrets Manager / KMS │
│ CloudWatch / X-Ray / OpenTelemetry │
└──────────────────────────────────────┘
2.4 Synchronous vs Asynchronous Communication
The most consequential design choice inside a distributed system.
SYNCHRONOUS (REST / gRPC)
Caller waits for a response.
✔ Simple, immediate consistency, easy to reason about
✘ Temporal coupling — callee down means caller down
✘ Latency compounds: 5 chained calls at 100ms = 500ms
✘ Cascading failure risk
Use for: queries, user-facing reads, operations needing
an immediate answer (auth check, price lookup)
ASYNCHRONOUS (events / queues)
Caller publishes and moves on.
✔ Temporal decoupling — consumer can be down, work is buffered
✔ Natural buffering absorbs traffic spikes
✔ Easy to add new consumers without touching the producer
✘ Eventual consistency; harder debugging; needs idempotency
Use for: side effects, fan-out, long-running work, integration
between bounded contexts
Practical rule: synchronous within a bounded context; asynchronous between bounded contexts.
Checkout illustrates this well:
SYNCHRONOUS (user is waiting — must succeed now)
validate cart → check inventory → authorize payment → create order
│
▼
publish OrderPlaced event
│
ASYNCHRONOUS (user does not need to wait) ────────────────────┤
├─▶ send confirmation email
├─▶ decrement inventory
├─▶ notify warehouse
├─▶ update analytics
└─▶ trigger loyalty points
If the email provider is down, the order still succeeds. That isolation is the entire point.
2.5 Distributed Data Patterns
Once each service owns its own database, familiar guarantees disappear. The patterns that replace them:
Saga — a business transaction spanning services, implemented as a sequence of local transactions with compensating actions on failure.
Order Created → Payment Charged → Inventory Reserved → Shipment Booked
│ FAILS
▼
Compensate: Refund Payment
Cancel Order
Choreography (services react to each other's events) is simpler for short flows; orchestration (a coordinator like AWS Step Functions drives the flow) is clearer for long or complex ones.
Outbox pattern — writing to your database and publishing an event are two separate systems; without care you can commit one and lose the other. Instead, write the event into an outbox table in the same local transaction as the business data, then have a relay publish it. This guarantees at-least-once delivery without distributed transactions.
Idempotency — because at-least-once delivery means duplicates are inevitable, every consumer must tolerate replays. Standard technique: an idempotency key per operation, recorded on first processing and checked thereafter.
CQRS — separate the write model from read models when their shapes genuinely diverge (e.g. normalised writes in a relational DB, denormalised search-optimised reads in OpenSearch).
API composition / BFF — a Backend-for-Frontend aggregates calls to several services so the client makes one request instead of seven. Prevents chatty mobile clients and leaks of internal service topology.
2.6 API Contract Design
- Contract-first. Define the OpenAPI/AsyncAPI/protobuf schema before implementing. It unblocks frontend and backend to work in parallel and lets you generate mocks.
- Version from day one (
/v1/...or header-based). You will need v2 sooner than you expect. - Backward compatibility is mandatory while any client still uses the old version. Additive changes only; never repurpose a field's meaning.
- Consumer-driven contract tests verify that a provider change does not break real consumers, catching integration breaks in CI rather than in staging.
- Standardise the cross-cutting concerns once, org-wide: error envelope, pagination, filtering, correlation-ID header, auth scheme, rate-limit headers.
2.7 Recording Decisions — ADRs
Every significant choice gets a short Architecture Decision Record committed alongside the code:
# ADR-014: Use Aurora PostgreSQL for the Order Service
Status: Accepted (2026-03-11)
Context: Orders require multi-row ACID transactions, complex reporting
joins, and an RPO under 5 minutes. Team has deep SQL experience.
Decision: Aurora PostgreSQL, Multi-AZ, with point-in-time recovery.
Alternatives considered:
- DynamoDB — rejected: transactional/reporting query patterns
would require significant denormalisation and app-side joins
- Self-managed PostgreSQL on EC2 — rejected: operational burden
Consequences:
+ Familiar tooling, strong consistency, mature ecosystem
− Higher cost than DynamoDB at low volume
− Requires connection pooling (RDS Proxy) as services scale out
Six months later, ADRs answer "why on earth did we do it this way?" without archaeology. They are cheap to write and enormously valuable.
PHASE 3 — Data Layer Design
3.1 Polyglot Persistence — Right Tool per Job
Each service picks the store matching its access pattern. On AWS:
| Need | AWS service | Use when |
|---|---|---|
| Relational, ACID, complex queries | RDS / Aurora (PostgreSQL, MySQL) | Orders, payments, users — anything with invariants across rows |
| Key-value / document at massive scale | DynamoDB | Sessions, carts, event logs, high-write telemetry; predictable access patterns |
| Full-text & faceted search | OpenSearch | Product search, autocomplete, log analytics |
| In-memory cache / ephemeral state | ElastiCache (Redis / Valkey) | Hot reads, sessions, rate limiting, leaderboards |
| Object / blob storage | S3 | Images, documents, exports, backups, data lake |
| Time series | Timestream | IoT, metrics, sensor data |
| Graph | Neptune | Recommendations, fraud rings, social graphs |
| Analytics / warehouse | Redshift or Athena over S3 | BI, reporting, ad-hoc analysis |
| Wide-column | Keyspaces (Cassandra) | Very high write throughput, time-partitioned |
Critical rule: one service owns its database. No other service reads it directly. Shared databases are the fastest way to recreate a monolith with extra network hops — a schema change becomes a cross-team negotiation, and you lose every benefit of independent deployment.
3.2 Relational vs NoSQL — the Real Decision
Choose RELATIONAL when:
- Data is highly relational with meaningful joins
- You need multi-row/multi-table ACID transactions
- Query patterns will evolve unpredictably (ad-hoc reporting)
- Strong consistency is a business requirement (money, inventory)
- Team knows SQL (almost always true)
Choose NoSQL (DynamoDB) when:
- Access patterns are known, simple, and stable
- You need single-digit-millisecond latency at extreme scale
- Write volume exceeds what a single writer node can absorb
- The schema is genuinely flexible/sparse
- You want zero operational overhead and true pay-per-request
DynamoDB rewards designing the table around the queries (single-table design, careful partition keys). It punishes ad-hoc querying. If you cannot enumerate your access patterns up front, that is strong evidence for relational.
3.3 Reliability and Recovery
Multi-AZ deployment
Primary in AZ-a, synchronous standby in AZ-b.
Automatic failover in 30–120s. Non-negotiable for production.
Read replicas
Offload reporting and read-heavy traffic from the primary.
Note: replicas are eventually consistent — never read your own
write from a replica without care (the read-after-write trap).
Automated backups + point-in-time recovery
Restore to any second within the retention window.
Cross-region replication (Aurora Global Database)
For strict RTO/RPO or regional DR requirements.
RESTORE DRILLS
An untested backup is not a backup. Schedule real restore
exercises — quarterly at minimum. Teams routinely discover
their backups were unusable only during an actual incident.
3.4 Schema Migrations Without Downtime
Migrations are the most common cause of failed deploys. Use expand/contract and never combine a destructive change with the code depending on it:
Step 1 ADD the new column (nullable, with default) ← old code unaffected
Step 2 Deploy code that writes to BOTH old and new
Step 3 Backfill historical rows (batched, throttled)
Step 4 Deploy code that reads only the new column
Step 5 Stop writing the old column
Step 6 DROP the old column ← only now
Each step is independently deployable and independently reversible. Compressing this into one release makes rollback impossible: the moment the old column is gone, the previous application version cannot run.
Also: migrations run as a distinct, gated pipeline step — not on application startup, where N instances would race each other.
3.5 Connection Management
A frequently missed failure mode. Each service instance holds a connection pool; autoscaling to 50 instances × a 20-connection pool = 1,000 connections, which will exhaust the database long before CPU becomes the constraint.
- Use RDS Proxy (or an equivalent pooler) to multiplex connections
- Size pools deliberately; more connections is not more throughput
- Set aggressive connection and statement timeouts
- Serverless/Lambda workloads especially need a proxy — otherwise each cold start opens a new connection
PHASE 4 — Caching Strategy
Caching is the highest-leverage performance work available, and the easiest to get subtly wrong.
4.1 The Caching Layers
① BROWSER CACHE Cache-Control, ETag on static assets
│ Cheapest possible request: none at all
▼
② CDN — CloudFront Static assets, images, cacheable API GETs
│ Served from an edge location near the user
▼
③ API GATEWAY CACHE Response caching per endpoint + key
│
▼
④ APPLICATION CACHE In-process memory (fast, per-instance,
│ inconsistent across instances — use only
│ for small, rarely-changing reference data)
▼
⑤ DISTRIBUTED CACHE ElastiCache (Redis) — shared across all
│ instances: sessions, hot entities,
│ computed aggregates, rate-limit counters
▼
⑥ DATABASE CACHE Buffer pool, query cache, materialized views
│
▼
⑦ ORIGIN The actual computation / storage
Every layer you satisfy a request at is a layer of cost, latency, and load you avoid below it.
4.2 Caching Patterns
Cache-Aside (lazy loading) — the default, used ~80% of the time:
read(key):
value = cache.get(key)
if value exists: return value # hit
value = database.query(key) # miss
cache.set(key, value, ttl)
return value
Only requested data is cached; a cache failure degrades performance rather than breaking correctness. Downside: every miss pays the full latency, and the first request after a deploy is always slow.
Write-Through — write to cache and database together. Cache is always fresh; writes are slower, and you cache data that may never be read.
Write-Behind — write to cache, flush to the database asynchronously. Very fast writes, but data loss risk if the cache dies before flush. Use only where that loss is acceptable.
Refresh-Ahead — proactively refresh hot keys before expiry, so users never pay the miss penalty on popular items.
4.3 Invalidation — the Hard Part
"There are only two hard things in Computer Science: cache invalidation and naming things." — Phil Karlton
Strategies, in rough order of preference:
- TTL-based expiry — simplest and most robust. Accept bounded staleness. Ask the business how stale is acceptable; the answer is usually "a few minutes", which TTL handles trivially.
- Event-driven invalidation — the service publishes
ProductUpdated, consumers evict the key. Accurate, but adds coupling and can miss events. - Versioned keys —
product:v7:123. Bumping the version invalidates everything atomically; old entries age out naturally. Excellent for schema changes. - Write-through — no invalidation needed because the cache is updated at write time.
Never rely on manual invalidation ("we'll remember to clear it"). It will be forgotten.
4.4 Cache Failure Modes to Design Against
Thundering herd / cache stampede — a popular key expires and 5,000 concurrent requests all miss and hit the database simultaneously, often taking it down.
Mitigations: lock or single-flight so only one request recomputes while others wait; add random jitter to TTLs so keys do not expire in lockstep; refresh-ahead for known-hot keys.
Cache penetration — repeated requests for a key that does not exist bypass the cache entirely every time (sometimes maliciously).
Mitigation: cache the negative result with a short TTL; use a bloom filter for existence checks.
Cache avalanche — a large set of keys expires at once, or the cache cluster restarts cold, and full traffic hits the origin.
Mitigations: jittered TTLs, cache warming on deploy, circuit breakers protecting the origin, multi-AZ Redis with replicas.
Stale-data bugs — the most damaging category, because they are silent. A user updates their profile and still sees the old value; a price change does not propagate. Be explicit about which data can tolerate staleness and which cannot. Never cache authorization decisions or account balances with a long TTL.
4.5 What to Cache — and What Not To
GOOD CACHE CANDIDATES
Read-heavy, write-light data (product catalog, categories)
Expensive computations (aggregates, recommendations)
Reference/lookup data (countries, tax rates, config)
Session data (naturally keyed, short-lived)
Rendered fragments (rarely-changing page sections)
Rate-limit counters (Redis is ideal for this)
POOR CACHE CANDIDATES
Data written more often than read (cache churn, no benefit)
Highly personalized, low-reuse (near-zero hit rate)
Financial balances / inventory (staleness has real cost)
Authorization decisions (revocation must be immediate)
Anything where staleness is a correctness bug
Always measure the hit rate. A cache below ~80% hit rate for hot paths is usually mis-keyed, under-sized, or caching the wrong things — and it may be adding latency rather than removing it. Track hit rate, eviction rate, and memory pressure as first-class metrics.
PHASE 5 — Development
5.1 Environments
LOCAL Developer machine. Containerised dependencies
(database, cache, message broker) so setup is one command.
LocalStack or equivalent can emulate AWS services.
EPHEMERAL/PR Spun up per pull request, destroyed on merge.
Reviewers and QA see the change running, not a screenshot.
Highest-value environment investment a team can make.
DEV Shared integration environment. Auto-deploys on merge.
Expected to be occasionally broken.
QA / STAGING Production-like: same topology, same instance classes
(scaled down), anonymised production-shaped data.
This is where load tests and release candidates live.
PRODUCTION The real thing.
Enforce environment parity: same artifact, same infrastructure code, same runtime versions. Everything that differs is configuration.
5.2 Source Control and Review
- Trunk-based development with short-lived branches (hours to a couple of days). Long-lived feature branches produce painful merges and hide integration problems until the worst possible moment.
- Small pull requests. Review quality collapses above ~400 changed lines; a 2,000-line PR gets "LGTM" and nothing else.
- Required checks before merge: build, tests, coverage threshold, linting, dependency and secret scanning, at least one approving review.
- Protected main branch, no direct pushes.
- Conventional commits so changelogs and semantic versions can be generated automatically.
5.3 Feature Flags — Decoupling Deploy from Release
The technique that makes trunk-based development and continuous deployment practical:
Deploy = the code is running in production (a technical event)
Release = users can see the behaviour (a business event)
Incomplete work ships behind a disabled flag. This enables:
- Merging continuously instead of maintaining long branches
- Progressive rollout: internal staff → 1% → 10% → 50% → 100%
- Instant kill switch — turning a flag off is far faster and safer than a rollback deploy
- A/B experiments and per-tenant enablement
- Decoupling launch timing from deployment timing
Discipline required: flags are technical debt with a half-life. Every flag needs an owner and a removal date, or the codebase accumulates hundreds of dead conditionals and an untestable combinatorial explosion of states.
5.4 Testing Strategy
▲ Few · slow · expensive · realistic
│
│ ┌─────────────────────────────┐
│ │ Manual exploratory │ Judgement, not regression
│ ├─────────────────────────────┤
│ │ E2E (critical journeys) │ Signup, checkout, payment
│ ├─────────────────────────────┤
│ │ Contract tests │ Service boundaries hold
│ ├─────────────────────────────┤
│ │ Integration tests │ Real DB/cache in containers
│ ├─────────────────────────────┤
│ │ Unit tests │ Business logic, fast, many
│ └─────────────────────────────┘
│
▼ Many · fast · cheap · isolated
Additional gates for a production-grade system:
- Load tests against staging, sized to the NFR targets — before launch, and before any expected traffic event
- Chaos experiments — kill instances, inject latency, sever a dependency, verify graceful degradation
- Security tests — SAST, DAST, dependency CVE scanning, periodic penetration testing
- Accessibility tests — automated WCAG checks in CI plus manual screen-reader passes
- Smoke tests post-deploy against the real environment
Test the critical revenue path obsessively. If checkout breaks, nothing else matters.
5.5 Twelve-Factor Application Discipline
Non-negotiable for anything running on cloud infrastructure:
- Config in the environment, never in the image or the repository
- Stateless processes — any instance can serve any request; state lives in a database, cache, or object store. This is what makes horizontal scaling and instance replacement possible.
- Backing services are attached resources — swapping a database is a config change
- Logs to stdout as structured JSON; the platform handles collection
- Graceful shutdown — handle the termination signal, drain in-flight requests, finish or requeue background work
- Disposability — fast startup, safe termination; instances are cattle, not pets
- Dev/prod parity — minimise the gap in time, personnel, and tooling
PHASE 6 — Build and Continuous Integration
6.1 The Pipeline
Commit / PR
│
├─ Lint + format check ~10s ┐
├─ Unit tests ~2min │ fail fast — cheapest first
├─ Build artifact ~3min │
├─ SAST + secret scan ~1min ┘
├─ Build container image ~2min
├─ Image vulnerability scan ~1min ← block on HIGH/CRITICAL
├─ Integration tests ~5min
├─ Contract tests ~1min
│
▼
Push to ECR → ONE immutable artifact, tagged with the commit SHA
Target under ~10 minutes for the whole CI run. Beyond that, developers context-switch away and the feedback loop breaks.
6.2 Build Once, Promote Everywhere
The single most important CI/CD rule, repeated because it is violated so often:
myapp@sha256:9f2c4a...
│
┌───────────────────┼───────────────────┐
▼ ▼ ▼
DEV QA PRODUCTION
(env config) (env config) (env config)
The bytes tested in QA are the bytes running in Production. Rebuilding per environment means QA validated an artifact that never shipped — dependency resolution, base image patches, and build-time nondeterminism all drift between builds.
6.3 Infrastructure as Code
All infrastructure defined in version-controlled code — Terraform, CDK, CloudFormation, or Pulumi.
✔ Reviewable infrastructure changes go through pull request
✔ Reproducible recreate an entire environment from scratch
✔ Auditable git log answers "who opened that security group?"
✔ Consistent Dev/QA/Prod from the same modules, different variables
✔ Disaster recovery rebuild a region from source
Rules that matter in practice:
- No manual console changes in production. Ever. Console drift is invisible, undocumented, and lost on the next apply. Grant read-only console access in production; make changes through the pipeline.
- Remote state with locking (S3 + DynamoDB for Terraform)
- Plan output posted to the PR so reviewers see what will actually change
- Modules per environment, parameters per environment — never copy-pasted stacks that silently diverge
- Drift detection running on a schedule
6.4 Deployment Strategies
ROLLING Replace instances gradually.
Default for ECS/EKS. No downtime.
Both versions run at once → requires backward-
compatible schema and API changes.
BLUE-GREEN Stand up the full new version alongside the old,
switch the load balancer, keep the old warm.
Instant rollback; ~2x cost during the switch.
AWS CodeDeploy automates this.
CANARY Route 5% → 25% → 100%, watching error rate and
latency at each step, with automatic rollback on
alarm. Lowest risk; needs good metrics.
The industry standard for high-traffic systems.
FEATURE FLAG Deploy dark, enable progressively per user segment.
Fastest possible kill switch.
Mature setups combine canary deployment with feature flags: the code rolls out safely, and the behaviour is enabled independently.
6.5 Progressive Delivery with Automatic Rollback
The deployment gate that turns "we deployed and hoped" into an engineering control:
Deploy canary (5% traffic)
│
├─ Watch for 10 minutes:
│ error rate < baseline + 0.5%
│ p99 latency < baseline + 20%
│ no new alarm states
│
├─ Breached? ──▶ automatic rollback, page on-call, stop the pipeline
│
└─ Healthy? ──▶ 25% → observe → 50% → observe → 100%
Automating the rollback decision removes the worst failure mode in deployment: a human under pressure deciding whether the graphs look "bad enough" at 2 a.m.
PHASE 7 — AWS Infrastructure and Deployment
7.1 Account Structure
Production-grade AWS starts with multiple accounts, not one account with tags:
AWS Organization
├── Management account (billing, SCPs — no workloads)
├── Security / Log Archive (CloudTrail, GuardDuty, immutable logs)
├── Shared Services (CI/CD, artifact registries, DNS)
├── Development (permissive, low limits, auto-shutdown)
├── QA / Staging (production-like)
└── Production (locked down, minimal human access)
Separate accounts give a hard blast radius boundary — a mistake in Dev physically cannot touch Production — plus clean cost attribution and independent service quotas. Service Control Policies enforce guardrails organisation-wide (e.g. deny disabling CloudTrail, deny unapproved regions).
7.2 Network Design
VPC (10.0.0.0/16) — spanning at least 3 Availability Zones
PUBLIC SUBNETS ALB, NAT Gateway, bastion (if any)
Internet-routable
PRIVATE SUBNETS Application containers/instances
Outbound via NAT only, no inbound from internet
ISOLATED SUBNETS Databases, caches
No internet route at all
- Security groups are the real firewall — reference other security groups rather than CIDR ranges (
allow 5432 from sg-app), so rules stay correct as instances change. - VPC endpoints for S3, DynamoDB, ECR, Secrets Manager keep traffic on the AWS network — lower latency, lower cost, and it avoids NAT Gateway charges, which are a notorious surprise on the bill.
- Multi-AZ everything. An AZ is a real failure domain; single-AZ deployment forfeits the availability target you promised in Phase 1.
- Multi-region only when the requirement genuinely demands it — it multiplies complexity in data consistency, deployment, and cost.
7.3 Choosing Compute
| Option | Best for | Trade-off |
|---|---|---|
| Lambda | Event-driven, spiky, low-duration work; glue code | Cold starts, duration limits, harder local testing |
| ECS on Fargate | Standard long-running services | Simplest container path on AWS; less control than EKS |
| EKS (Kubernetes) | Complex platforms, multi-team, portability needs | Powerful, portable — but a genuine platform-team commitment |
| EC2 Auto Scaling | Specialised or licensed workloads | Most control, most operational burden |
| App Runner / Amplify | Small teams, simple web apps | Fastest to ship, least flexible |
The honest recommendation: most teams should default to ECS on Fargate. It gives containerised, autoscaled, multi-AZ services without a dedicated platform team. Adopt EKS when there is a concrete need — heavy multi-tenancy, a rich operator ecosystem, or multi-cloud portability — not because Kubernetes is fashionable. Kubernetes is a full-time responsibility, not a deployment target.
Use Lambda for genuinely event-driven work (S3 triggers, scheduled jobs, stream processing, webhooks) rather than as a general application runtime.
7.4 Autoscaling
HORIZONTAL (add instances) ← strongly preferred
Scale on the metric that reflects real load:
- request count per target
- queue depth (for workers — the best signal available)
- p95 latency
- CPU/memory (crude, but a reasonable default)
Scale OUT fast, scale IN slow.
Aggressive scale-in causes thrash and drops in-flight work.
VERTICAL (bigger instances)
Simpler but bounded, and usually requires a restart.
SCHEDULED
Pre-warm ahead of known events — business-hours ramp, campaign
launches, Black Friday. Reactive autoscaling always lags a spike;
scheduled capacity meets it.
Load-test to find the real breaking point rather than guessing thresholds, and remember to scale the layers behind the application — database connections, cache capacity, and third-party API quotas frequently become the true ceiling.
7.5 Going Public
The final steps that put the system on the open internet:
DNS Route 53 hosted zone, health checks, latency or
failover routing policies
TLS ACM certificate (free, auto-renewing), TLS 1.2+ only,
HSTS enabled
CDN CloudFront in front of both static assets and the API
Compression, HTTP/3, sensible cache policies,
Origin Access Control so S3 is not publicly readable
WAF Managed rule sets (OWASP Top 10, bad inputs, bot control),
rate-based rules, geo restrictions if applicable
DDoS Shield Standard is automatic; Shield Advanced for
high-value targets
EDGE LOGIC CloudFront Functions / Lambda@Edge for redirects,
header manipulation, simple auth at the edge
Pre-launch checklist worth running explicitly: load test passed at target scale, alerts configured and tested, runbooks written, on-call rota staffed, rollback rehearsed, backups verified by an actual restore, security review completed, cost alarms set, and a status page ready.
PHASE 8 — Observability and Monitoring
You cannot operate what you cannot see. Monitoring tells you that something broke; observability lets you work out why without shipping new code.
8.1 The Three Pillars — Plus One
METRICS Numeric time series. Cheap, aggregatable, alertable.
"p99 latency is 800ms", "error rate is 2.3%"
→ CloudWatch, Managed Prometheus
LOGS Discrete structured events with full context.
"Order 4471 failed: payment gateway timeout after 3 retries"
→ CloudWatch Logs, OpenSearch
TRACES The path of one request across every service.
"Checkout took 2.1s: 1.8s waiting on the inventory service"
→ X-Ray, OpenTelemetry
PROFILES Where CPU and memory actually go inside a process.
→ CodeGuru Profiler, continuous profiling tools
Use OpenTelemetry as the instrumentation standard. It is vendor-neutral, so the backend can change without re-instrumenting every service — which matters, because observability vendors get expensive and teams do switch.
8.2 Correlation IDs — the Prerequisite
Without this, distributed debugging is guesswork. Generate a correlation/trace ID at the edge, propagate it through every synchronous call and every asynchronous message, and include it in every log line and every error response.
Client → [trace: abc123] → API Gateway
→ Order Service [trace: abc123]
→ Payment Service [trace: abc123]
→ SQS message [trace: abc123]
→ Notification Svc [trace: abc123]
Then one query reconstructs the entire failed request across every service and every queue hop. Returning the trace ID to the client also means a user-reported problem arrives with the exact key needed to investigate it.
8.3 What to Actually Monitor
The Four Golden Signals — the standard starting point for every service:
LATENCY Response time distribution (p50/p95/p99 — never the mean;
averages conceal exactly the tail users complain about).
Track successful and failed requests separately, because
fast failures otherwise flatter your latency graph.
TRAFFIC Requests per second, active users, queue depth
ERRORS Explicit failures (5xx) and implicit ones (200 responses
with a wrong or empty body)
SATURATION How full the system is: CPU, memory, connection pool
utilisation, queue backlog, disk. The leading indicator —
saturation rises before latency and errors do.
Layer the monitoring:
| Layer | Watch |
|---|---|
| Business | Orders/minute, signup rate, revenue, cart abandonment, payment success rate |
| Application | Golden signals per endpoint, dependency call health, cache hit rate |
| Infrastructure | CPU, memory, disk, network, container restarts, task placement failures |
| Data | Replication lag, connection count, slow queries, deadlocks, storage growth |
| External | Third-party API latency and error rate, certificate expiry, quota consumption |
Business metrics are the most valuable alerts you have. A drop in orders-per-minute catches broken checkouts that every infrastructure metric reports as perfectly healthy — the servers are fine, they are just serving a broken flow.
8.4 SLIs, SLOs, and Error Budgets
The framework that converts "the site feels slow" into an engineering decision:
SLI (Indicator) What you measure
"% of requests served successfully in < 300ms"
SLO (Objective) The target
"99.9% of requests over a rolling 30 days"
ERROR BUDGET The permitted failure: 100% − 99.9% = 0.1%
≈ 43 minutes of failure per month
Budget remaining → ship features, take risks
Budget exhausted → freeze features, work on reliability
This makes reliability a shared, quantified conversation rather than an argument between product and engineering. It also prevents over-engineering: if the budget is consistently untouched, the system is more reliable than it needs to be, and that surplus is being paid for in velocity and infrastructure cost.
8.5 Alerting That People Trust
PAGE (wake someone) User-visible impact requiring immediate action
Checkout failing · site down · data loss risk
· error budget burning fast
TICKET (business hours) Degradation with headroom
Disk at 80% · elevated retries · one node unhealthy
DASHBOARD ONLY Informational; no notification
Rules for alerts that stay useful:
- Alert on symptoms, not causes. "Checkout error rate > 2%" beats "CPU > 80%" — high CPU may be harmless, while users failing to check out never is.
- Every page must be actionable and have a runbook. If the responder's only action is to acknowledge it, delete the alert.
- Alert fatigue is a real outage risk. A team that receives 40 noisy pages a week will miss the one that matters. Ruthlessly tune or delete noisy alerts.
- Use multi-window burn-rate alerts on the error budget rather than raw thresholds: page on a fast burn, ticket on a slow burn.
- Test alerts — deliberately break something in staging and confirm the page actually arrives at the right phone.
8.6 Dashboards
Build for the audience:
- Service dashboard — golden signals, dependencies, deploy markers overlaid on the graphs (so "when did this start?" is answered instantly)
- Business dashboard — funnel, revenue, active users
- On-call dashboard — one screen answering "is anything broken and where?"
- Cost dashboard — spend by service, trend, anomalies
Annotating deploys on every graph is a small change that resolves a large share of incident investigations in seconds.
PHASE 9 — Reliability Engineering
9.1 Designing for Failure
In a distributed system, failure is the steady state, not the exception. Every remote call must assume the other side can be slow, wrong, or absent.
Timeouts — every network call has one. A missing timeout means one slow dependency exhausts your threads and takes down a healthy service. Timeouts should tighten as you go deeper in the call stack.
Retries with exponential backoff and jitter — retry only idempotent, transient failures. Naïve immediate retries turn a brief blip into a self-inflicted DDoS as every client retries in perfect synchrony; jitter spreads them out. Cap total attempts.
Circuit breakers — after N consecutive failures, stop calling the failing dependency and fail fast, then probe periodically for recovery. This prevents one sick service from consuming every caller's thread pool and cascading a single failure system-wide.
Bulkheads — isolate resource pools per dependency so exhaustion in one path cannot starve the others.
Graceful degradation — decide in advance what "reduced but working" means. Recommendations unavailable → show bestsellers. Search down → show the category tree. Never let a non-critical dependency break a critical path.
Dead letter queues — messages that fail repeatedly go to a DLQ for inspection rather than blocking the queue forever. Alert on DLQ depth; a filling DLQ is a silent, ongoing data-loss event.
Idempotency everywhere — retries and at-least-once delivery guarantee duplicates. Every mutating operation should be safe to execute twice.
9.2 Disaster Recovery
Backups Automated, encrypted, cross-region copies,
retention matching compliance requirements
Restore drills Quarterly at minimum, timed against the stated RTO.
An untested backup is a hope, not a plan.
Runbooks Step-by-step recovery procedures, written before
the incident, tested by someone who did not write them
Game days Deliberately break production-like systems and
practise the response as a team
Chaos engineering Continuously inject failure (AWS Fault Injection
Simulator) to prove resilience holds as the
system changes
DR strategies, cheapest to fastest: backup-and-restore (hours) → pilot light (tens of minutes) → warm standby (minutes) → active-active (seconds, at roughly double the cost). Pick based on the RTO/RPO committed in Phase 1, not on ambition.
9.3 Incident Response
DETECT Automated alert (ideally before any customer notices)
TRIAGE Assess severity, page the right people, open a channel
MITIGATE Restore service FIRST — roll back, flip the flag, shift
traffic, scale up. Root cause can wait; users cannot.
RESOLVE Fix the underlying issue properly
LEARN Blameless post-mortem with concrete, owned action items
Defined severity levels, a named incident commander, a status page for customer communication, and — most importantly — blameless post-mortems. The moment retrospectives assign blame, people stop reporting problems early, and you lose the early warnings that prevent the next outage. The goal is always "what about the system allowed this?" not "who did it?"
PHASE 10 — Security
Security is continuous and layered, not a pre-launch checkbox.
10.1 Defence in Depth
EDGE WAF, DDoS protection, rate limiting, bot control
NETWORK VPC isolation, security groups, private subnets, no
public database endpoints
IDENTITY Strong authN, MFA, short-lived tokens, RBAC/ABAC
APPLICATION Input validation, output encoding, parameterised queries,
CSRF/XSS/SSRF protection, secure headers
DATA Encryption at rest (KMS) and in transit (TLS 1.2+),
field-level encryption for sensitive PII, tokenised
payment data
SECRETS Secrets Manager / Parameter Store with automatic
rotation — never in code, config files, images, or
environment files committed to git
AUDIT CloudTrail everywhere, immutable log archive in a
separate account, GuardDuty, Security Hub, Config rules
10.2 Identity and Access
- Least privilege, always. Start from deny, grant narrowly, review regularly.
- No long-lived IAM users for services. Use IAM roles — roles for ECS tasks, IRSA for EKS pods, instance profiles for EC2.
- Federated human access via SSO with MFA; no shared credentials.
- Just-in-time elevation for production access, time-boxed and logged. Standing production admin access is an unnecessary standing risk.
- Rotate everything automatically — credentials, keys, certificates.
10.3 Security in the Pipeline (Shift Left)
Pre-commit Secret scanning (a leaked key in git history is
permanently leaked — assume compromise and rotate)
Commit SAST on application code
Dependencies SCA/CVE scanning with automated update PRs
Build Container image scanning; block HIGH/CRITICAL
IaC Terraform/CloudFormation policy scanning (public S3
buckets, open security groups, unencrypted volumes)
Deploy Image signature verification, admission policies
Runtime GuardDuty, Inspector, anomaly detection
Periodic Penetration testing, red-team exercises, access reviews
10.4 Compliance and Privacy
Depending on jurisdiction and industry: GDPR (consent, right to erasure, data residency), PCI-DSS (never store raw card data — tokenise via the payment provider), SOC 2, HIPAA. These are architectural constraints, not paperwork bolted on afterwards. "Right to erasure" in particular is genuinely hard once personal data has been copied into caches, search indexes, event streams, analytics warehouses, and backups — design for deletion from the beginning.
PHASE 11 — Cost Management
Cloud cost is an engineering concern. Uncontrolled AWS bills are one of the most common ways a technically successful product becomes commercially unviable.
11.1 Visibility First
- Tag everything — environment, service, team, cost-centre — and enforce tagging with Config rules. Untagged resources are unattributable spend that nobody will ever claim.
- Enable Cost Explorer and budgets with alerts on day one, not after the first shocking invoice.
- Set anomaly detection to catch runaway spend within hours rather than at month-end.
- Review spend monthly with the engineers who own the services, not only with finance.
11.2 Common Optimisations
Right-size Most workloads are provisioned 2–4x larger than
needed. Use Compute Optimizer.
Savings Plans / RIs 40–70% savings on predictable baseline load.
Cover the baseline, leave the peak on-demand.
Spot instances Up to 90% off for fault-tolerant work — batch
jobs, CI runners, stateless workers
Auto-shutdown Non-production environments off nights and
weekends: roughly 70% saving on those accounts
S3 lifecycle Transition to Infrequent Access → Glacier by age
Log retention Unbounded log retention silently becomes a major
line item
NAT Gateway A classic surprise cost — use VPC endpoints for
AWS service traffic
Data transfer Cross-AZ and cross-region egress adds up fast;
keep chatty services in the same AZ where possible
Idle resources Unattached EBS volumes, orphaned load balancers,
forgotten dev stacks, old snapshots
Caching Every cache hit is a database query not billed —
caching is a cost optimisation as much as a
performance one
The highest-leverage habit: make cost visible to the engineers who create it. Teams that see the cost of their own service optimise it without being asked.
PHASE 12 — Operate and Evolve
12.1 Day-2 Operations
Launch is the beginning, not the end. The ongoing work:
- On-call rotation — sustainable, fairly compensated, with a real handover process. Burning out the on-call engineer is a reliability risk.
- Runbooks for every alert, kept current — a stale runbook is worse than none, because it is trusted.
- Patching cadence — OS, runtime, base images, dependencies. Automate the boring updates so the urgent ones get attention.
- Regular restore and failover drills.
- Capacity planning ahead of known events.
- Deprecation discipline — remove old API versions, dead feature flags, unused services. Systems accumulate; without deliberate removal, complexity only grows.
12.2 Managing Technical Debt
Debt is not inherently bad — it is a loan, useful when taken deliberately and repaid on schedule. It becomes dangerous when it is invisible and unacknowledged.
- Maintain a visible debt register with impact and estimated cost
- Allocate a standing capacity budget (commonly 15–20% of each sprint) — debt work that competes with features in every planning meeting always loses
- Apply the boy-scout rule: leave touched code better than you found it
- Prioritise debt that slows delivery or threatens reliability over purely aesthetic refactoring
12.3 Continuous Improvement Loop
Measure → Learn → Change → Measure
│ │ │
│ │ └── ship, experiment, refactor
│ └── post-mortems, user research, analytics
└── DORA metrics, SLOs, business KPIs, cost per transaction
The DORA metrics are the standard benchmark for delivery health:
| Metric | Elite performance |
|---|---|
| Deployment frequency | On-demand, multiple times per day |
| Lead time for changes | Under one hour, commit to production |
| Change failure rate | Under 15% |
| Time to restore service | Under one hour |
These four correlate strongly with organisational performance — and, importantly, speed and stability rise together rather than trading off. Teams that deploy more frequently have fewer failures, because small changes are easier to verify, and easier to reverse.
Reference: AWS Service Selection
| Concern | Primary choice | Alternatives |
|---|---|---|
| DNS | Route 53 | — |
| CDN | CloudFront | — |
| Edge security | WAF + Shield | — |
| API entry | API Gateway (managed features) | ALB (simpler, cheaper at volume) |
| Containers | ECS on Fargate | EKS (complex platforms), App Runner (simple apps) |
| Serverless compute | Lambda | — |
| Relational data | Aurora / RDS | — |
| Key-value at scale | DynamoDB | — |
| Cache | ElastiCache (Redis/Valkey) | DAX for DynamoDB |
| Search | OpenSearch | — |
| Object storage | S3 | — |
| Events | EventBridge | SNS (fan-out), Kinesis (streams) |
| Queues | SQS | Amazon MQ (legacy protocols) |
| Workflows | Step Functions | — |
| Secrets | Secrets Manager | Parameter Store (cheaper, simpler) |
| CI/CD | CodePipeline / CodeBuild | Jenkins, GitHub Actions, GitLab CI |
| Artifacts | ECR | CodeArtifact for packages |
| IaC | Terraform or CDK | CloudFormation, Pulumi |
| Observability | CloudWatch + X-Ray | Managed Prometheus/Grafana, third-party |
| Identity (end users) | Cognito | Auth0, Okta, self-managed |
| Analytics | Athena over S3 | Redshift for a full warehouse |
Maturity Roadmap — What to Build When
Not everything above belongs in version one. Building it all up front is its own failure mode: the team spends months on platform work and never validates the product.
STAGE 1 — VALIDATE (0 → first users)
Modular monolith · one managed database · App Runner or Fargate
Basic CI/CD · CloudWatch logs and a handful of alarms
Managed auth · daily backups
Goal: learn whether anyone wants this. Optimise for speed of change.
STAGE 2 — STABILISE (real users, real money)
Multi-AZ everything · read replicas · Redis caching layer
Staging environment · automated tests as merge gates
Structured logging · dashboards · real on-call and runbooks
Blue-green or canary deploys · IaC for all infrastructure
Goal: stop breaking. Reliability now has a business cost.
STAGE 3 — SCALE (growth, larger team)
Extract services where scaling or team boundaries demand it
Async event backbone · CQRS/read models where justified
Distributed tracing · SLOs and error budgets
Multi-account AWS org · progressive delivery with auto-rollback
Feature flag platform · cost management practice
Goal: grow without the team or the system seizing up.
STAGE 4 — OPTIMISE (scale and maturity)
Multi-region if genuinely required · chaos engineering
Platform/internal-developer-platform team · self-service tooling
Advanced cost optimisation · compliance certifications
Goal: efficiency, resilience, and developer velocity at scale.
Do not skip stages, and do not build stage 3 during stage 1. Most failed engineering efforts are not caused by insufficient architecture — they are caused by architecture built for a scale that never arrived, at the cost of the iteration speed needed to reach it.
The Principles That Actually Carry
Everything above compresses into a short list:
- Requirements drive architecture. Get real numbers before drawing boxes.
- Start simple; earn complexity. A modular monolith beats premature microservices almost every time.
- Build once, promote the same artifact through every environment.
- Configuration comes from the environment, never the artifact.
- Everything is code — application, infrastructure, pipeline, policy, documentation.
- Design for failure, because in distributed systems failure is the normal case.
- Instrument first. You cannot fix, scale, or optimise what you cannot see.
- Automate the repeatable; reserve human judgement for the genuinely ambiguous.
- Security and cost are design constraints, not clean-up phases.
- Small, frequent, reversible changes are the foundation of both speed and stability — they are the same discipline, not a trade-off.