source: .forge/guides/aws-setup-explained.md · generated — edit the markdown, not this file

The AWS setup, explained

A plain-English map of how the CRM’s AWS estate is organised, who and what can reach it, how the database is reached, and how a new environment is made. True as of 14 Oct 2026; §3, §4, §6 and §7 updated on 21 Oct 2026, after forge-et-iaac#3 was applied; §2 and §3 updated on 26 Oct 2026. The record of what is built and what is still owed is .forge/design/environments-and-identity.md §3–§4; when this guide and that file disagree, that file wins.

1. The words, in one line each

Word What it means here
Region A physical AWS location. Everything we build is in us-east-1 (North Virginia). Two older things sit in us-west-2 (Oregon): the Terraform state bucket and the sign-in service (IAM Identity Center).
Account A sealed box with its own resources, its own bill line and its own logins. Nothing in one account can see another unless someone grants it.
Organization A group of accounts under one management account, which pays for all of them and can set rules over them.
Environment A complete running copy of the CRM: dev (for the team), UAT (for Empire to test a release), production (the real one). An environment lives inside an account; F38 plans one account for dev and one for production.
IAM user A named identity inside one account, with a password for the console and/or an access key (a long-lived key pair) for the command line.
IAM role An identity with no password or key of its own. Something takes on the role for a short time and gets temporary credentials — the running backend, a GitHub job, or a person. Roles are the safer way to grant access.
Policy The list of actions an identity may perform (for example “read this secret” or “everything”). It is attached to a user or a role.
Identity Center (SSO) AWS’s single sign-on for people: one login with MFA that lets you take on a role in each account you are allowed into.
Terraform Code that describes the infrastructure. Running it makes AWS match the code. Ours is in forge-et-iaac.
Terraform state Terraform’s record of what it has built, kept in an S3 bucket so every run knows what exists.

2. Our accounts

flowchart TB
  subgraph ORG["AWS Organization — paid by Empire Teas"]
    MGMT["Management account (ets-org, 331342166625)<br/>• pays the bill for all accounts<br/>• Identity Center: people's logins (us-west-2)<br/>• DNS zone crm.empire.lk<br/>• Terraform state bucket et-ops (us-west-2)<br/>• role dns-dev: lets dev write only dev.crm.empire.lk names"]
    DEV["Dev account (ETS-DEV, 570417736903, us-east-1)<br/>• the dev environment: backend, database, CRM frontend<br/>• Bedrock (Claude) and the AgentCore Gateway<br/>• IAM users svc-github and svc-harness"]
    PROD["Production account — planned (F38-a FR-1), not confirmed yet"]
  end
  MGMT --- DEV
  MGMT --- PROD
  • The management account is the top of the Organization. Kavindu Anthony holds it for Empire Teas, and Empire’s card pays for everything in it and below it. Nothing of the CRM runs there. It holds what the other accounts share: the people’s logins (Identity Center), the crm.empire.lk DNS zone, and Terraform’s state bucket. Dev’s Terraform writes DNS records into the zone by taking on the dns-dev role, which can only change dev.crm.empire.lk names.
  • The dev account holds the whole dev environment (§3). The team works here.
  • The production account is required by F38 (FR-1) and AD #4 but has not been confirmed as created. It gets the same building blocks as dev, with production settings (§6).
  • UAT is temporary: one is built with terraform apply for Phase 1’s acceptance round, kept for Phase 2’s round too, and removed afterwards with terraform destroy (AD #23). It runs in the dev account (AD #4 as refined on 2026-10-23, F38 OQ-A, F38-h), so production’s data and keys stay alone in production.

3. What runs in dev

flowchart LR
  U["Browser"] -->|https://dev.crm.empire.lk| CF["CloudFront"] --> S3["S3 bucket et-dev-spa<br/>(the React app's files)"]
  U -->|https://api.dev.crm.empire.lk| ALB["Load balancer (HTTPS)"] --> ECS["ECS Fargate: one backend task"]
  ECS --> RDS[("RDS PostgreSQL et-dev")]
  ECS --> SM["Secrets Manager<br/>DB password, Entra values"]
  ECS --> SQS["SQS queues<br/>scheduler, mailbox ingest, lead triage"]
  EB["EventBridge Scheduler"] --> SQS
  ECS --> S3A["S3 bucket et-dev-attachments<br/>(uploads)"]
  ECS --> KMS["KMS keys<br/>storage, mailbox token"]
  ECS --> BR["Bedrock (Claude)"]
  • The frontend is static files in an S3 bucket, served by CloudFront. A push to forge-et-fe main builds it and copies it there.
  • The backend is one container on ECS Fargate behind a load balancer. A push to forge-et-be main builds the image, stores it in ECR and starts a new task.
  • The database is RDS PostgreSQL. It has no public address, and only machines inside the dev network may connect to it (§5).
  • The backend reads its database password and the Microsoft sign-in values from Secrets Manager when it starts. It reaches AWS through its task role, so it holds no access key.
  • Built on 20 Oct 2026 by the delivery team’s Terraform (forge-et-iaac#3): the uploads bucket, the KMS keys, the mailbox ingest queue, EventBridge Scheduler’s schedules and the backend’s permissions for them. Since 22 Oct 2026 the backend’s task definition names the uploads bucket, the mailbox token key and the ingest queue (forge-et-be#300), so uploads and the mailbox are ready to be tried on dev (F38-f, F38-g).
  • Also built: the capture app’s hosting (forge-et-iaac#5, serving the app since 25 Oct 2026) and the alert topic with the email sent when a backend deploy fails (forge-et-iaac#7). Dev has no CloudWatch alarms, by decision (F38 Rev 23). Still to build: a way for people to reach the database (§5). The up-to-date list is environments-and-identity.md §4.
  • Left out of dev on purpose (F38 Rev 13): a CloudTrail trail (AWS keeps 90 days of who-did-what for free), root-user protection, a password policy, alternate contacts and the account-wide S3 public-access switch. Production gets all of them. Budgets, their alerts and the cost allocation tags are handled by Shyamin Ayesh, Kavindu Anthony, Malinda Wellalage and Thilina Abeyrathna, not the delivery team, which keeps costs down through its design and daily use (F38 Rev 18).

4. Who can reach the account, and how

Identity Kind Who or what uses it Access
Shyamin’s login Identity Center user Shyamin Ayesh Administrator, through the ADMINISTRATORS permission set
svc-github IAM user, access key only GitHub Actions: forge-et-iaac (Terraform), forge-et-be and forge-et-fe (deploys) AdministratorAccess. One key, stored in each repository’s dev environment secrets. Accepted for dev; production gets separate, narrowed users (F38 Rev 9)
svc-harness IAM user, access key (and a console password) The delivery team and Claude on their laptops Read everything, plus IAM, Bedrock and Marketplace. Accepted for dev (F38 Rev 9). Claude only reads with it, and asks the lead before any call that is not a read — even a one-line test call to a model (.claude/CLAUDE.md § ASK FIRST)
et-dev-backend-execution Role ECS, when starting the backend Pull the image, read the DB and Entra secrets, write logs
et-dev-backend-task Role The running backend Reads the scheduler, mailbox-ingest and lead-triage queues, and sends to the last two. Reads and writes the uploads bucket. Uses the storage and mailbox-token KMS keys. Calls the Claude models on Bedrock (since 20 Oct 2026; more as features need it).
et-dev-scheduler Role EventBridge Scheduler Send to the scheduler queue only
et-dev-agentcore-gateway Role The AgentCore Gateway Call web search only
OrganizationAccountAccessRole Role Administrators of the management account Full administrator in dev. It is present in dev and trusts the management account, so whoever controls the management account controls dev too
Personal logins for the developers Identity Center users — Not used in dev: the team reaches dev through svc-harness’s access key (the next paragraph). Production gets read-only personal logins with MFA (F38 FR-2, F38-i)

How the AWS CLI gets in. The CLI reads a profile. Today the team’s profile is et-dev: svc-harness’s access key, stored on the laptop in ~/.aws/credentials, so aws … --profile et-dev acts as svc-harness. This is how the team works in dev, and it stays that way. Shyamin suggested it when aws login failed on 13 Oct, and the lead confirmed it on 14 Oct. There are no per-person Identity Center logins for dev, and Shyamin asked that no more IAM users be created. To set it up on a laptop, put the key into a profile with aws configure --profile et-dev, region us-east-1. The key is shared through the password vault, never in chat or a file. Identity Center’s aws configure sso route, with short-lived credentials, is what production’s personal logins use.

A user versus a role, in practice: svc-github is a user with a permanent key, so anyone holding the key is svc-github until the key is replaced. The backend’s task role has no key: ECS hands the running task credentials that expire within hours.

5. Reaching the database

Today nobody outside AWS can reach it. The database has no public address. Its firewall (security group) admits port 5432 only from inside the dev network. Nothing in that network is set up to pass a connection through: there is no access host, and ECS Exec is off on the backend service. Only the backend talks to it.

Do we need it? Not often, and not yet (the lead, 14 Oct). Laptops run the same database with the same migrations, the backend’s dev logs are in CloudWatch, and the app shows most data. Direct access helps for checking what a feature wrote, the audit trail (no screen in V1), a migration’s result, or test data. So it is set up the first time a real need comes (environments-and-identity.md §4 item 15). Before asking for it, check whether the logs answer the question:

  • On every start, the backend logs Flyway’s Successfully validated <n> migrations.
  • The audit service logs each event as Audit Event Recorded at DEBUG. A user holding both settings.users.manage and settings.roles.manage (AD #41) can switch that on for a short window through /actuator/loggers.

F38-f proved its database checks that way instead of building the tunnel (26 Oct, F38-f spec Rev 4).

The usual practice, which we follow:

  • In development, engineers read through a private tunnel with a read-only login. Shyamin noted on 26 Oct that opening a path to the database is not the DevOps team’s usual practice, though it is fine for dev with a real use. Agree it with the DevOps team when the need comes.
  • In production, nobody has standing access. Access is for emergencies only: read-only, approved by the lead each time, time-limited and logged (F38-i).

The chosen way costs nothing: a Session Manager tunnel through the running backend (ECS Exec). The backend task already sits inside the dev network and can reach the database, and dev runs around the clock, so it can pass a connection through.

  1. Once, Shyamin:

    • turns on ECS Exec for the backend service;
    • gives the task role the four ssmmessages permissions it needs;
    • lets svc-harness start Session Manager sessions on the dev cluster’s tasks;
    • creates a read-only database login, dev_readonly, with its password in Secrets Manager. This is a database change, so the lead approves it (CLAUDE.md § ASK FIRST).

    People never use the backend’s own login (app): RDS manages its password, and a person’s mistake with it is the backend’s mistake.

  2. A developer’s laptop opens a tunnel through AWS Systems Manager to the backend’s container: the database appears on the laptop at localhost:15432. No open port and no SSH key, and AWS records every session.

  3. Any PostgreSQL client connects to localhost:15432.

The backend task gets a new id on every deploy, so each time the tunnel is opened the two ids are looked up first (below). The tunnel closes if a deploy replaces the task; open it again.

Not chosen:

  • a small EC2 access host (about USD 7 a month, but separate from the backend);
  • a public database open only to our addresses (about USD 3.65 a month for the public address, with the database on the internet, and home addresses change);
  • a VPN (about USD 70 a month or more);
  • the RDS query editor (it works only with Aurora, not with this database).

Setting it up on a laptop (once Shyamin has done step 1)

Step Mac Windows
AWS CLI brew install awscli Install the AWS CLI v2 MSI from AWS, or winget install Amazon.AWSCLI
Session Manager plugin (needed for the tunnel) brew install --cask session-manager-plugin Install SessionManagerPluginSetup.exe from AWS’s “Install the Session Manager plugin” page
A database client DBeaver Community (free): brew install --cask dbeaver-community; or brew install libpq then brew link --force libpq for the psql command line (Homebrew does not put psql on the PATH without the second step) DBeaver Community from dbeaver.io; or pgAdmin 4
Credentials the et-dev profile (svc-harness’s access key, §4) the same

Open the tunnel and leave the window open while you work. First look up the running backend task’s id and its container’s runtime id, then start the session. The --parameters value is in double quotes so PowerShell does not split it at the commas.

On a Mac (Terminal):

TASK_ID=$(aws ecs list-tasks --profile et-dev --region us-east-1 --cluster et-dev --service-name backend --query 'taskArns[0]' --output text | awk -F/ '{print $NF}')
RUNTIME_ID=$(aws ecs describe-tasks --profile et-dev --region us-east-1 --cluster et-dev --tasks "$TASK_ID" --query 'tasks[0].containers[0].runtimeId' --output text)
aws ssm start-session --profile et-dev --region us-east-1 --target "ecs:et-dev_${TASK_ID}_${RUNTIME_ID}" --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters "host=et-dev.c8d6c2gekzm3.us-east-1.rds.amazonaws.com,portNumber=5432,localPortNumber=15432"

On Windows (PowerShell):

$taskId = (aws ecs list-tasks --profile et-dev --region us-east-1 --cluster et-dev --service-name backend --query 'taskArns[0]' --output text).Split('/')[-1]
$runtimeId = aws ecs describe-tasks --profile et-dev --region us-east-1 --cluster et-dev --tasks $taskId --query 'tasks[0].containers[0].runtimeId' --output text
aws ssm start-session --profile et-dev --region us-east-1 --target "ecs:et-dev_$($taskId)_$($runtimeId)" --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters "host=et-dev.c8d6c2gekzm3.us-east-1.rds.amazonaws.com,portNumber=5432,localPortNumber=15432"

Then, in DBeaver: New connection → PostgreSQL → host localhost, port 15432, database et, user dev_readonly, password from Secrets Manager. With psql:

psql "host=localhost port=15432 dbname=et user=dev_readonly sslmode=require"

sslmode=require matters: the database refuses unencrypted connections (rds.force_ssl is on).

Can Claude query it? Yes, in the same way, once the tunnel is open and psql is installed. Opening the tunnel (aws ssm start-session) is not a read, so Claude asks the lead before opening it (CLAUDE.md § ASK FIRST); the developer may also open it themselves. Claude then runs read-only queries as dev_readonly. It never writes to the database without the lead’s explicit permission, and it does not read a secret’s value unless asked to.

6. Making another environment from the Terraform

forge-et-iaac is built for this. Every building block lives once under base/ (networking, rds, ecs, alb, s3, cloudfront, and the rest) and takes the project and environment name as inputs. An environment is one folder under overlays/ that calls those blocks with its own values. Today only overlays/dev exists.

Updated 17 Nov 2026: UAT’s overlay (overlays/uat) and its uat.yaml workflow are merged, and the uat instance was first applied on 2026-11-13. deploy-uat is built for the backend and the CRM, and first released v0.1.0-rc1 to UAT on 2026-11-16 (tracker.yaml F38-h notes). Updated 19 Nov 2026: the capture app is now hosted on UAT at capture.uat.crm.empire.lk and has its own deploy-uat, which first runs with the next -rc tag. Production (F38-i) is still to build. The list below is what each environment needs.

A new environment (UAT or production) needs:

  1. An overlay folder, overlays/uat or overlays/prod, calling the same blocks with that environment’s host names (api.uat.crm.empire.lk, crm.empire.lk, …) and sizes.
  2. Its own state — a separate state file; F38 FR-3 asks for a state bucket in each account.
  3. Credentials for the target account:
    • UAT runs in the dev account. It uses dev’s own svc-github key (F38 Rev 26) and a way to write its DNS names: a dns-uat role, or dns-dev widened to the uat names (the DevOps team’s choice).
    • Production has its own narrowed deploy and Terraform users (FR-19), and its own DNS role (dns-prod) that can change only its own names.
  4. Production-safe settings made into inputs. Some values are written into the blocks today and suit dev only: the database skips its final snapshot, has deletion protection off and is single-AZ, and has no backups or encryption set. Production needs deletion protection, backups (14 days in the proposal) and a final snapshot, and encryption at rest is undecided (F38 Rev 17); a temporary UAT needs the opposite of deletion protection so it can be destroyed. F38-b’s forge-et-iaac#3 already makes the attachments bucket, the ECR repository, the ECS service’s task definition and the Entra secrets take per-environment values for a UAT in the dev account.
  5. A workflow per environment that applies only from the right branch or tag. Dev has one: since F38-b’s forge-et-iaac#3 merged on 20 Oct 2026, dev.yaml runs only on a push to dev or by hand and applies only on dev. UAT and production need their own.
  6. The hand-made steps AWS does not allow in code: the Entra values put into Secrets Manager by hand, SES production access, Bedrock quotas, and people’s logins.

So the structure is right for “one terraform apply per environment”, and the work left is mostly turning dev’s fixed settings into per-environment inputs and adding the overlay, state and credentials for each new environment.

7. The database password rotation, and why it can break dev

  • The decision (the lead, 16 Oct 2026 — F38 Rev 14): the database password is not rotated on a schedule in any environment. RDS keeps managing it in Secrets Manager, so nobody sees it, but its automatic rotation is turned off in Terraform, and no automatic restart is built. It is changed by hand, followed by a redeploy (below), when a leak is suspected or when someone who could read it leaves. The delivery team’s forge-et-iaac#3 (F38-b) turns rotation off in base/rds (aws_secretsmanager_secret_rotation with rotation_enabled = false); it was merged and applied on 20 Oct 2026, and dev’s RDS-managed secret has shown RotationEnabled: false since. Why: the password is never seen by a person and the database is reachable only from inside AWS, so a scheduled change protects little, while one that nothing follows with a restart cuts dev off (the rest of this section).

  • What it is. RDS keeps the database password in Secrets Manager and can change it automatically on a schedule, so a leaked password stops working soon. The default for every new RDS-managed secret is every 7 days. Dev’s does not rotate. A yearly schedule was set by hand on 15 Oct 2026, then turned off by forge-et-iaac#3 on 20 Oct (re-check with aws secretsmanager list-secrets --query 'SecretList[?RotationEnabled==\true`].[Name,RotationRules,NextRotationDate]'`, which lists no RDS secret). The proposal includes “secrets held in AWS Secrets Manager with rotation” (§A.2.3; §A.4’s Secrets Manager row: “database credentials, with rotation”), which is why the decision above is recorded as a post-issue note on it.

  • Why it breaks the backend. The backend reads the password once, when it starts, and keeps it in memory. The database connections it already has stay open after a rotation, but the backend’s connection pool (HikariCP, at its default settings) replaces each connection about every 30 minutes, and a new connection needs the new password. So within about half an hour of a rotation the backend starts failing every request that needs the database, until it is restarted and reads the new password. No data is lost; the app just cannot reach its data.

  • If a rotation does run (one started by hand), any later deploy of the backend (a merge to forge-et-be main) fixes it, because the new task reads the new password, but over a weekend nobody may merge. Then run forge-et-be’s build workflow by hand on main (Actions › build › Run workflow; the manual trigger exists since F38-d, 19 Oct 2026): it deploys again with no merge and no AWS access.

  • The short-term fix, done on 15 Oct 2026 and replaced on 20 Oct by turning rotation off: the schedule was made much longer (a year), which AWS allows for an RDS-managed secret. Changing it with aws secretsmanager rotate-secret needs --no-rotate-immediately, or the command rotates the password at once. If a rotation happens anyway, restart the backend: aws ecs update-service --cluster et-dev --service backend --force-new-deployment (Shyamin or the lead; it is a change, so Claude does not run it unasked).

  • Changing the password by hand. This is the runbook the decision relies on. Shyamin or the lead runs it, never Claude unasked.

    1. Rotate the secret once, now: aws secretsmanager rotate-secret --secret-id <the RDS-managed secret>.
    2. Wait until its LastRotatedDate moves.
    3. Redeploy the backend with aws ecs update-service --cluster <cluster> --service backend --force-new-deployment, and check /actuator/health/liveness.
    4. Turn automatic rotation off again. rotate-secret turns it back on, so run the environment’s Terraform workflow (dev.yaml by hand on dev), which applies rotation_enabled = false again. Check that aws secretsmanager describe-secret --secret-id <the secret> --query RotationEnabled prints false.

    The old password stops working at once, so requests that need the database fail between steps 1 and 3. Do it when the app is quiet.

  • Not chosen: either would let a short schedule run safely, but the lead chose neither (F38 Rev 14, OQ-K).

    • Restarting the backend automatically after each rotation: a rule on the rotation-succeeded event that forces a new deployment, with two database users alternating.
    • Having the backend fetch the password from Secrets Manager each time it opens a connection.

Sources

.forge/design/environments-and-identity.md §3–§4 (what is built and owed); .forge/audits/2026-10-13-aws-dev-account-audit.md (the account review); .forge/specs/F38/cloud-bring-up-spec.md (FR-1, FR-3, FR-10, FR-19, NFR-1, OQ-A, Rev 9); forge-et-iaac at 9eac094; AD #4, AD #22, AD #23 in .claude/CLAUDE.md; AWS documentation: “Password management with Amazon RDS and AWS Secrets Manager”, “Session Manager port forwarding to remote hosts”.

Generated from .forge/guides/aws-setup-explained.md by .claude/lib/render-docs-html.js. Markdown is canonical.