Cloud-gated validation & implementation register
What this is. The one place that lists everything we have deliberately not finished or not
proved because a cloud environment, or the part of it an item needs, did not exist when the item
was deferred. As of 2026-10-26, dev has the AWS account, the Entra certificates and reply URLs,
SES, the KMS keys, the alert topic and deployed hosts; Bedrock serves Haiku 4.5 only; UAT and
production do not exist yet (“Start here” below and environments-and-identity.md §4 hold the
detail). When a cloud
environment is stood up, walk this file top to bottom and validate or implement each open item.
Why it exists. Local development runs against a stub login, a Docker Postgres and LocalStack (fake SQS + S3). Several behaviours are structurally unreachable locally — the real Microsoft sign-in, cross-host cookies and CORS, live Graph, live Bedrock, SES mail, KMS, CloudWatch alarms, EventBridge schedules, the deploy pipelines. They pass “green” locally only because the local path substitutes for them. This register makes the gap explicit so nothing silently reaches UAT unvalidated.
How it is kept current (the rule). Every feature and slice that defers, fakes or stubs something for want of a cloud environment adds its rows here in the same PR that ships the deferral — never a follow-up. Concretely:
- The spec carries an FR naming its cloud-gated items and promising the rows (F20-a FR-16 is the
model), or states in
## Constraints and Dependenciesthat nothing is cloud-gated. - The plan carries a subtask that appends the rows, with this file in
## Files to Modify. spec-reviewer§7,plan-reviewer§6 and/forge-pre-pr-revieweach check for it;/forge-reflectstep 5b is the last check before a feature readsdone, and records a “nothing cloud-gated” line for features that had none, so an absence is a decision, not an omission.- Rows live in §B under the owning feature or slice — one row, one home. §C is the view by capability and points back at §B; only engagement-level rows with no feature or slice owner live in §C (most are carried out by F38, the cloud bring-up).
- Every change adds a dated line to
## Log.
The rule itself lives in .claude/CLAUDE.md § Boundaries (“Register every cloud-gated item …”).
Columns. Item — what is unproved or unfinished. Why cloud-gated — which missing thing
blocks it. Source — the spec FR/AC, plan Decision/Risk, finding or code path that deferred it,
so the reader can go and check. Validate / finish when cloud is up — the concrete step and the
expected result. Owner — who does it, where named. Status: open (not yet validatable) →
validated (checked on a real environment, with evidence) or implemented (was app work owed,
now done), or withdrawn (a later decision removed the thing the row was about; the row stays,
struck through, naming the decision and where its replacement lives). Keep entries accurate; cite
the artifact that deferred the item; do not invent items.
This is descriptive, not a gate. The authoritative risks live in project-prd.md § Risks (R-PRD-),
tracker.yaml setup.* (R-ARCH-), the Foundation review in engagement-gate-runs.md (R-FND-*), the
signal W-1 in design/architecture.md, and the AD table; this file is the actionable walk-list derived from them.
Start here — the checklist for when the AWS accounts exist
The order of work, from today to go-live. Each step points at the rows that hold the detail; the
rows, not this list, are where results are recorded. F38 (.forge/specs/F38/cloud-bring-up-spec.md)
carries steps 1–27 — its definition of done is the dev environment up, the retest recorded and the
first uat instance applied, which is what the Phase 1 client UAT needs. Steps 28–29 are
production and go-live, after Phase 1: a Phase 2 sibling of F38 or the tracker’s releases: list. A step
that names another owner is theirs. As of 2026-09-24 steps 1 and 4 are mostly done (both
certificates generated and uploaded, all reply URLs registered — what remains of each is written in
the step), and step 3’s decision is made (AD #33); no other step had started then. The updates
below and environments-and-identity.md §4 record what has happened since.
Update 2026-09-29 — the AWS account exists. It is Empire Teas’ own account, held by Kavindu
Anthony and billed to Empire Teas directly; Shyamin Ayesh has administrator IAM credentials and is
writing the Terraform, the environments and a login for each developer, expected within the week
(environments-and-identity.md §3). The revised Statement of Work puts the environments and the
deployment pipeline in the Foundation stage, target Fri 16 Oct 2026, assuming the environment
and the team’s access are ready by Fri 2 Oct (.forge/client/project-timeline.md) — so steps 7–20
have about two weeks. Step 6’s mid-October trigger no longer applies. On hold until Shyamin
finishes his setup, and answered from it rather than decided now: how the team reviews the
Terraform (step 2 — its location, forge-et-iaac, was settled the same day), the account structure and sign-in model in step 7. GuardDuty is decided (2 Oct 2026): threat
detection and Malware Protection for S3 are both on (step 13, A-1 — the backend’s cloud profile
needs a real FileScanProvider, and the one built is GuardDuty’s; switching Malware Protection off
later waits until F38-j is done and deployed, and then follows forge-et-be/docs/messaging/scan-result-cloud-contract.md
§ “Switching scanning off and back on (F38-j)” — the setting first, the protection plan after; §B FS-11). (Superseded 2026-10-09: Altrium management decided to go without GuardDuty — neither Malware Protection for S3 nor threat detection — until it is needed or Empire Teas asks; every environment runs F38-j’s scanning-off setting (§B FS-11 block note, §C-8). The client-facing PRD (revised 9 Oct) states it; on 11 Oct the lead decided Empire Teas is not told separately.) The search-provider choice in step 14 waits on Dinil’s spike, as before.
Update 2026-09-30 — the first Terraform is applied. forge-et-iaac (commit daf4f1c) holds the
dev VPC with two public subnets and an empty ECS cluster, applied by its GitHub workflow — in
us-west-2, not us-east-1 (step 7), state bucket included; the move is to be raised with
Shyamin Ayesh while it is only those two resources. (Update 2026-10-01: the move to us-east-1
was agreed with Shyamin, and was applied to forge-et-iaac’s dev on 2026-10-05 (commits dff66ce, 5808711; PR #1, which proposed it, still open as of that date).) Shyamin has also created a Route 53 zone for
all of crm.empire.lk, outside Terraform, and asked for its NS delegation from Empire IT.
The lead accepted the whole-subdomain delegation the same day: F38 Rev 2 and F38-a replace the
uat-only delegation and the two per-environment requests, and step 11 is reworded to match
(environments-and-identity.md §1.1). The same change moved the synthesis model to Sonnet 5.5
(step 9). What step 13 still
needs is listed in environments-and-identity.md §4. Both halves of A-1 are on main (F5-f done 2026-10-04), so a first dev boot no
longer stops there; the boot itself is proved by F38-f (§B F5-f).
Update 2026-10-09 — the backend runs on dev. Shyamin Ayesh applied the rest of the backend’s path
in forge-et-iaac (b7e923a: RDS, ECR, the ECS service and its roles, the load balancer with its
api.dev.crm.empire.lk certificate and record, and the scheduler queue with its dead-letter queue)
and wired forge-et-be’s build workflow to build, push and deploy every merge to main;
https://api.dev.crm.empire.lk/actuator/health/liveness answers UP. The backend boots in a cloud
shape with scanning switched off (F38-j) and Entra placeholders, so this is step 20’s first deploy
without its sign-in: step 15 (the non-prod key in Secrets Manager) and the rest of step 13 (S3,
KMS, the mailbox ingest queue, EventBridge Scheduler, both CloudFront
distributions, the AgentCore Gateway) are still owed, and step 9 is half done — Haiku 4.5 answers,
while Sonnet 5.5 and Haiku 5.5 are refused for the account (§C-5). What was built, and how it
differs from F38 and the lead’s recorded decisions, is listed in environments-and-identity.md §4
(“Checked against F38”); the two items with a date are the database password rotation (the first
RDS-managed rotation, about 16 Oct, leaves the running task with a stale password) and keeping the
Entra private key out of the committed task definition. No row is validated yet: the walk in step
21 starts once sign-in works. (Update 2026-10-13: the AgentCore Gateway is built, and the SP-ARCH-004
stage-2 row in §C is the first validated — the search comparison needs no sign-in or deploy.) The same day, GuardDuty was dropped (Altrium management; §B FS-11
block note): no Malware Protection, no threat detection, scanning switched off by setting in every
environment, the GuardDuty rows withdrawn; Empire Teas is not told separately (lead, 2026-10-11 — the client-facing PRD already states it).
Update 2026-10-14 — sign-in works on dev. On 13 Oct Shyamin Ayesh added the Entra values to
Secrets Manager (forge-et-iaac 259230c) and pointed the backend’s task definition at them
(forge-et-be dfc9be15), and built the CRM frontend’s hosting (dev.crm.empire.lk: certificate,
bucket, CloudFront, record) and its deploy on every push to forge-et-fe main. CRM01 signed in
on https://dev.crm.empire.lk the same day, so steps 15 and 20 are done and A-5 closes on its own
test. Step 18 is done for the backend and the CRM frontend, not the capture app (update
2026-10-26, F38-d WI-7.1: done for the capture app too, since its first deploy on 25 Oct). Still owed from
step 13: the attachments bucket, the KMS keys, the mailbox ingest queue (no job queue — F38 Rev 16), EventBridge
Scheduler and the capture app’s hosting; from step 9: Sonnet 5.5 and Haiku 5.5, whose agreements
are accepted but whose quotas are 0 tokens a minute (§C-5); from step 7, the dev account exists
under the Organization, production is not confirmed, the team reaches dev through svc-harness’s
access key rather than personal logins (the lead, 14 Oct), and CloudTrail does not
exist; from step 8, the
one state bucket (et-ops, management account, us-west-2) is not yet one per account (dev’s
stays there, F38 Rev 13); step 12 (budgets) is not the delivery team’s (F38 Rev 18). The database password still rotates on 16 Oct 23:59:59
UTC (15 Oct: the schedule is now every 365 days, next run 15 Oct 2027. 16 Oct, F38 Rev 14: no
scheduled rotation and no restart in any environment; F38-b’s forge-et-iaac#3 turns the managed
secret’s automatic rotation off in base/rds — applied on 20 Oct, below; §C-3.) What is built and every open ask:
environments-and-identity.md §4. The walk in step 21 can start now for anything that does not
need uploads, the mailbox or the 5.5 models.
Update 2026-10-21 — the delivery team’s Terraform is applied. After Shyamin Ayesh approved
them, the lead merged forge-et-iaac#3 (F38-b wave 1) into dev on 20 Oct (18:33 UTC) as
d230b133, with forge-et-iaac#4 (F11-d’s lead-triage queue) merged into its branch first, and the
merge’s dev.yaml run 39475850266 applied both (28 resources added, 5 changed, 1 replaced). Read in the account on 21 Oct: the
storage and mailbox token KMS keys, the attachments bucket et-dev-attachments, the mailbox ingest
and lead-triage queues with their dead-letter queues, EventBridge Scheduler’s group et-dev with
its schedules (all enabled) and its role, the task role’s S3, KMS, Bedrock and SQS grants, the CRM
distribution’s security headers, the ECR rule for build-* images, the service in the database’s
zone, the faster health check, and the RDS-managed secret’s rotation off. dev.yaml now runs
only on a push to dev or by hand and applies only on dev. So step 13 is done except the capture
app’s hosting. Uploads and the mailbox still do not work on dev: the task definition still
holds the bucket, mailbox URL and token-key placeholders and has no ingest-queue key (step 16,
F38-d FR-6, its WI-4.1). (Update 2026-10-23, F38-d wave 4: forge-et-be#300 (c2e89efa, merged
2026-10-22) set all four values, and the running revision backend:58 carries them (aws ecs describe-task-definition, read 2026-10-23); WI-4.1 adds the certificate check, JAVA_TOOL_OPTIONS
and stopTimeout (forge-et-be#306). Whether uploads and the mailbox work on dev is unproved until
F38-d WI-6.1 checks them.) Checking each applied resource against its §B row is F38-b’s wave 2 (WI-2.1).
Update 2026-10-22 — SES works in dev; Bedrock is refused again. Read as svc-harness on 21 Oct
at 21:13–21:15 UTC. Step 10 is done for dev: the DevOps team added base/ses (forge-et-iaac
0eee623, a direct push to dev applied by dev.yaml run 39553084687), production access was
granted on 21 Oct, and the lead’s test message reached the CRM01 Inbox with SPF, DKIM and DMARC
pass (§B FS-5 SES row). The backend has no SES sender yet. Step 22 (F38-e) proves sending with the backend’s task role through a one-off task, and the backend’s own sender is built by its first consumer, F36-c (F38-e spec, decision 5, 2026-10-23).
Lead triage (F11-d) has its queue, schedule and task-definition key on dev since 20 Oct (§B F11-d).
The service runs backend:51, which still holds the step-16 placeholders. Step 9 is not done: AWS
refused three more quota requests on 21 Oct, so Sonnet 5.5 and Haiku 5.5 are still at 0 tokens a
minute and Haiku 4.5 at 10 requests a minute; the next step is a Support case (§C-5).
Update 2026-10-25 — the task definition is filled; every forge-et-iaac PR is merged. Read as
svc-harness on 25 Oct. Step 16 is done for dev: since forge-et-be#300 (F38-d WI-4.1, 22 Oct) the
task definition sets the attachments bucket, the mailbox’s public address, its token key and its
ingest queue, and the running backend:62 holds them (environments-and-identity.md §4); none of
§D’s “must stay unset” keys is among its 23 environment names. What
they serve, uploads and the mailbox, is still to be proved by use (§B F38-g FR-6/bucket and
FR-6/mailbox, F38-f). The schedules run: on 24 Oct the backend consumed all 173 messages
EventBridge Scheduler sent, the daily price-expiry sweep among them, and none was dead-lettered
(§B F38-g FR-6/timed-jobs). forge-et-iaac#8 (F11-c), #6 (F38-d WI-3.1: immutable ECR tags, a
30 s deregistration delay) and #7 (F38-e WI-1.1: the alert topic et-dev-alerts and the
failed-deploy rule, part of step 22) merged and were applied on 23 Oct; none is open. Step 18 for
the capture app needs only its workflow: forge-et-pwa’s dev Environment has its variables and
the deploy key since 23 Oct (F38-d WI-5.1). Step 9 is unchanged: the 5.5 models at 0 tokens a minute,
Haiku 4.5 at 10 requests a minute, and AWS’s answer to the reply sent on 22 Oct not reported (§C-5).
Before the accounts exist — can be done now
Generate the certificates— done 2026-09-24: two pairs (prod, and non-prod for dev, UAT and the laptops), generated by Shyamin Ayesh, valid two years to 23 Sep 2028, both expiry dates recorded inenvironments-and-identity.md§5, private keys in a restricted vault in the password manager. Still open with Shyamin: where UAT reads the non-prod key from, and the rotation routine (environments-and-identity.md§2.3); a calendar reminder a month before expiry (§B FS-6, the expiry row).Decide whether Terraform lives inDecided 2026-09-29: a separate repository,forge-et-be/terraform/(AD #23) or a separate repository (§C-3).altrium-io/forge-et-iaac, for all DevOps work (AD #23 revised). How the delivery team reviews it waits on Shyamin Ayesh’s setup.Make the cross-host cookie decision (A-4).Decided 2026-09-24 — AD #33: the backend keeps itsapi.*address and the apps call it cross-site. The cookie code change is built — F5-j (forge-et-be#78, forge-et-fe#59, merged 2026-09-27) — so what it leaves is the per-environment values in §D and the §B F5-j check that both sides use the same name.- Make the Empire IT asks on the §5 agenda — Kavindu Anthony is the only person with Azure
access. Done 2026-09-24: the three mailbox reply URLs on
api.*, the two laptop reply URLs and both certificates’ public halves (confirmed by Kavindu’s portal screenshots). Still to ask: a re-read of the granted Graph permissions (User.ReadBasic.All), the renewal routine, and the production administrator’s email — the F5-k placeholder value, a plain value in production’s task definition (AD #40, 2026-10-28) (§B F5-k;environments-and-identity.md§5). The client-facing.forge/client/dependencies.md§1.2 marks the done items and lists the administrator’s email as an item from Empire. Draft F38’s spec (and its split into smaller siblings) so the build starts the day the accounts land. Optionally build F5-i to prove sign-in and the mailbox from a laptop first (§C-1).Done: F38’s spec is approved and split into siblings (each one’s spec status is intracker.yaml); F5-i is done — its live run asCRM01was recorded on 2026-09-30.If the accounts are still missing by mid-October, raise the Phase 1 UAT date with Empire (A-5).Overtaken 2026-09-29: the account exists, and the revised Statement of Work moves every date day for day if the environment is not ready by Fri 2 Oct. Separately, ask Empire the three §C-8 client decisions — single- or multi-AZ database, cross-region backup copy, WAF at go-live — so the production Terraform is not held up.
Day one — accounts, Terraform bootstrap and the long-lead requests (they have approval delays)
- Accounts per environment (dev and prod permanent, the
uatmodule applied once for Phase 1’s round and kept to the end of Phase 2’s — AD #4, #23), inus-east-1; IAM Identity Center with MFA and the rest of the security baseline (§C-8). - Bootstrap Terraform per account — the S3 state bucket, locked by S3’s own lock file (
use_lockfile, AD #23 as revised 2026-10-02 — no DynamoDB table). From here on every resource, including the ones in steps 11–12, is created by Terraform (step 10 is a support request, not a resource). Update 2026-10-15 (F38 Rev 13): in dev the state stays inet-opsin the management account, and thedns-devrole,et-opsand the two IAM users stay made by hand, as named dev exceptions to AD #23; production’s bootstrap is F38-i’s. - Make Bedrock usable for Haiku 4.5 and Sonnet 5.5 (§C-5; Sonnet 5.5 replaced Sonnet 5 on 2026-09-30). Model access is on by default, so
there is no access request: submit Anthropic’s first-time-use form once (from the
Organization’s management account, which the member accounts inherit); give the identity that
makes the first call the three
aws-marketplacesubscription permissions; confirm each account has a payment method; then read both accounts’ quotas for both models in Service Quotas and request an increase wherever one is below the expected volume (F38-a FR-7). The account is now Empire Teas’ own (2026-09-29), so the form is submitted by whoever holds its administrator access, and the payment method is Empire’s card. How each model is called (in-Region, through aus.inference profile, or — Sonnet 5.5’s only route today — the Global profile;ai-provider-strategy.md§4b) sets the model ids in the runtime config and the task role’s IAM policy — the §C-5 Haiku 4.5 row and F38-a Q-3. Update 2026-10-15 (F38 Rev 12, F38-a WI-1.1): in dev, F38-a covers Haiku 4.5, Sonnet 5.5 and Haiku 5.5 — the first-time-use form and the agreements are done, the quotas read before the increase are recorded (0 tokens a minute for both 5.5 models, 10 requests a minute for Haiku 4.5, no increase requested yet), and the increase and one call to each 5.5 model are what remain (environments-and-identity.md§4 “F38-a — the development account, checked”, AC-4). Production’s form check, quotas and the management account’s quota request template are F38-i’s. Update 2026-10-22: the increase was requested — six requests on 15 Oct and three on 21 Oct, each at the AWS default because Service Quotas refuses a lower value — and AWS refused all nine without raising a quota (“unable to approve your request at this time”). Next: Shyamin Ayesh sends the lead’s reply through an AWS Support “Service limit increase” case, asking for Haiku 4.5 at 100 requests a minute and the two 5.5 models at 400,000 tokens a minute, or any non-zero value (environments-and-identity.md§4 item 4, the 22 Oct update; every case number in its AC-4 subsection). Update 2026-10-26: the reply was sent on 22 Oct, and AWS answered with no increase: the account keeps new-account limits until it has a few billing cycles of Bedrock usage and spend, with no date. Step 9 stays open for dev and for production; meanwhile Haiku 4.5 runs every AI feature in every environment and the backend is to pace its Bedrock calls (ai-provider-strategy.md§5 row 10; §C-5). Update 2026-10-26 (F38-a closed): F38-a’s part of this step ends with the quotas recorded; the one call each to Sonnet 5.5 and Haiku 5.5 in dev is the §C-5 row “Sonnet 5.5 and Haiku 5.5 answer one call each in dev” (F38-a spec Rev 4). - Request SES production sending access (§B FS-5 SES row). Update 2026-10-21 (F38-a WI-3.1):
dev’s production access was requested and granted on 2026-10-21, case 179256302200929, the same
day the
dev.crm.empire.lkidentity was created; production’s request is F38-i’s; evidence: environments-and-identity.md §4 “F38-a — the development account, checked”, AC-5. - Delegate
crm.empire.lkonce, then let Terraform write every record (F38 Rev 2, F38-a FR-10). Empire IT’s one request is the four NS records namedcrmin theempire.lkzone (environments-and-identity.md§1.1 — asked 2026-09-30, live 2026-10-01). Now the delegation is live, the zone is imported into Terraform, and Terraform writes into it the ACM validation records, a CAA record allowingamazon.com, the SES DKIM, MAIL FROM and DMARC records (when the sending domain is undercrm.empire.lk), and the host records for the load balancer and each CloudFront distribution (alias records at the apex; dev’s written since F38-b’s applies, F38-i’s for production). Only a sending domain onempire.lkitself would need a further request (F38 OQ-E). Update 2026-10-15 (F38 Rev 13): the zone is not imported yet, and its import moves to F38-i, before production’s records are written; dev writes itsdev.records into the zone through thedns-devrole. The CAA record is F38-i’s too. Dev’s SES records are F38-a FR-8. - AWS Budgets and alerts per account (§C-8) — not the delivery team’s since 18 Oct (F38 Rev 18).
The estate and the first deploy
- Terraform the estate: network, ECR, RDS (with the backup settings in §C-8, and single- or
multi-AZ per Empire’s answer), S3, SQS + DLQ, EventBridge Scheduler and its role, KMS keys,
Secrets Manager, ECS + ALB, both CloudFront distributions, the AgentCore Gateway (A-2, §C-3,
§B FS-1/FS-2/FS-7/FS-9/FS-11). (Done for dev on 2026-10-20, when forge-et-iaac#3 was
applied, except the capture app’s CloudFront distribution and bucket — the update of
2026-10-21 above.) (Update 2026-10-21, F38-b WI-2.1: the keys, the attachments bucket,
the ingest queue, the schedules and the grants are recorded as applied in
environments-and-identity.md§4 “F38-b — the dev estate, checked”. Still owed for dev: the capture app’s hosting, and the DevOps team’s optional database and network changes, which F38-b’s WI-4.1 records as found.) (Update 2026-10-22: the capture app’s hosting is done for dev — forge-et-iaac#5 (F38-b wave 5) merged as5e078c0dand applied bydev.yamlrun 39740047356; its bucketet-dev-capturestays empty until F38-d’s capture deploy, which waits onforge-et-pwa’sdevEnvironment.environments-and-identity.md§4, “What it holds”.) (Update 2026-10-25: that Environment is complete since 23 Oct, so the capture deploy waits only on its workflow, F38-d WI-5.1. forge-et-iaac#6 (F38-d WI-3.1) and #7 (F38-e WI-1.1) are applied too, 23 Oct.) (Update 2026-10-26: the workflow exists —forge-et-pwabuild.yaml(f7d6b3f, F38-d wave 5) ran on 25 Oct by hand (19:02 UTC) and on a push tomain(20:02 UTC), both successful, andhttps://capture.dev.crm.empire.lk/answers200. F38-d WI-7.1 records the capture deploy.) (Update 2026-10-26, F38-b WI-4.1: done for dev — the database, network and both frontends’ hosting recorded as found, and a sweep finding nothing outside Terraform; the DevOps team makes no more optional dev changes (the lead, 26 Oct). The no-changedev.yamlplan, the last part, was recorded the same morning — run 40278225094 (A-2, closed;environments-and-identity.md§4 “F38-b — the dev estate, checked”, AC-1). Production anduatare F38-i’s and F38-h’s.) - SP-ARCH-004 stage 2 — as soon as Bedrock access (step 9) and the AgentCore Gateway
(step 13) exist; it needs no dev deploy. The search-provider comparison (Bedrock AgentCore Web
Search versus the pre-decided fallback, Tavily) with Claude judging the saved results, plus the
one-off Java AgentCore call; record the provider choice before the F11-c spec (§C-5;
design/ai/lead-research-source-options.md§7, §10). Owner Dinil. (Done 2026-10-13 for the comparison: AgentCore came last on quality and Exa first —lead-research-source-options.md§11. The provider is not chosen yet; the Java call is owed only if AgentCore is.) (Update 2026-10-21: the provider is Exa on its free credit, provisional until Empire answers (§11, Revs 8–9), so no Java AgentCore call is owed; F11-c’s cloud items will be its §B block, written by its shipping PRs — spec FR-H3.) - Copy the non-prod private key from the vault into dev’s Secrets Manager (the vault copy stays
while laptops and UAT still read it); confirm the non-prod certificate and the dev mailbox
reply URL are on the registration
(
environments-and-identity.md§2.2, §2.3). (Done 2026-10-13 for the key, the certificate and both IDs, assecretsreferences; the mailbox reply URL is proved with F20-a’s rows.) - Fill the task definition from §D — including
ET_SECURITY_SEED_ADMIN_EMAIL,crm01@empire.lkon dev (§B F5-k); confirm the “must stay unset” list is unset. Land the Trivy enforcement flip before the backend is reachable from the internet (§B FS-4).Removed by AD #38 (2026-10-02): vulnerability scans report on every PR and dev deploy and never block them. The blocking check is the release check indeploy-uatanddeploy-prod(F38 FR-35, §B FS-4).- Wire the deploy pipelines and the deploy service user’s key as a GitHub Environment secret in
each app repo (F38 FR-19, Rev 4; §B FS-1, §C-6), and set the GitHub Actions
environment variables each build needs — each frontend’s
VITE_API_BASE_URLandVITE_CSRF_COOKIE_NAMEper environment (non-secret only; the bundle is public — AD #22, §D).forge-et-fe’svite buildalready refuses a cloud build that setsVITE_API_BASE_URLwithoutVITE_CSRF_COOKIE_NAME(F5-j, built); what the pipeline still owes is setting both, with the name equal to the backend’s for that environment (§B F5-j). (Done 2026-10-13 forforge-et-beandforge-et-feon dev —forge-et-fe’sdevenvironment setsVITE_API_BASE_URL=https://api.dev.crm.empire.lkandVITE_CSRF_COOKIE_NAME=XSRF-TOKEN-DEV, matching the backend;forge-et-pwahas no deploy yet. Update 2026-10-26:forge-et-pwa’sdevEnvironment holds the same two variables, which the delivery team set when it created the Environment for forge-et-iaac#5, and itsbuild.yamlhas deployed the capture app to dev since 25 Oct (step 13 above). How both workflows differ from F38 FR-19 and FR-20:environments-and-identity.md§4 rows 9, 11 and 15.) - Clear A-1: the GuardDuty scanner is on
main(F38-c wave 1) — unused since 2026-10-09, when GuardDuty was dropped and every environment runs F38-j’snoneprovider — and the realRecordVisibilityCheckis onmain(F5-f wave 1, forge-et-be#153); the first dev boot proves the backend passes the guard (§B F5-f). F5-j (the cross-site CSRF cookie) is onmain(forge-et-be#78 and forge-et-fe#59, merged 2026-09-27); only its per-environment values remain (§D). Check A-3: F20-b’s Wave 1 is never deployed without its Wave 2. - First dev deploy; the backend boots; the F5-k migration seeds the IT Admin role and user from
dev’s placeholder value (
crm01@empire.lk); sign in asCRM01through the real Microsoft sign-in (§B F5-k). The runtime first-administrator job this step once named is retired by F5-k. (Done 2026-10-13: Shyamin Ayesh and the lead each signed in asCRM01onhttps://dev.crm.empire.lk. The §B F5-a and F5-k rows are stillopen; they are markedvalidated, with their evidence, in the walk in step 21.)
Prove it on dev
- Walk §B slice by slice (FS-1 … FS-11), then §C-1 to §C-6, marking each row
validatedwith evidence. - Observability: log groups and 30-day retention, the alarm substrate, every feature-owned alarm, SES mail (§B FS-5, §C-4, §C-8). Since 2026-10-23 (F38 Rev 23): in dev the alarm substrate is the alert topic, a one-off test alarm and the failed-deploy email (§B F38-e); the feature-owned alarms are built only where UAT (F38-h) or production (F38-i) decides.
- AI: one call per pinned model from the task role; each AI feature on
mainexercised against real Bedrock for quality, latency and cost (§C-5, the AI row in the §B backlog table). - Make the browser suites runnable against dev: a configurable Playwright base URL (it
was fixed to
localhost:5173withpnpm devuntil F38-g FR-1 addedET_E2E_TARGET, which builds the smoke configuration againsthttps://dev.crm.empire.lk), a real-account sign-in in place of the stub (§B F5-a, thestorageStaterow), and a real-path alternative or a recorded manual check for every step that calls a stub-only endpoint (stub login, stub connect, stub notify, stub grant). Update 2026-10-22 (F38-g WI-2.1): the smoke tier is set up on dev and recorded green (§B FS-4); the full tier is F38-h’s. Update 2026-10-26: the full tier runs only insidedeploy-prod, if F38-i settles how it signs in (design/architecture.mdRev 72); UAT gets the smoke run by hand. - The end-to-end retest (§C-7): list every feature with code on
mainfrom the tracker that day, and retest each end to end on dev with a real Entra sign-in. Since F38 Rev 21 (2026-10-19): by a per-feature checklist walked by hand, not the browser suites; step 24 then applies to the smoke suite only (F38-g spec FR-1, FR-2, FR-4). - Wire the standing smoke and full E2E CI tiers against dev (§B FS-4). Since F38 Rev 21: the
smoke tier is run by hand after each
deploy-dev, with no CI job (F38-g); the full tier is F38-h’s. Update 2026-10-22 (F38-g WI-2.1): the smoke tier is set up on dev and recorded green (§B FS-4); the full tier is F38-h’s. Update 2026-10-26: the full tier runs only insidedeploy-prod, if F38-i settles how it signs in (design/architecture.mdRev 72); UAT gets the smoke run by hand.
The first UAT round (the end of F38)
- Apply the
uatinstance for the round withforge-et-iaac’suat.yamlworkflow, started by hand ondev, and destroy it by hand after Phase 2’s round (one instance serves both rounds, AD #23 revised 2026-11-18), when the lead decides it is no longer needed — never on a schedule (overlays/uat; F38-h FR-6, FR-7;.claude/rules/tracker.md § Release & UAT Lifecycle). Each round comes up in three steps (F38-h spec Rev 7):uat.yamlapply withserviceunticked, thenforge-et-be’sdeploy-uatregisters the task definition from.aws/td.uat.jsonand stops, thenuat.yamlapply withserviceticked, which creates the backend’s ECS service. UAT reads dev’s Entra secrets, Exa key, SES identity andcrm/backend, so nothing is copied per round (F38-h decisions 2–4 and 10;environments-and-identity.md§2.3); its DNS records go through the DevOps team’s role for UAT’s names (role/dns-uatin the management account, working since 13 Nov 2026);deploy-uatinforge-et-beandforge-et-fedeploys a release tag to it (F38-h wave 4;architecture.md§ CI/CD Pipelines). Open until F38-h’s rehearsal (WI-5.1 – WI-5.3) has applied, deployed to and destroyed it: the threebase/options for a destroyable instance (forge-et-iaac#11) and the overlay are proved on a real instance only there (F38-h FR-6, FR-12; AC-8, AC-9, and AC-5’s rollback case). No UAT alarms until the alarm set is decided (§B FS-5) (§C-3). Update 2026-11-13: forge-et-iaac#11 merged (b175dae), and Senuthmi Thimansa ran the first step the same day (uat.yamlapply withserviceunticked, run 43378306661, 75 resources);deploy-uatandtd.uat.json(F38-h wave 4) do not exist yet, so steps 2 and 3 wait for them; the lead keeps the instance up, and the rehearsal’s destroy and re-apply still run before the round (F38-h spec Rev 8;environments-and-identity.md§4, “Theuatinstance”). Update 2026-11-19: no rehearsal destroy or re-apply (F38-h spec Rev 13, the lead’s decision): the instance is the round’s and is destroyed by hand when no longer needed, so AC-9 and FR-7’s list of what a destroy leaves are not proved before then.
Production and go-live (after Phase 1 — outside F38’s done)
- Production:
Empire’s answers on multi-AZ, cross-region copy and WAF(none is built, F38-i decision 13); move the prod private key from the vault into Secrets Manager and delete every other copy; put the confirmed production administrator’s email into production’s task definition as the plain valueET_SECURITY_SEED_ADMIN_EMAILbefore the first production deploy (AD #40) — without it the backend refuses to start, naminget.security.seed-admin-email(§B F5-k); confirm the prod mailbox reply URL; the restore rehearsal; the security review; the removal of the laptop reply URLs (and, if chosen, replacing the non-prod pair that laptops held); monthly reporting set up (§C-8, §B FS-6). - Close A-5, update this register’s
## Log, and reconcile.forge/active-context.md.
A. Read this first — what blocks a cloud environment outright
These things stop a first deploy from working at all, or make it dangerous. Do these before walking the rest. A-5 comes first in time: nothing else here can start until it is resolved.
| # | Blocker | Why | Source | What to do | Status |
|---|---|---|---|---|---|
| A-1 | The backend refuses to start in any cloud profile. SeamGuard (@Profile("!local & !test")) requires a real FileScanProvider and a real RecordVisibilityCheck. The scanner half is on main (F38-c wave 1): GuardDutyFileScanProvider is the FileScanProvider in every cloud shape, and it makes no AWS call — the GuardDuty protection plan scans each new object and the verdict arrives on the scan-result queue. The visibility half is on main too (F5-f, done 2026-10-04): RoutingRecordVisibilityCheck is the one RecordVisibilityCheck in every profile, answering account and lead attachments through the one visibility rule; the local/test stand-in (PermissiveVisibilityStub) is deleted, and SeamGuardIT proves a fully configured cloud-shape context now starts past the guard. What remains is the proof on dev. A cloud boot with scanning on also needs ET_MESSAGING_SCAN_RESULT_QUEUE (§D; ScanResultQueueGuard, which skips the check when ET_FILES_SCAN_PROVIDER=none, as every environment runs while GuardDuty is off) |
Both halves built (scanner: F38-c wave 1; visibility: F5-f wave 1); awaiting the dev boot (F38-f) | forge-et-be platform/scan/SeamGuard.java, platform/scan/GuardDutyFileScanProvider.java, attachment/service/RoutingRecordVisibilityCheck.java; .forge/plans/foundation/011-file-pipeline-plan.md Decision 13 + Notes (“operational corollary”); 011-file-pipeline-findings.md F-004; .forge/specs/F38-c-guardduty-scan-adapter-spec.md FR-1; .forge/specs/F5-f-team-visibility-scope-spec.md FR-C1 (§B F5-f) |
Once F5-f wave 1 is on main, confirm on dev that a cloud-profile boot passes the guard before testing anything else (§B F5-f). The GuardDuty path itself (protection plan → EventBridge → scan-result queue → verdict) is proved on dev by F38-f (§B FS-11). Update 2026-10-09: GuardDuty is not enabled, so the cloud scanner is F38-j’s none provider; dev booted past the guard that way on 2026-10-09 (liveness UP), and F38-f records it with the real visibility check. The GuardDuty path is not proved unless GuardDuty is added (§B FS-11) |
open |
| A-2 | No Terraform exists anywhere. (Update 2026-09-29: the Terraform’s home is altrium-io/forge-et-iaac, created empty that day — AD #23 revised; Shyamin Ayesh is writing it. This row closes when the dev estate’s Terraform is merged there.) (Update 2026-09-30: the dev VPC and ECS cluster are applied from it, in us-west-2; the rest of the estate is not written yet — environments-and-identity.md §4.) (Update 2026-10-09: the backend’s path is applied, in us-east-1 — RDS, ECR, ECS service and roles, load balancer, certificate, scheduler queue; S3, KMS, the other queues, EventBridge Scheduler, CloudFront and the AgentCore Gateway are not — environments-and-identity.md §4.) (Update 2026-10-21: the CRM’s CloudFront and the AgentCore Gateway were applied on 13 Oct, and the delivery team’s forge-et-iaac#3 — the KMS keys, the attachments bucket, the ingest and lead-triage queues, EventBridge Scheduler and the task-role grants — was merged into dev as d230b133 and applied by dev.yaml run 39475850266 on 20 Oct. Left for dev: the capture app’s hosting. The row stays open until F38-b’s wave 4 confirms no drift between the Terraform and the account — environments-and-identity.md §4.) (Update 2026-10-21, F38-b WI-2.1: the merge run 39475850266 and what it applied are recorded with evidence in environments-and-identity.md §4 “F38-b — the dev estate, checked”; the row closes with WI-4.1’s no-drift check.) (Update 2026-10-26, F38-b WI-4.1: the no-drift check’s sweep is met — nothing in the dev account outside Terraform beyond what F38-b FR-1 allows — and no resource can write a secret’s value into the state; still open: a dev.yaml run on dev whose plan prints No changes. after c728e06b, which no run has shown yet (.forge/owed.md, “Record F38-b AC-1’s no-change dev.yaml run”); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-1.) (Update 2026-10-26, later: dev.yaml run 40278225094 on dev at c728e06b, started at the lead’s request, planned No changes. and applied nothing, so dev’s estate is in Terraform with no drift (F38-b AC-1, ticked) and this row closes; production’s estate is F38-i’s and uat’s F38-h’s, in §C-3.) forge-et-be/terraform/ is absent; no .tf file in any repo. Everything in §C-3 is design only |
AWS account not yet provisioned | AD #23; architecture.md § Foundation Backlog (FS-1 cloud half); code sweep 2026-09-23 |
Author the FS-1 cloud-half spec and the Terraform modules; §D lists every value the task definitions must carry | closed 2026-10-26 — dev in Terraform with no drift (F38-b AC-1) |
| A-3 | Do not deploy F20-b Wave 1 on its own. Wave 1 ships ingestion with no subscription renewal; the plan calls the gap harmless only because “no environment is deployed”. On a real environment, mail sync stops silently after Graph’s 7-day maximum | F20-b Wave 2 not yet merged | .forge/plans/F20-b/F20-b-decomposition-plan.md § Known gap between the waves |
Deploy F20-b only once Wave 2 (renewal + delta poll) is on main, or accept the 7-day window knowingly and record it here |
closed 2026-09-25 — Wave 2 merged (forge-et-be#57, forge-et-fe#55), so main never carries Wave 1 without renewal |
| A-4 | The cross-host cookie decision — decided 2026-09-24 as AD #33: the backend keeps its api.* address and the apps call it cross-site. The problem: the CSRF cookie is host-only on api.*, so an SPA on crm.* cannot read it to echo X-XSRF-TOKEN, and every write would fail. The fix: give the CSRF cookie Domain= the environment’s CRM host, with a cookie name that differs per environment (crm.empire.lk is the parent of the dev and UAT hosts). The code change is built (F5-j, 2026-09-27 — forge-et-be#78, forge-et-fe#59); only the per-environment values remain |
No deployed hosts to test against | forge-et-be/docs/auth/README.md § “The cross-site CSRF cookie (AD #33)” (F5-j); FS-6 plan Notes (Pre-PR Important) |
Built by F5-j (.forge/specs/F5-j-cross-site-csrf-cookie-spec.md). Backend: SecurityConfig.csrfTokenRepository reads et.security.csrf.cookie-domain / cookie-name (ET_SECURITY_CSRF_COOKIE_DOMAIN / _NAME; unset → host-only XSRF-TOKEN, as before) and refuses to start on a bad pair. Frontend: forge-et-fe src/lib/api/apiClient.ts reads the name from VITE_CSRF_COOKIE_NAME, and vite build refuses a cloud build without it; the PWA’s client reads the same variable when F36-a builds it. The §E comments are corrected. What remains is F38’s: per environment set the §D values: VITE_API_BASE_URL = the api.* address, the CORS list = that environment’s CRM and capture origins, the cookie domain and name, and login-redirect-uri = the CRM origin. Neither the change nor the values affect F20-b’s in-flight branches (checked 2026-09-24) |
open |
| A-5 | Update 2026-09-29: the account exists — Empire Teas’ own, held by Kavindu Anthony and billed to Empire Teas by AWS; SOW ET-002 is superseded. Shyamin Ayesh has administrator access and is building the environments and the developers’ logins; this row stays open until dev is usable, and the rest of it describes what waits until then. Update 2026-10-09: the backend is deployed to dev and answers at https://api.dev.crm.empire.lk (liveness UP), but nobody can sign in yet (the Entra values are placeholders), and uploads, the mailbox and both frontends are not set up; dev counts as usable once a person signs in as CRM01 on dev.crm.empire.lk (“Start here” step 20). Bedrock is usable for Haiku 4.5 only (§C-5). Update 2026-10-13: met — CRM01 signed in on https://dev.crm.empire.lk (Shyamin Ayesh, then the lead), so this row closes. Uploads, the mailbox, the capture app and the 5.5 models are still owed, each in its own row (§C-3, §C-5, “Start here”). Update 2026-10-21: the resources uploads and the mailbox need — the attachments bucket, the storage and mailbox token keys, the ingest queue — exist since forge-et-iaac#3 was applied on 20 Oct; both still wait on their values in the task definition (“Start here” step 16). As written 2026-09-23: The AWS accounts do not exist yet. As of 2026-09-23 they wait on a billing discussion between Empire Teas and Altrium management (AWS is Altrium-run under SOW ET-002). Until they exist, no row in this file can be validated, the FS-1 and FS-5 cloud halves cannot start, and nothing built so far has run end to end (§C-7). There is no Bedrock without them, so every AI feature — F12 builds the AIProvider seam; Phase 1 also has F11-c, F11-d, F11-g, F11-h and F13 — can be built only against the local stub, with its real quality, latency and cost unproved (§C-5). SP-ARCH-004 stage 2 (Bedrock AgentCore Web Search versus Tavily) waits on the accounts too; its recorded fallback is that F11-c proceeds on the best available provider and AgentCore becomes a later swap (tracker.yaml setup.architecture.spikes SP-ARCH-004 settle_by). It also puts the dated plan at risk: the Phase 1 client UAT (to Thu 10 Dec since the revised Statement of Work, .forge/client/project-timeline.md) runs in a temporary UAT environment, and development never doubles as UAT |
Commercial discussion outside the delivery team | .forge/design/environments-and-identity.md §3 and §5; SOW ET-002 (.forge/client/aws-sow.md); team note 2026-09-23 |
Track the billing decision (Tharinda, with Altrium management). When the accounts land, follow the “Start here” checklist above; the cloud work is tracked as F38. If dev is not usable in time for the Foundation milestone (Fri 16 Oct), raise the dates with Empire as a stated risk, not a silent slip — the Statement of Work moves them day for day after the environment is ready | closed 2026-10-13 — CRM01 signed in on dev |
B. By owner — every feature and foundation slice
One block per slice or feature that has shipped, is in flight, or has a spec that already names a
cloud dependency. A block is either a table of rows or one line saying nothing is cloud-gated and
what was checked. New features append a block here; keep the order FS-* then F* as in
features.md.
Nothing cloud-gated, or carried in another block (recorded, not omitted)
| Slice / feature | What was checked | Recorded |
|---|---|---|
| FS-3 Design primitives | Spec says it runs fully local with no AWS dependency; findings and plan have no cloud items. Its one deploy-related note (service-worker cache key) is carried under FS-1 | 2026-09-23 |
| FS-10 API conventions | Spec NFR-2 and Dependencies: no AWS dependency, runs offline. One thing to watch, not a gap: the OpenAPI document builds its server URL from the incoming request, so behind the ALB it depends on FS-6’s forwarded-headers row | 2026-09-23 |
| FS-12 / FS-13 | Cancelled before start (AD #29). Nothing to carry | 2026-09-23 |
| FS-15 Retire product catalog | Only notes that nothing is deployed, so no data migration is needed. Plan and findings have no cloud items | 2026-09-23 |
| F1-a Account and contact substrate | Spec and all six WI plans checked. FR-S9’s migration is written so an empty cloud database gets no placeholder account, and CI proves the empty-database case | 2026-09-23 |
| F1-b Account list and create/edit | Spec checked (no plans yet). Its only stub dependency is the seeded stub-login users inherited from F5-a/F5-b | 2026-09-23 |
| F1-b Account list and create/edit — as shipped | Reflect (2026-10-07) confirmed: nothing cloud-gated. Wave 1 (forge-et-fe#171) is frontend-only against F1-a’s account and contact API and the user directory on the local backend; its unit tests mock the API and its browser suite runs against a local forge-et-be with the stub login, the sign-in path F5-a’s §B rows already cover. The new optional ET_E2E_WORKERS is a local test setting with no cloud counterpart |
2026-10-07 |
| FS-8 Developer onramp | Its one cloud item — the cloud onboarding docs — is carried as a row under §B FS-1 | 2026-09-23 |
| FS-14 Single module | Its cloud items are carried elsewhere: the RDS search_path row under §B FS-2, and the capture-PWA same-origin check under §B FS-1 |
2026-09-23 |
| F5-i Laptop sign-in | Checked against the spec (FR-11) and the Wave 1 plans: laptop mode is a local profile and a dev-server flag, deploys nothing and adds no cloud-gated item of its own. Its items are the §C-1 laptop row, the §B FS-6 go-live removal row, and the Part B evidence owed by F5-i-WI-2.1 against §B F5-a, §B F20-a and the §B FS-6 certificate-login row, marked “laptop, not dev” | 2026-09-27 |
| F38 Cloud bring-up | This register is F38’s work list; F38 has no §B block — the engagement-level rows it carries out are in §A, §C-3, §C-5, §C-7 (the retest), §C-8, §D (the task-definition values) and §E (stale comments) | 2026-09-24 |
| F39 Audit trail substrate | Spec confirmed (2026-09-26): nothing cloud-gated. The audit_event table, the one audit service writing in the caller’s transaction, the security events, the SYSTEM actor, the timestamp columns and both write guards (the runtime commit check and AuditWriteGuardSpec) all run and are proved on the local Postgres, with no AWS dependency (spec FR-14). The sign-in events on the real Microsoft callback are proved with the §B F5-a round-trip row, and a scan result’s attachment.scan_completed event on the real transport with the §B FS-11 scan-result row — not separately. The runtime check’s cloud behaviour is one ERROR log line (Unaudited Write Detected), which goes through the same logging FS-5’s cloud half wires; no new row |
2026-09-26 |
| F11-a Lead substrate and intake contract | Spec FR-D5 and both work-item plans checked: nothing cloud-gated. The lead and lead_tag tables, note.lead_id, the local seed leads, the intake, the single-lead API and its permission checks all run and are proved on the local Postgres, with no AWS, Entra, Bedrock or SES call. The audit events it writes use F39’s service, whose cloud behaviour the F39 row above covers |
2026-09-26 |
| F40 Accessibility and responsive checks as a PR gate | The spec’s Constraints (“Cloud-gated items: none”) and the Decomposition Plan checked: nothing cloud-gated. The local gate pnpm e2e:gate, the description check at the end of the Fast lane’s full run (node scripts/browser-gate.mjs ci-check), the harness hook and the skill changes all run on laptops and in the existing CI, with the stub login; no CI job depends on a deployed environment. AD-O-11’s post-deploy smoke and tag tiers are unchanged and stay with FS-1’s cloud half |
2026-09-29 |
F5-l Reach from the reporting line (grants without scope, all keys) |
Spec FR-22 and the WI-1.1 plan checked: nothing cloud-gated. The catalog’s account.all / lead.all, the V31 migration (its all keys, its NOTICEs, the dropped role_grant.scope), the grant rules, the reach rule in the checker and the account, contact and lead read paths, F4-a’s checks, /me without a scope and the laptop seed all run and are proved on the local Postgres (Testcontainers), with no AWS, Entra, Bedrock or SES call. When dev first migrates, V31 runs on dev’s grants like any migration and reports its reach changes in the Flyway log; F38’s end-to-end retest of main covers it, with no row of its own |
2026-09-30 |
| F5-d Roles editor | Spec FR-19 (“Nothing cloud-gated of its own”) and the seven WI plans checked at Reflect: the permissions catalog read, the roles list, create, save and delete endpoints, the editor screens and the full browser suite all run and are proved on the local Postgres with the stub login, with no AWS, Entra, Bedrock or SES call. The F5-k row on losing access across sites (AD #33) uses this editor on dev to remove a key and stays F5-k’s | 2026-10-05 |
| F11-f Lead detail and archive/nurture | Spec FR-H3 (“Cloud-gated items: none”) and the six Wave-1 plans checked: nothing cloud-gated. Archive and restore, the lead notes and tags routes and the lead tag create, the matched lead’s account timeline row with its backfill migration (V36__lead_timeline_backfill.sql), the local seed fixtures and the extended lead view all run and are proved on the local Postgres (Testcontainers) and the local stack with the stub login, with no AWS, Entra, Bedrock or SES call. The audit events they write use F39’s service, whose cloud behaviour the F39 row above covers. When dev first migrates, V36 backfills the timeline row of any lead already matched there, like any migration; F38’s end-to-end retest of main covers it, with no row of its own |
2026-10-06 |
| F41 Dependency bumps that need no gate run | The spec’s Constraints (“Cloud-gated items: none”) and the plan checked: nothing cloud-gated. The dependency-only rule runs in scripts/browser-gate/ on laptops, in the harness hook and in the existing Fast lane description check (contents-API reads only); the check:boundaries rule and the Dependabot groups need no deployed environment |
2026-09-30 |
| F26-c Weekly workbook upload as draft prices | Spec FR-23 and D-3 checked against the F26-c-WI-1.1 backend: nothing cloud-gated. The workbook is read in memory through platform.spreadsheet and discarded — never stored, scanned or served back, so neither S3 nor GuardDuty is involved. The auction-sale endpoints, the check, the create, the V36__price_import.sql migration (lkr_per_usd, price_one_draft_per_sale, workbook_tab_mapping; renumbered from V37, and it moves to V37 again if F11-f’s V36 reaches main first), the local seed (sale 2026 / 42, standards 90101 – 90191) and the lkrPerUsd member on the price reads all run and are proved on the local Postgres (Testcontainers) with the stub login, with no AWS, Entra, Bedrock or SES call. Its audit events use F39’s service, whose cloud behaviour the F39 row above covers. The NFR-1 timings (spec AC-27) are taken on the local stack by the spec’s own rule, not on dev |
2026-10-07 |
| F5-e Effective-permissions viewer | The spec (“Nothing cloud-gated”) and the four WI plans checked at Reflect: nothing cloud-gated. The read GET /api/v1/users/{id}/effective-permissions, the effective-permissions section on the user editor and the full browser suite all run and are proved on the local Postgres with the stub login, with no AWS, Entra, Bedrock or SES call. The section signs nobody in and writes nothing, so the F5-k cross-site rows (AD #33) cover its behaviour in a deployed environment |
2026-10-07 |
| F1-d Account detail: profile grid and contacts | The spec (FR-H3, “Cloud-gated items: none”) and the five WI plans checked: nothing cloud-gated. The record fields (lastContactAt, dormant, owner.displayName), the contact primary-flag rule, the account page, the contacts panel and the standards mount all run and are proved on the local stack with the stub login, with no AWS, Entra, Bedrock or SES call. The audit events use F39’s service, which the F39 row above covers |
2026-10-09 |
| F5-h Offboarding | Spec FR-C1 and the Decomposition Plan checked: nothing cloud-gated. The bulk reassign is synchronous, with no queue (lead decision, 2026-10-07); the bulk reassign, offboarding summary and reports-reassign routes, the offboarding screen and its browser suites all run and are proved on the local Postgres with the stub login, with no AWS, Entra, Bedrock or SES call. Deactivation’s session end and mailbox release are F5-c’s and stay under §B F5-c | 2026-10-07 |
| F5-g Record sharing screens on accounts and leads | Spec FR-G1 (“Nothing is cloud-gated”) and the four WI plans checked at Reflect: frontend only, against the sharing endpoints F5-f put on main. The sharing panel, the Share and Remove dialogs, the list marker and the Shared with me filter, and the browser suites all run and are proved on the local Postgres with the stub login, with no AWS, Entra, Bedrock or SES call. The standing E2E storageState row for cloud origins covers its screens as it covers every screen |
2026-10-11 |
| F1-e Notes and tags on accounts and contacts | Spec FR-H3 (“Nothing in this feature is cloud-gated”) and the Wave 1 plans checked: nothing cloud-gated. The account and contact notes routes and the note migration adding deleted_at / deleted_by, the account note’s timeline row, the account and contact tag routes, the account and contact tag create, the tag configuration (GET / PATCH /api/v1/tags/{id}) and the tag list’s active filter, the account list’s tagId filter, the local seed fixtures, the account page’s panels, the contact Modal and Settings › Tags all run and are proved on the local Postgres (Testcontainers) and the local stack with the stub login, with no AWS, Entra, Bedrock or SES call. The audit events use F39’s service, which the F39 row above covers |
2026-10-13 |
| F1-f Contacts page, including contacts with no account | The spec (FR-H3, “Cloud-gated items: none”) and the five WI plans checked: nothing cloud-gated. The contact list’s accountName and hasAccount, the contacts page, the contact view and its places, edit, archive, restore and attach all run and are proved on the local stack with the stub login, with no AWS, Entra, Bedrock or SES call. The attach is F1-a’s audited PATCH, which the F39 row above covers |
2026-10-13 |
| F1-c Communication timeline and dormancy | Checked at Reflect against the spec (FR-H4, “Nothing in F1-c is cloud-gated”) and the four WI plans: nothing cloud-gated. The timeline read GET /api/v1/accounts/{id}/timeline, the local seed’s timeline rows and Kowloon Tea House, the dormancy rule and its setting (et.account.dormancy-threshold, the same in every environment), the account page’s timeline panel and its refresh after a note change, and the seam and browser suites all run and are proved on the local Postgres (Testcontainers) and the isolated local stack with the stub login, with no AWS, Entra, Bedrock or SES call. The read writes no audit event (AD #36; spec AC-7) |
2026-10-21 |
| F20-d Email templates and merge fields | Checked at Reflect against the spec (“Cloud-gated: nothing”) and the four WI plans: nothing cloud-gated. Everything runs and is proved on the local stack with the stub login, with no AWS, Entra, Bedrock or SES call: the template API, the fill-in, the V46/V47 migrations with the starter set, WeeklyPricesTemplateService, the templates screen, the editor and Insert template, and the seam and browser suites. An email sent from a template goes through F20-c’s send path, which §B F20-c covers; the audit events use F39’s service, which the F39 row covers |
2026-10-16 |
| F26-d Send this week’s prices | Checked at Reflect against the spec (“Cloud-gated: nothing of its own”) and the four WI plans: nothing cloud-gated of its own. Everything runs and is proved on the local stack, with the stub login and the local mailbox stub, and with no AWS, Entra, Bedrock or SES call. That covers the preparation, fill and record endpoints, the V48 migration, the account-header action, the preparation dialog and the marker, and the seam and browser suites. The send itself is F20-c’s, so a real price email on dev is proved when §B F20-c is. The audit event uses F39’s service |
2026-10-17 |
| F20-l Several email addresses per contact | Spec FR-17 (“Cloud-gated items: none”) and the Decomposition Plan checked: nothing cloud-gated. The contact_email table and indexes, otherEmails on the contact API with its validation and audit, the search, the address lookup and F11-d’s match, the contact form and page, and the migration’s mapping, comparison and write run all run and are proved on the local stack with the stub login and export fixtures, with no AWS, Entra, Bedrock or SES call. Matching mail on the addresses is F20-f’s and F20-j’s, which their own blocks cover |
2026-10-28 |
| F38-b The dev estate in Terraform | Checked at Reflect: cloud work by construction, so no §B block of its own (spec FR-10). Every item it could not finish is open, with its reason, in the row that owns it: §B FS-11’s task-role S3 row, F20-a’s KMS-key row, F20-b’s ingest-queue row and F12 row 1 wait on F38-f’s real-call or end-to-end proof; §C-3’s estate, rotation and DNS rows wait on uat (F38-h) and production (F38-i). The closing sweep that read them is environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-10 |
2026-10-26 |
| F3-a Follow-up tasks on leads and accounts | Spec FR-H3 (“Cloud-gated items: none”) and the eight WI plans (WI-1.1, WI-1.2, WI-1.3, WI-2.1, WI-2.2, WI-2.3, WI-3.1, WI-4.1) checked at the wave-4 close: nothing cloud-gated. The activity_type and activity tables, the read, create, edit, complete, reopen and delete routes, the assignee candidates, the local seed fixtures, the account timeline row a done account follow-up writes, the Activities page, the navigation badge, the record pages’ Add follow-up and Follow-ups panel, and the seam and browser suites all run and are proved on the local Postgres (Testcontainers) and the isolated local stack with the stub login, with no AWS, Entra, Bedrock or SES call. The buckets are computed at read time, so no scheduled job is added. The audit events use F39’s service, which the F39 row above covers |
2026-11-07 |
| F32-c Bulk reassign leads and accounts (the import hand-out) | Spec FR-11 (“nothing cloud-gated”) and the four WI plans (WI-1.1, WI-1.2, WI-2.1, WI-2.2) checked at Reflect: nothing cloud-gated. The Pipedrive-owner names routes and list filter, Change Owner on both lists, the Pipedrive Owner filter and the browser test data (a generated Pipedrive export run through the migration routes) all run on the local stack against F5-h’s bulk reassign and F4-b/F4-d’s migration routes; the audit events are F5-h’s through F39’s service, which the F39 row above covers | 2026-11-09 |
| F34-c Custom field definitions and forms on accounts, contacts and leads | Spec § Dependencies (“Cloud-gated items: none”) and the seven WI plans (WI-1.1 to WI-1.3, WI-2.1 to WI-2.4) checked at Reflect: nothing cloud-gated. The definitions table, the JSONB value columns, the Settings page, the form fields and the record pages’ Custom Fields card all run and are proved on the local Postgres and the local stack, with no AWS, Entra, Bedrock or SES call; the audit events use F39’s service, which the F39 row above covers | 2026-11-11 |
| F32-d List views and saved filters on accounts, leads and contacts | Spec NFR-7 and § Dependencies (“Cloud-gated items: none”) and the eleven WI plans (WI-1.1 to WI-1.3, WI-2.1 to WI-2.3, WI-3.1 to WI-3.4, WI-4.1) checked at Reflect: nothing cloud-gated. The saved_view and list_preference tables, their API, the views row, the Columns menu and the new filters all run and are proved on the local Postgres and the local stack, with no AWS, Entra, Bedrock or SES call; the audit events use F39’s service |
2026-11-13 |
| F36-d Existing-client check at scan | Spec FR-CG1 (“Nothing in this feature is cloud-gated”) and AC-28 checked at Reflect: nothing cloud-gated. The badge check, the lead page’s check, the To Review marker and the link on convert read and write the database only, with no AI, AWS service or Microsoft API, and are proved on the local stack; the capture app’s own round trip on dev is F36-a’s §B rows; the audit context uses F39’s service. Whether NFR-2’s one-second target needs a reading on dev is an owed.md decision, not a register row |
2026-11-19 |
| F45 Split the largest backend classes | Spec § Constraints (“Cloud-gated: no new row”) checked at Reflect: nothing cloud-gated of its own. The split changes no behaviour and is proved by the backend suite; FR-9’s check against real Graph ran in laptop mode (WI-2.1, ## Log 2026-11-21), which needs no cloud environment. The Graph calls laptop mode cannot make (subscription create, renew and delete, change notifications) stay owed on dev by the F20-a and F20-b rows |
2026-11-21 |
FS-1 — App shell (the cloud half is the Terraform estate; §C-3 has the estate rows)
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
CI’s AWS credentials — revised 2026-10-02 (F38 Rev 4): an IAM service user per workload account (a deploy user for the app repos, a Terraform user for forge-et-iaac) with its access key held as a GitHub Environment secret, replacing the GitHub OIDC provider and deploy roles first planned. Update 2026-10-14: in dev one user, svc-github (AdministratorAccess), serves forge-et-iaac, forge-et-be and forge-et-fe with one key (created 2026-10-13; FR-19’s 90-day rotation would have been due by 2027-01-11, but since 18 Oct, F38 Rev 18, the shared dev key is replaced only if it may have leaked — no due date), held in each repository’s dev environment secrets; the lead accepted this for dev (F38 Rev 9), and UAT and production follow FR-19 |
Needs the AWS account and IAM | .forge/specs/foundation/001-app-shell-spec.md § Out of Scope; 004-build-ci-spec.md Out-of-scope table (FS-1 cloud-half row); F38 FR-19 |
A deploy job shows aws sts get-caller-identity returning the deploy user; the key is an Environment secret, not a repository secret; each key’s creation date and rotation due date (at most 90 days) are recorded here (F38 AC-19). Update 2026-10-19 (F38-d WI-1.1): svc-github has one key, Active, created 2026-10-13T06:55:37Z, with no console password and AdministratorAccess only; the dev Environment secrets AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY were last updated 2026-10-13T06:55:51Z and 06:56:16Z in forge-et-be and 06:56:09Z and 06:56:25Z in forge-et-fe, with no repository or organisation secret of those names; no replacement due date in dev (F38 Rev 18); evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-1. The printed identity and forge-et-pwa are F38-d WI-4.2’s, WI-6.1’s and WI-7.1’s. Update 2026-10-23 (F38-d WI-4.2): the deploy jobs print arn:aws:iam::570417736903:user/svc-github — in forge-et-be runs 39226901426, 39228765669 and 39801944235 and in forge-et-fe runs 39226907639 and 39287294321; forge-et-pwa is F38-d WI-7.1’s; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-1. Update 2026-10-26 (F38-d WI-7.1): svc-github has two active keys — the one of 2026-10-13T06:55:37Z, still held by forge-et-be and forge-et-fe (their dev secrets unchanged), and a second created 2026-10-23T11:49:08Z, held by forge-et-pwa, whose dev secrets were set at 11:49:21Z and 11:49:40Z; a leak response looks at both; no repository or organisation secret of those names in forge-et-pwa; its deploy jobs print arn:aws:iam::570417736903:user/svc-github (runs 40224118047, 40228839882, 40279386333); no replacement due date in dev (F38 Rev 18); UAT and production follow FR-19, so the row stays open for them; evidence: §4, AC-1. |
Tharinda (cloud slice) | open |
deploy-dev / deploy-prod / terraform GitHub workflows — backend: build → Trivy → image → ECR → task-definition revision → ECS rolling update; frontends: build → S3 sync → CloudFront invalidation |
No environment to deploy to; today each repo has only pr-validation.yml (Fast + Slow lane) and uploads dist/ as an artifact. Update 2026-10-14: forge-et-be (since 9 Oct) and forge-et-fe (since 13 Oct) each have a build workflow that deploys every push to main to dev, written by the DevOps team; neither runs Trivy, neither has a concurrency group or timeout-minutes, and the frontend’s S3 sync is not staged (environments-and-identity.md §4 rows 9, 15). forge-et-pwa has none; no deploy-prod or Terraform-plan-on-PR workflow exists |
001-app-shell-spec.md § Out of Scope; architecture.md § CI/CD Pipelines; CLAUDE.md “Automated deployment — designed, not yet built” |
Author the workflows; a merge to main deploys dev automatically; a vX.Y.Z tag deploys UAT/prod. Update 2026-10-19 (F38-d WI-1.1): the four deploy files are reviewed (DR-1 – DR-35), each finding with its home. Correcting the 14 Oct note above (DR-35): forge-et-fe’s workflow has had a concurrency group that never cancels a run and the staged S3 sync since forge-et-fe#259 (14 Oct, BUG-136); forge-et-be’s has neither a concurrency group nor timeout-minutes, and neither runs Trivy. No deploy-prod workflow exists in any app repository; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, the review and AC-6. Update 2026-10-20 (F38-d spec Rev 3): the backend’s dev deploy runs no Trivy scan by decision (ECR scans each image on push) and does not wait for the rollout; Trivy and the wait belong to deploy-uat and deploy-prod. Update 2026-10-23 (F38-d WI-4.2): since forge-et-be#275 and forge-et-fe#298 (19 Oct) both app workflows meet FR-20’s rules as written for dev — separate build and deploy jobs with AWS credentials only in the deploy job, a concurrency group that never cancels, timeout-minutes, persist-credentials: false, paths-ignore for workflow-only pushes and manual dispatch — and the backend’s dev deploy runs no trivy image (forge-et-be#286); forge-et-pwa’s is F38-d WI-5.1’s, and no deploy-uat, deploy-prod or Terraform-plan-on-PR workflow exists; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-2, AC-3 and AC-12. Update 2026-10-26 (F38-d WI-7.1): forge-et-pwa has had a build workflow meeting the same rules since forge-et-pwa#91 (25 Oct), and every non-workflow merge to its main deploys the capture app to dev; no deploy-uat, deploy-prod or Terraform-plan-on-PR workflow exists (F38-h, F38-i); evidence: §4, AC-3, AC-5 and AC-6. |
Tharinda | open |
ET_ENVIRONMENT on every ECS task definition |
Required key with no default; the task will not boot without it | .forge/plans/foundation/001-app-shell-plan.md Risk R11 + Notes |
Terraform sets it per environment; the first dev boot succeeds | — | open |
ALB target group health check polls /actuator/health/liveness; the database check sits in readiness only |
Target-group settings exist only in Terraform; locally the probes are tested in-process (ActuatorSpec) |
001-app-shell-plan.md Decisions 17, 17a; 002-data-layer-spec.md FR-D4 |
Confirm the target group uses the liveness path and that an RDS blip does not restart tasks | — | open |
CloudFront “404 → index.html” SPA rule must exclude the API path |
Only exists on the real distribution | 001-app-shell-findings.md F-005 |
curl an /api/... path through CloudFront; it must not return the app-shell HTML |
— | open |
The CRM distribution’s security headers (HSTS, X-Content-Type-Options, X-Frame-Options, a referrer policy; no CSP) reach every page, including the 403/404 → index.html responses, and nosniff blocks no script or stylesheet |
AWS’s documentation does not say whether a custom error response carries the response-headers policy. Update 2026-10-21: the policy exists since forge-et-iaac#3 was applied on 20 Oct, and curl -sI on https://dev.crm.empire.lk/ and on /leads (x-cache: Error from cloudfront, so the 403/404 → index.html response) each shows the four headers and no CSP; the browser load is still to do (F38-b wave 2). |
F38 FR-14 and AC-14 (Rev 20); F38-b spec FR-7, AC-7, Rev 7 | curl -sI https://dev.crm.empire.lk/ and a deep path (for example /leads) each show the four headers; the CRM loads in a browser with no blocked script or stylesheet in the console Update 2026-10-21 (F38-b WI-2.1): the policy et-dev-spa-security-headers is on the default cache behaviour, the root and the 403 → /index.html response for /leads carry the four headers and no CSP, and all ten scripts and stylesheets index.html loads answer with text/javascript or text/css; the browser load is still to do (plan D-10); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-7. Update 2026-10-22: the browser load is F38-b WI-4.1’s, made by the session that runs it (the lead’s decision; WI-4.1 plan ## Notes). Update 2026-10-26 (F38-b WI-4.1): the browser load is done — https://dev.crm.empire.lk/ rendered its sign-in screen with every script and stylesheet at 200 and no refusal under nosniff in the console (the one console error is the signed-out /api/v1/me 401); the capture app’s distribution carries the same four headers; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-7. |
F38-b | validated |
| Capture PWA installable and its service worker correct on the real capture domain (S3 + CloudFront + ACM) | Proven only on pnpm preview at localhost:4173; install requires a real HTTPS origin; the worker’s same-origin check has only seen a fake http://localhost base |
001-app-shell-findings.md F-005 (AC-18/19); 014-single-module-findings.md F-003; forge-et-pwa/src/main.tsx 25–33, public/sw.js 9–27 |
In Chromium on the capture domain: the app installs; /api/ requests come from the network, never the cache |
— | open |
| Service-worker cache key changes once per delivery phase but dev deploys every merge, so a stale entry is never evicted (uncertain — verify) | Only visible with frequent real deploys | 003-design-primitives-plan.md “Remaining pwa Importants” |
Deploy twice to dev; confirm the old caches are cleared | — | open |
The built bundle’s version is readable in the browser on a deployed bundle (document.documentElement.dataset.appVersion) |
Verified on build output only | 001-app-shell-spec.md FR-F8 / AC-16; specs/F5-b-findings.md F-001 |
Read the version from the CloudFront-served bundle; it matches the release tag | — | open |
| Cloud onboarding docs — getting AWS access, deploying, reading CloudWatch | “None of it exists yet”. Update 2026-10-14: .forge/guides/aws-setup-explained.md covers the accounts, AWS access from the CLI and reaching the database; deploying and reading CloudWatch are still unwritten |
008-developer-onramp-spec.md § Out of Scope (FS-8) |
Write them after the FS-1/5/7/9 cloud halves land | — | open |
Static files without a content hash (the favicons, the touch icon, the capture app’s install icons, manifest.webmanifest, sw.js) are uploaded with max-age=0, must-revalidate, never immutable |
The frontend S3 sync step is not built, and the headers exist only on the real bucket and distribution. Update 2026-10-14: forge-et-fe’s sync step exists (13 Oct) as one aws s3 sync --delete with no cache headers, so no file carries Cache-Control on dev (environments-and-identity.md §4 row 15 (e)). Update 2026-10-19 (F38-d WI-5.1 plan, approved): the capture app’s half — its install icons, manifest.webmanifest and sw.js — is built by WI-5.1’s forge-et-pwa build.yaml and checked on dev by WI-7.1 (§B F38-d, “The capture app’s deploy on dev”) |
architecture.md § CI/CD Pipelines (staged S3 sync; Rev 49); forge-et-fe#89 and forge-et-pwa#39 review |
Author the sync step with that header for the non-hashed files; after a dev deploy, curl -sI each of /favicon.ico, /apple-touch-icon.png and /icons/icon-512.png through CloudFront and confirm the Cache-Control value. Update 2026-10-26 (F38-d WI-7.1): validated on dev for both halves — through https://capture.dev.crm.empire.lk/, sw.js, manifest.webmanifest, icons/icon-192.png, icons/icon-512.png, icons/icon-maskable-512.png, apple-touch-icon.png, favicon.ico and favicon-32.png each answer cache-control: max-age=0, must-revalidate; through https://dev.crm.empire.lk/, the CRM’s three non-hashed static files (favicon.ico, favicon-32.png, apple-touch-icon.png — the CRM has no install icons) answer the same; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-5. |
Tharinda (cloud slice) | validated |
FS-2 — Data layer
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
| RDS engine version pinned — decided PostgreSQL 18 (2026-10-02); local, the test suite and the pricing proofs moved to 18 in forge-et-be#146 | The pin lives in forge-et-iaac, which has no RDS resource yet |
002-data-layer-spec.md Constraints, Q-12, Rev 8; F38 FR-10 |
engine = "postgres", engine_version = "18", auto_minor_version_upgrade = true in forge-et-iaac; the dev instance reports 18.x Update 2026-10-26 (F38-b WI-4.1): the dev instance reports PostgreSQL 18.3 with AutoMinorVersionUpgrade true; base/rds/variables.tf pins default = "18" with no override in overlays/dev (c728e06b); no forge-et-iaac file sets auto_minor_version_upgrade, so the AWS provider’s default true applies, which meets this row’s auto_minor_version_upgrade = true (WI-4.1 plan D-11); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-3. |
F38-b | validated |
The migration role can create the btree_gist extension (needs rds_superuser) |
Locally the Compose superuser has it | 002-data-layer-spec.md Constraints, Q-11; plan R6 |
First cloud migrate succeeds; if not, it fails loudly at startup Update 2026-10-26 (F38-b WI-4.1): dev’s first task, on 2026-10-09, migrated an empty schema to version “1 - extensions” (V1__extensions.sql, CREATE EXTENSION IF NOT EXISTS btree_gist) at 09:06:26 UTC and on through 37 migrations, then started (aws logs get-log-events on the stream F38-d recorded); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-3. |
F38-b | validated |
Database credentials from Secrets Manager; ET_DATASOURCE_* and ET_BUSINESS_TIMEZONE on task definitions |
No task definitions exist | 002-data-layer-spec.md FR-D5; plan Notes |
Dev boots with no credential in env files or repo | — | open |
UAT data entered by hand by the IT Admin after each uat apply (AD-O-8, answered by F38-h decision 5) |
Needs a uat instance; no seed or script may write to a cloud database |
002-data-layer-spec.md Q-8; test-strategy.md § Test Data; F38-h spec FR-8, decision 5; .forge/plans/F38-h/uat-data-checklist.md |
After each apply and first deploy, the IT Admin enters the checklist through the app; first entered in F38-h’s rehearsal (WI-5.2, spec AC-10). Revise the checklist for Phase 2’s round once deals, quotes and samples declare their keys (the Managing Director role then too). Update 2026-11-19 (F38-h WI-5.2): first entered in the rehearsal through the app, Parts 0–5 (record .forge/plans/F38-h/F38-h-WI-5.2-record.md R-4, R-6, R-9); the task log showed only Flyway’s migrations (R-11). Stays open for the release track: the instance is kept for the rounds (AD #23), so a round adds what its own checklist revision asks, and a fresh apply would need it all again. |
F38-h (the rehearsal); the release track (each round) | open |
RDS app role keeps the default search_path and has no schema of its own name |
Otherwise Flyway silently creates every table outside public |
014-single-module-plan.md Pre-PR review (last Nit, “Hand-off to the FS-1 cloud half”) |
After the first migrate, flyway_schema_history and all tables are in public |
— | open |
| A migration numbered below the dev database’s latest version fails only at the next dev deploy | CI databases always start empty | 014-single-module-plan.md Pre-PR Nit (docs/migrations.md); 006-auth-scaffolding-plan.md R12; 002-data-layer-findings.md F-001 |
Watch the first multi-branch merges on the long-lived dev database; rebase before opening a PR | — | open |
FS-4 — Build & CI
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The smoke tier after each deploy-dev — run by hand (F38 Rev 21) |
No deployed environment; the tiers belong to the deploy pipeline | 004-build-ci-spec.md FR-X3 / AC-19; .forge/test-strategy.md § When Tests Run; forge-et-fe/playwright.config.ts 3–6, 55–63; e2e/README.md |
Validated 2026-10-22 (F38-g-WI-2.1 R-3, R-4): pnpm e2e:smoke green on dev after a forge-et-fe deploy (run 39714930703) and a forge-et-be deploy (run 39727038992), as the reserved view-only account CRM05; R-4 passed on its third attempt (attempt 2 failed check 6 once — see the record); record .forge/plans/F38-g/F38-g-WI-2.1-record.md |
F38-g | validated |
| The full E2E tier on the UAT tag | No deployed environment; the tiers belong to the deploy pipeline | 004-build-ci-spec.md FR-X3 / AC-19; .forge/test-strategy.md § When Tests Run; forge-et-fe/playwright.config.ts 3–6, 55–63; e2e/README.md |
Wire the full tier on the UAT tag; green against a deployed environment. Update 2026-10-19 (F38 Rev 21, architecture.md Rev 68): the smoke tier is run by hand after each deploy-dev, with no CI job — F38-g’s AC-2 validates it; the full tier and its sign-in are F38-h’s |
F38-h | open |
| Which tests make up the smoke suite | Deliberately chosen only once deploy-dev exists |
004-build-ci-spec.md § Test-Strategy Ratifications; test-strategy.md § Critical Flows & the Smoke Set |
Choose per the selection rules (login first; read-mostly; self-cleaning; under 5 minutes) when wiring the tier. Update 2026-10-19: chosen by F38-g’s plan within its spec’s FR-2 (a view-only smoke account, the backend’s and the CRM’s versions, one silent-breakage path per pipeline, a deep link) Validated 2026-10-22 (F38-g-WI-2.1 R-3, R-4): the nine checks of WI-1.1’s plan, green on dev; record .forge/plans/F38-g/F38-g-WI-2.1-record.md |
F38-g | validated |
Playwright browser-test flakiness on a cold runner (mailbox suites are start-up and memory sensitive): pin workers, raise timeouts, or accept reruns |
Decision belongs to whoever owns the cloud CI tiers | .forge/plans/F20-a/F20-a-WI-1.3-test-verify.md “Not closed” |
Run the tier on a cold CI runner, decide, apply in playwright.config.ts. Update 2026-10-19: the smoke tier has no CI runner (F38 Rev 21); the row stays for the full tier, if F38-h runs it in CI |
F38-h | open |
TRIVY_GATE_EXIT_CODE: "0" stays in all three pr-validation.yml); deploy-uat and deploy-prod stop on an open HIGH/CRITICAL finding in the image or the tagged lockfiles that has no .trivyignore.yaml exception |
The check lives in workflows that need the uat and production environments |
AD #27, AD #38; F38 FR-35, AC-35, AC-39 (Rev 5); issued proposal §A.4 standing controls | A deploy-uat run for a tag with an open finding stops and names it; the same finding with an exception proceeds and is listed; the backend’s deploy-dev for an image with it deploys and reports it. The backend’s Tomcat and Jackson findings were fixed by overrides (forge-et-be PR #145), removed by forge-et-be PR #311 on the Spring Boot 4.1.2 upgrade; RC-1 clean-baseline attestation at Phase-1 handover |
F38-h / F38-i | open |
dependency-submission workflow so GitHub’s dependency graph sees Gradle packages (named a candidate for the cloud workflow set, not structurally cloud-gated) |
Parked with the deploy workflows | 004-build-ci-findings.md F-004 |
Add on push to main; Maven packages appear in the dependency graph |
— | open |
FS-5 — Observability stubs (the cloud half is CloudWatch / SNS / SES)
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
| CloudWatch log groups, retention, and the ECS log driver shipping stdout JSON to CloudWatch | No account; the app writes only to stdout | 005-observability-stubs-spec.md FR-L1, Out-of-Scope “substrate” row |
Search CloudWatch Logs for one request’s traceId. Read 2026-10-23 (F38-e spec Context): /ecs/dev/backend exists with 30-day retention and the task definition ships stdout to it through awslogs; the query is F38-e’s AC-2 Update 2026-10-23 (F38-e WI-1.2): every log group in the dev account, in every Region, keeps 30 days — the one group, /ecs/dev/backend, declared in forge-et-iaac with 30 days — and the committed and running task definitions ship stdout to it through awslogs; the traceId query is F38-e WI-3.3’s (AC-2); evidence: §B F38-e “F38-e evidence — AC-1”. Update 2026-11-21 (F38-e WI-3.3): a Logs Insights query on one request’s traceId in /ecs/dev/backend returned that request’s two lines; with AC-1, the row is met for dev; evidence: §B F38-e “F38-e evidence — AC-2”. |
F38-e | validated — 2026-11-21, F38-e WI-3.3 (dev) |
LOGGING_STRUCTURED_FORMAT_CONSOLE=ecs (plus per-environment LOGGING_LEVEL_*) on the task definitions |
The key is deliberately absent from application.yml; without it cloud logs are plain text |
005-observability-stubs-plan.md Decisions 3, 6; spec FR-L3 |
Cloud log lines are ECS JSON carrying traceId/spanId. Read 2026-10-23: forge-et-be .aws/td.dev.json sets LOGGING_STRUCTURED_FORMAT_CONSOLE=ecs; F38-e’s AC-1 and AC-2 confirm it on the lines Update 2026-10-23 (F38-e WI-1.2): LOGGING_STRUCTURED_FORMAT_CONSOLE=ecs is set in the committed .aws/td.dev.json (forge-et-be 2efd29cd) and in the task definition the service runs (backend:60); the lines themselves are checked by AC-2 (F38-e WI-3.3); evidence: §B F38-e “F38-e evidence — AC-1”. Update 2026-11-21 (F38-e WI-3.3): the lines are ECS JSON carrying traceId and spanId at the top level; evidence: §B F38-e “F38-e evidence — AC-2”. |
F38-e | validated — 2026-11-21, F38-e WI-3.3 (dev) |
| ECS log field names against CloudWatch Logs Insights conventions | Only judgeable on real Logs Insights | 005-observability-stubs-spec.md OQ-2; plan R5 |
Run Insights queries; switch to a custom formatter only if the field names get in the way Update 2026-11-21 (F38-e WI-3.3): traceId, spanId, log.level, log.logger and message each came back under its own name with no escaping, and the filter on traceId worked — workable, no custom formatter; evidence: §B F38-e “F38-e evidence — AC-2”. |
F38-e | validated — 2026-11-21, F38-e WI-3.3 (dev) |
| Where the frontend / PWA error reporter sends errors (e.g. CloudWatch RUM); both reporters are empty functions today | Decision open | 005-observability-stubs-spec.md OQ-1; forge-et-fe/src/app/telemetry.ts 6–16; forge-et-pwa/src/app/telemetry.ts 6–16 |
Decide, implement, update FS-1’s deferral note. Decided 2026-10-23 by the lead (F38 W-9, Rev 23; F38-e spec decision 3): none for V1 — both reportError functions stay empty and send nowhere. The two telemetry.ts comments and the two frontends’ CLAUDE.md lines that call the decision open are §E rows. F38-e closes this row (its AC-8). Update 2026-10-23 (F38-e WI-1.2): the code matches the decision — both reportError bodies on origin/main hold only their comment, and neither frontend depends on an error-reporting or RUM package; evidence: §B F38-e “F38-e evidence — AC-8”. |
lead (decided); F38-e (records) | validated — none in V1, decided 2026-10-23 (the lead); the code matches (F38-e WI-1.2, 2026-10-23) |
| CloudWatch alarms → SNS → delivery-team email — the alarm substrate itself | No CloudWatch/SNS locally | AD (infra); CLAUDE.md Active Context (“FS-5 CloudWatch/SNS/SES … deferred”) | Trip a test alarm; the SNS email arrives. Update 2026-10-23 (F38-e spec FR-4, FR-6): in dev the topic emails the four developers; one test alarm is tripped and then deleted, and the topic also carries the failed-deploy email (§B F38-e). Update 2026-10-25 (F38-e WI-2.1): the topic et-dev-alerts exists with a policy admitting only CloudWatch and EventBridge on this account; 4 of four subscriptions confirmed; the test alarm is F38-e WI-3.1’s; evidence: §B F38-e “F38-e evidence — AC-3”. Update 2026-11-21 (F38-e WI-3.1): the test alarm’s email was delivered by SNS to all four subscriptions (4 delivered, 0 failed) and the alarm is deleted; each recipient’s inbox report is owed (owed.md); evidence: §B F38-e “F38-e evidence — AC-5”. |
F38-e | validated — 2026-11-21, F38-e WI-3.1 (dev, SNS delivery; the four inbox reports owed; UAT and production decide their own alarms, F38-h, F38-i) |
| SES — sending identity, SPF/DKIM, system mail |
No SES locally | AD-O-3; PRD (F20 email~~, F36 capture notifications~~ — AD #42); FS-5 dependency row 5; .forge/client/dependencies.md §1.1 (SES row) |
Request production access as soon as the account exists; send a test system mail dev.crm.empire.lk is verified with 2048-bit Easy DKIM and the MAIL FROM domain bounce.dev.crm.empire.lk, DMARC is p=none with no report address, production access was granted on 2026-10-21 (case 179256302200929, transactional), the account suppresses bounces and complaints, and all of it is declared in forge-et-iaac at 0eee623; still open for the test send to a non-verified address (F38-e; who sends the first test message is .forge/owed.md, “Send the first test message through dev’s SES”) and for production (F38-i); evidence: environments-and-identity.md §4 “F38-a — the development account, checked”, AC-5. Update 2026-10-21 (later): the first test message went through dev’s SES — sent by the lead from noreply@dev.crm.empire.lk to the CRM01 mailbox (not verified in SES) at 08:41 UTC through a temporary policy on svc-harness, removed at 08:45; it reached the Inbox with spf=pass (MAIL FROM bounce.dev.crm.empire.lk), dkim=pass for dev.crm.empire.lk and dmarc=pass (same AC-5 subsection, “The first test message, delivered”). Still open: the backend sending through its task role (F38-e, F38 FR-26) and production (F38-i). Update 2026-10-23 (F38-e spec decision 5, FR-8): F38-e sends one system mail from a one-off task that runs under the backend’s task role. The backend’s own sender, and the capture lead notification, are F36-c’s (Phase 2), which proves both through it. Update 2026-11-12 (AD #42): there is no capture lead notification to send — the capture app’s notifications are in-app only — so this row’s capture half is void and F36-c builds no sender. Still open: the backend sending through its own sender, which now has no V1 consumer named; which feature builds it and proves F38 FR-26’s system mail is .forge/owed.md (“The backend’s SES sender has no consumer after AD #42”). Update 2026-11-21 (F38-e WI-3.2): a system mail sent from a one-off task under the backend’s task role et-dev-backend-task reached the CRM01 Inbox with spf, dkim (dev.crm.empire.lk) and dmarc pass; the probe and its permission are removed; evidence: §B F38-e “F38-e evidence — AC-7”. Still open: production (F38-i) and the backend’s own sender (the owed.md row above). |
F38-a (dev identity and production access); F38-e (delivery test); F38-i (production) | open |
| Trace sampling is set to 1.0 but no span exporter is configured, so nothing is exported (uncertain — verify the intended exporter, e.g. X-Ray) | No collector locally | forge-et-be/src/main/resources/application.yml 403–410 (was 210–217); build.gradle (the Micrometer Tracing dependencies) |
Decide the exporter; a request’s spans appear in it. Decided 2026-10-23 by the lead (F38 W-9, Rev 23; F38-e spec decision 3): no span exporter in V1 — a request is followed by its traceId in the CloudWatch log lines (the log-group row above). F38-e closes this row (its AC-8). Update 2026-10-23 (F38-e WI-1.2): the code matches the decision — forge-et-be’s gradle.lockfile at 2efd29cd holds no span exporter or meter registry (only FS-5’s Brave tracer and its AWS propagation); evidence: §B F38-e “F38-e evidence — AC-8”. |
lead (decided); F38-e (records) | validated — none in V1, decided 2026-10-23 (the lead); the code matches (F38-e WI-1.2, 2026-10-23) |
Feature-owned alarms that need this substrate — lapsed Graph subscription (F20-b), RECONNECT_REQUIRED gauge (F20-a), DLQ arrival + queue depth (FS-7), failed/missed scheduled job (FS-9), INFECTED/scan failure (FS-11), AI spend |
Each ships its signal with its feature; the alarm wiring waits for FS-5 | 005-observability-stubs-spec.md Out-of-Scope table; the owning rows below |
Wire each alarm as its owning row says; trip each once. Update 2026-10-23 (the lead; F38 Rev 23, F38-e spec decision 1): none of these is built in dev. UAT (F38-h) and production (F38-i) each decide their set from the .forge/owed.md row “Decide the UAT and production alarm sets”; the mailbox alarms, if built, use metric filters on the existing status-change log lines (F38-e decision 6). AI spend is an AWS Budgets alert belonging to the billing owners (F38 Rev 18). Update 2026-10-23 (F38-e WI-1.2): the owners read as F38-h and F38-i on this row and on each row it points at (the billing owners for §C-5’s AI-spend row), and dev has no metric filter in any Region; evidence: §B F38-e “F38-e evidence — AC-6”. Update 2026-10-26 (F38-h WI-3.1): F38-h’s uat overlay is built with no alarm (the lead, 2026-10-25; F38-h Decomposition Plan § Notes); UAT’s set comes as a later F38-h work item once the .forge/owed.md row “Decide the UAT and production alarm sets” is decided. The row stays open. |
F38-h (UAT), F38-i (production) | open |
FS-6 — Auth scaffolding (Entra, sessions, CORS, CSRF)
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
Certificate-based Entra login (client secrets are blocked by Empire’s tenant policy) — the signed assertion (private_key_jwt, PS256, x5t#S256) is never built by any test; the PEM is only parsed at first sign-in |
Needs the Entra registration + the certificate and private key in Secrets Manager | .forge/design/environments-and-identity.md; FS-6 FR-A2 Rev 6, plan R13, findings F-008; forge-et-be auth/config/EntraLoginConfig.java 216–250 |
First cloud sign-in succeeds; a cloud smoke login is owed before the Phase-1 handover. Also covers F20-a’s microsoft-mailbox registration, which reuses the same certificate |
Both certificates (prod, and non-prod for dev, UAT and laptops) generated by Shyamin Ayesh and uploaded by Kavindu Anthony on 2026-09-24 (thumbprints in environments-and-identity.md §2.3); what remains is the private keys in Secrets Manager and the live sign-in — F38 Laptop, not dev (2026-09-29, Nanndish, F5-i-WI-2.1 R-1, R-3): the certificate sign-in (private_key_jwt) worked against the real tenant for login and for the mailbox code exchange. Still owed on dev. |
open |
Entra app registration wiring — Client/Tenant IDs, reply URLs for /login/oauth2/code/microsoft per host, Graph scopes — via task-definition secrets; et.security.entra.* missing makes a cloud boot throw |
Part of the cloud identity setup | .forge/design/environments-and-identity.md; FS-6 Q-6; EntraLoginConfig.java 90–127 |
The app registration exists (since 10 Sep); set its values on the task (IDs as plain values, the certificate private key via Secrets Manager); the backend boots and the login redirect names the right tenant | Kavindu Anthony registers anything in Azure; Shyamin Ayesh sets the task-definition secrets on the AWS side | open |
Forwarded headers behind the ALB (server.forward-headers-strategy: framework) — the login redirect_uri and Secure cookie detection depend on them |
Tests connect directly; only a TLS-terminating proxy shows the failure (“Entra refuses every cloud sign-in with a redirect_uri mismatch” without it) | 006-auth-scaffolding-findings.md F-003; plan Notes Pre-PR “Blocker 2”; application.yml 119 |
The login redirect_uri is https://api.<env>/login/oauth2/code/microsoft (AD #33 — the backend keeps its own address), never the ALB’s own name or a private IP; the CSRF cookie (the environment’s name, e.g. XSRF-TOKEN-DEV — F5-j) carries Secure |
F38 | open |
Custom domains under empire.lk exist before any cloud login test; default CloudFront/ALB hostnames fail the SameSite cookie rule |
Deployment precondition | 006-auth-scaffolding-spec.md Constraints (registrable domain), Q-5 |
Log in over crm. → api.crm.; reopen Q-5 if either host leaves empire.lk |
— | open |
Cross-host CSRF cookie readability — under AD #33 the CSRF cookie is set with Domain= the environment’s CRM host and an environment-specific name, so the SPA on crm.<env> (and the PWA on capture.<env>) can read it. The code is built (F5-j, 2026-09-27): CrossSiteCsrfCookieIT proves the name, the Domain attribute, the single logout clear and the refusal of other names on the wire — not a browser sharing the cookie across hosts |
Locally same-origin via the proxy, so the shared-domain cookie is never exercised | F5-a WI-1.1 Risk; FS-6 plan Notes; forge-et-be/docs/auth/README.md § “The cross-site CSRF cookie (AD #33)”; AD #33; F5-j FR-12 (a) |
On dev: the cookie arrives with the dev domain and dev name; the SPA reads and echoes it and a write succeeds; with a prod cookie also present in the same browser, dev writes still succeed | F38 | open |
Real per-environment CORS allow-list (et.security.cors.allowed-origins) — local is empty by design because the Vite proxy strips Origin |
Production is cross-origin (AD #33): the list must name that environment’s CRM and capture origins | FS-6 FR-S5; F5-a WI-1.4 (CORS 403 caught + fixed dev-side); SecurityConfig.java 173–181; forge-et-fe/vite.config.ts 40–53 |
Set per environment to https://crm.<env> and https://capture.<env>; browser POSTs succeed. Access-Control-Max-Age so the preflight is not repeated on every write.Access-Control-Max-Age: 3600, a code constant, proved locally by SecurityChainIT. On dev, also confirm a real cross-site preflight from https://dev.crm.empire.lk carries Access-Control-Max-Age: 3600 and the browser does not repeat it within the hour (F5-j FR-12 ©) |
— | open |
Secure session and CSRF cookies over HTTPS — secure: true in base config, false only under local |
Local serves plain HTTP | FS-6 FR-S3; application.yml 131 vs 292–296; SecurityConfig.java 111–120 |
The SESSION cookie and the CSRF cookie (the environment’s name, e.g. XSRF-TOKEN-DEV — F5-j) carry Secure; login works over HTTPS |
— | open |
et.security.login-redirect-uri per environment (the SPA origin; also the base for the mailbox ?connect= redirects) |
No default | FS-6 Decision 21; EtProperties.Security; application-local.yml.example |
Completed and failed logins, and mailbox connects, land on the SPA | — | open |
| Fresh environment has no roles besides the seeded IT Admin (AD #34 — no superadmin; revised 2026-09-24, was “besides the superadmin”); the recommended day-one roles are created by hand at go-live | Only matters on a real empty database | 006-auth-scaffolding-plan.md R10 (folded into the R11 hand-off); AD #34 |
After signing in as the seeded admin (§B F5-k), create the roles from forge-et-be/docs/auth/recommended-roles.md (F5-k rewrites it on the AD #34 keys) |
Altrium supervising | open |
| A bad grant at startup rejects the deploy (old task keeps serving), not the environment | Relies on ECS rolling-deploy behaviour | 006-auth-scaffolding-spec.md Q-11 |
Deploy a build with a stale grant; the old task still serves; the deploy is marked failed. Update 2026-11-17 (F38-h WI-5.1): not produced on UAT. A stale grant needs a database write, which F38-h FR-8 forbids, so F38-h AC-5’s stale-grant half stays carried by this row. The same ECS behaviour was shown on UAT with a missing seeded-admin setting: the failed deploy was rolled back and the previous task kept serving (.forge/plans/F38-h/F38-h-WI-5.1-record.md R-8). |
— | open |
| Spring Session expired-session cleanup runs on every ECS task (the recorded AD #12 carve-out) | Needs more than one real task | 006-auth-scaffolding-spec.md FR-S4 |
With 2+ tasks, duplicate cleanups are harmless | — | open |
The stub sign-in is really absent from deployed builds and APIs (POST /api/v1/auth/stub-login → 401; the bundle has no Continue as this user) |
Proven locally by pnpm build grep and a cloud-profile boot test only |
F5-a spec FR-9 / AC-11; StubLoginController.java 32 (@Profile({"local","test"})); LoginRoute.tsx 136–139 |
On each deployed environment, confirm both | — | open |
Before production go-live, remove the laptop reply URLs (http://localhost:8080/login/oauth2/code/microsoft, http://localhost:8080/authorized/microsoft-mailbox) from the CRM app registration — they were registered there on 2026-09-24 for the laptop sign-in (§C-1). Laptops use the non-prod certificate, which stays because dev and UAT use it; consider replacing the non-prod pair at the same time, since its private key has been on developer laptops |
Go-live step on the shared app registration | .forge/design/environments-and-identity.md §2.2, §2.3, §2.5 |
Ask Empire IT to remove both reply URLs; confirm in the portal that no localhost reply URL remains; if the non-prod pair is replaced, confirm only the new non-prod certificate and the prod certificate remain |
Tharinda to ask; Kavindu Anthony (Empire IT) | open |
| Entra certificate expiry. An expired certificate stops every sign-in on its environment, and nothing in AWS can see an Entra certificate’s expiry date. The two (prod and non-prod) are created before go-live, so their clock starts early — and the non-prod one’s expiry stops dev, UAT and laptop sign-in at once | Both certificates exist and are uploaded (2026-09-24); both expire 23 Sep 2028, recorded in §5 | environments-and-identity.md §2.3 (validity and rotation), §5 (the certificate-expiry row) |
A calendar reminder a month before expiry (late Aug 2028); renew by overlap (upload the new public half beside the old, switch the secret, remove the old). The backend refuses an unusable pair on every signing — unreadable, mismatched, expired or not yet valid — by throwing EntraKeyMaterialException instead of sending it to Entra and retrying (forge-et-be#95, and the expiry check in forge-et-be#99). The mailbox renewal and ingest paths log it at WARN with reason=EntraKeyMaterialException; only a sign-in logs it at ERROR (Entra Sign-In Refused - Client Assertion Unsignable, forge-et-be#99), and then redirects to ?login=unavailable (§B F5-a). Owed on dev (F38-e): a CloudWatch metric filter and alarm on EntraKeyMaterialException, plus a warning about 30 days before the certificate’s end — either the backend logs or publishes the days left at startup, or the calendar reminder stands. Update 2026-10-23 (the lead; F38-e spec decision 1): not built in dev; the alarm and the expiry warning are candidates for UAT and production (.forge/owed.md “Decide the UAT and production alarm sets”), and the calendar reminder stands meanwhile Update 2026-10-26 (F38-h WI-3.1): F38-h’s uat overlay is built with no alarm (the lead, 2026-10-25; F38-h Decomposition Plan § Notes); UAT’s set comes as a later F38-h work item once the .forge/owed.md row “Decide the UAT and production alarm sets” is decided. The row stays open. |
Tharinda; F38-h, F38-i (the alarm) | open |
FS-7 — Background jobs (SQS)
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
Real SQS main queue + DLQ with redrive matching the app config (maxReceiveCount=3, visibility ~30 s); queue names via et.messaging.*; QueueProvisioner creates queues only under local/test |
LocalStack only | 007-background-jobs-spec.md Out-of-Scope table; plan Decisions 11, 13; QueueProvisioner.java 36–37 |
Terraform the queues; send a poison message; it lands in the DLQ after 3 receives Update 2026-10-21 (F38-b WI-2.1): the scheduler and mailbox ingest queues each redrive to their dead-letter queue at maxReceiveCount 3, with a 30-second visibility timeout and SQS-managed encryption, and both dead-letter queues keep messages 14 days and are empty; the poison message is F38-b WI-3.1’s; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. Update 2026-10-23 (F38-b WI-3.1): a malformed message sent to each of et-dev-scheduler and et-dev-mailbox-ingest was received three times by the running backend and reached its dead-letter queue, and both test messages were then removed; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. |
— | validated (F38-b WI-3.1, 2026-10-23) |
Spring Cloud AWS 4.1.1 on Boot 4.1 against real SQS via the default credential chain (task role); region fixed us-east-1; spring.cloud.aws.sqs.endpoint and credentials.* must be unset in cloud |
Smoke-tested only against LocalStack with local/local creds |
007-background-jobs-findings.md F-001; plan Decision 13; application.yml 52–65 |
Round trip on real SQS with task-role permissions; confirm the account region is us-east-1 Update 2026-10-23 (F38-b WI-3.1): the running backend (backend:60, task role, us-east-1) received messages sent from the CLI to et-dev-scheduler and et-dev-mailbox-ingest on real SQS and returned each failed one until it was dead-lettered, so the receive half works; the backend’s own send to SQS is not part of this check, so the round trip stays open; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. |
— | open |
| Several tasks consuming one queue, and a message outrunning the visibility timeout (uncertain — verify) | Proven in one JVM only | 007-background-jobs-spec.md FR-C1, FR-I4 |
Run 2+ tasks; a replay only ever produces one effect | — | open |
| DLQ-arrival and queue-depth alarms | Need FS-5’s substrate | 007-background-jobs-spec.md Out-of-Scope “alarms” row |
Push a message to the DLQ; the email arrives. Update 2026-10-23 (F38-e spec decision 1): not built in dev; UAT and production decide Update 2026-10-26 (F38-h WI-3.1): F38-h’s uat overlay is built with no alarm (the lead, 2026-10-25; F38-h Decomposition Plan § Notes); UAT’s set comes as a later F38-h work item once the .forge/owed.md row “Decide the UAT and production alarm sets” is decided. The row stays open. |
F38-h, F38-i | open |
FS-9 — Scheduler substrate (EventBridge Scheduler)
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
EventBridge Scheduler schedules + the scheduler.amazonaws.com IAM role with sqs:SendMessage, deliberately not mocked in LocalStack |
The IAM policy is the part that fails in production | 009-scheduler-substrate-spec.md Context, Out-of-Scope table, FR-D1; forge-et-be/docs/messaging/scheduler-cloud-contract.md |
Provision per the cloud contract; one real tick reaches the worker Update 2026-10-21 (F38-b WI-2.1): the four F38-b schedules (and F11-d’s lead-triage-sweep) exist in the group et-dev as the contract states, the role et-dev-scheduler may only send to the scheduler queue and is trusted only by scheduler.amazonaws.com from this account and group, and every schedule has fired with no target error, its messages reaching et-dev-scheduler; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. Update 2026-10-26 (F38-f WI-1.1): the six schedules of forge-et-iaac origin/dev are the six ENABLED in the group et-dev, and each has its own completion line at INFO in /ecs/dev/backend in the 26 hours read, so one real tick of each reaches the worker; evidence: §B F38-f “F38-f evidence — AC-8 (d)”. |
— | validated — producer half (F38-b WI-2.1, 2026-10-21); the worker’s log line (F38-f WI-1.1, 2026-10-26) |
The schedule’s message template (<aws.scheduler.scheduled-time>) produces a body the app can read, with a key stable across redeliveries |
Tested only with hand-sent messages | 009-scheduler-substrate-spec.md FR-S1, FR-X3 |
Inspect a real delivered body; a redelivered tick is ignored Update 2026-10-21 (F38-b WI-2.1): each schedule’s Input is the contract’s template with <aws.scheduler.scheduled-time> unescaped; a delivered body is not read here (reading one needs a receive); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. |
— | open |
| Assumption that EventBridge messages carry no trace attributes, so the worker starts a fresh trace (uncertain — verify) | Needs a real message | 009-scheduler-substrate-spec.md FR-T1 |
A real tick logs a new traceId |
— | open |
| Shared or dedicated scheduler queue (OQ-5), and its cloud name | Left as a cloud-half decision | 009-scheduler-substrate-spec.md OQ-5; scheduler-cloud-contract.md 71 |
Decide during the Terraform work Update 2026-10-21 (F38-b WI-2.1): one shared queue, et-dev-scheduler, is the target of every schedule in the group et-dev — the four F38-b jobs and F11-d’s lead-triage-sweep (F38-b spec FR-4); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. |
— | validated |
A cloud listener for scheduled messages — fixed in code by F20-b WI-2.1: ScheduledJobListener is no longer profile-scoped (only the demo handler is), and SchedulerConfigGuard refuses a cloud shape that carries a scheduled job without et.messaging.scheduler-queue (FS-9 spec Rev 1). Proven in local cloud-shape contexts only (SchedulerConfigGuardIT) |
No deployed task has consumed a real tick | 009-scheduler-substrate-spec.md Rev 1; ScheduledJobListener.java; SchedulerConfigGuard.java |
On dev, with ET_MESSAGING_SCHEDULER_QUEUE set, one real EventBridge tick is consumed (a Mailbox Subscription Renewal Completed line) |
F20-b / FS-1 cloud half | open |
| Failed / missed scheduled-job alarm | Needs FS-5’s substrate | 009-scheduler-substrate-spec.md Out-of-Scope “alarm” row |
Trip it once. Update 2026-10-23 (F38-e spec decision 1): not built in dev; UAT and production decide Update 2026-10-26 (F38-h WI-3.1): F38-h’s uat overlay is built with no alarm (the lead, 2026-10-25; F38-h Decomposition Plan § Notes); UAT’s set comes as a later F38-h work item once the .forge/owed.md row “Decide the UAT and production alarm sets” is decided. The row stays open. |
F38-h, F38-i | open |
FS-11 — File pipeline (S3 + GuardDuty) — see A-1
2026-10-09 — GuardDuty is not enabled. Altrium management (Thilina, with Malinda) decided to go
without GuardDuty, both Malware Protection for S3 and account-wide threat detection, and to add it
later only if it is needed or Empire Teas asks for it (relayed to Shyamin Ayesh by Tharinda,
2026-10-09). Every environment runs with scanning switched off by F38-j’s setting
(ET_FILES_SCAN_PROVIDER=none with ET_FILES_SCAN_OFF_ACKNOWLEDGED=true), as dev already does, so
each upload is stored NOT_SCANNED, downloadable, audited and counted. The F38-c scanner code stays on
main unused, so turning scanning on later needs Terraform and two settings, no code. The rows below
that prove GuardDuty itself are withdrawn; the switching row is narrowed to the switched-off half.
Before scanning is switched back on (2026-11-13): F36-a’s card photos sit under capture/ in the
same bucket with no attachment row (F36-a spec Rev 2), so the change also needs owed.md’s row
“Scan capture photos, and handle capture/ keys in ScanResultEventConsumer, before malware
scanning is switched on” settled first.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
Real FileScanProvider (GuardDuty Malware Protection) and real RecordVisibilityCheck; until both exist SeamGuard blocks every non-local/test boot (A-1). The scanner is implemented (F38-c wave 1): GuardDutyFileScanProvider resolves in every cloud shape and makes no AWS call. The visibility check is implemented too (F5-f wave 1): RoutingRecordVisibilityCheck is the one RecordVisibilityCheck in every profile, and SeamGuardIT proves a fully configured cloud-shape context starts past the guard with it. What remains: only F38-f’s proof on dev that a cloud boot passes the guard with both real (§B F5-f). Update 2026-10-09: with GuardDuty not enabled (block note above), the cloud scanner is F38-j’s none provider, which SeamGuard accepts; dev booted that way on 2026-10-09, so what F38-f records is that boot with the real visibility check |
No deployed environment | 011-file-pipeline-plan.md Decision 13, Notes; 011-file-pipeline-findings.md F-004; SeamGuard.java; GuardDutyFileScanProvider.java; RoutingRecordVisibilityCheck.java; F38-c spec FR-1, AC-5; F5-f spec FR-C1, AC-17 |
Once F5-f wave 1 is on main: a cloud-profile boot on dev passes the guard (F38-f) |
F38-f (dev proof) | open |
Withdrawn 2026-10-09 — GuardDuty is not enabled (block note above); revisit only if it is added. GuardDuty Malware Protection enabled on the bucket, plus the EventBridge rule → scan-result queue → the backend’s consumer delivering scan results. The consumer is on main (F38-c wave 1): ScanResultQueueListener (attachment/service) reads each GuardDuty event from the queue named by ET_MESSAGING_SCAN_RESULT_QUEUE, under the runWithTrace job name attachment-scan, and hands the verdict to ScanResultHandler. What the infrastructure must build (F38-b) is forge-et-be/docs/messaging/scan-result-cloud-contract.md. Still to prove on dev (F38-f): (1) real EventBridge delivery of the GuardDuty event onto the scan-result queue, with no input transformer; (2) the pre-commit retry under real timing — a result redelivered until its upload’s row commits, then applied; (3) each deferred delivery or malformed body is a thrown listener exception, which Spring Cloud AWS logs as one ERROR line from its AbstractMessageProcessingPipelineSink logger (confirmed by WI-1.1) — F38-e’s alarm filters must exclude that logger rather than match ERROR broadly, and on dev a deferral must not page; the queue’s visibility timeout must also comfortably exceed Spring Cloud AWS’s batched-delete delay (up to about 1 s) plus the handler’s time (contract § “The queue”); (4) the FR-9 rescan procedure — run the contract’s § “Rescanning a PENDING attachment” on a PENDING attachment and see the verdict arrive through the same queue; (5) the protection plan’s service-role, kms:Decrypt/key-policy and queue-policy grants — an EICAR upload scans COMPLETED/THREATS_FOUND, not SKIPPED/ACCESS_DENIED |
Offline the consumer is proved only against GuardDuty-shaped sample events on LocalStack; no GuardDuty, EventBridge or IAM exists | 011-file-pipeline-spec.md FR-S2, FR-S4, Out-of-Scope row; F38-c spec FR-2, FR-4 to FR-7, FR-9, FR-10; WI-1.1 plan Decision 18, Risk R-3; docs/messaging/scan-result-cloud-contract.md |
Upload the EICAR test file; the attachment goes INFECTED and every stored copy is deleted. Also check the audit event (F39): exactly one SYSTEM attachment.scan_completed row for the attachment, labelled attachment-scan (the listener’s job name is the outermost declaration, so it labels the row — ScanResultHandlerSpec pins that rule; F39 WI-1.6 plan ## Notes). Then walk items (1) to (5) above |
F38-b (resources), F38-f (proof) | withdrawn |
S3 bucket with public-access block and versioning (Terraform); LocalBucketProvisioner creates it only under local |
LocalStack does not enforce the public-access block | 011-file-pipeline-spec.md AC-7, FR-C1; plan R-2; LocalBucketProvisioner.java 21–45 |
Unsigned GET → 403; signed 5-minute URL → 200 Update 2026-10-21 (F38-b WI-2.1): the bucket et-dev-attachments is versioned, with all four public-access blocks on, default encryption with the storage key and Bucket Keys on, and a bucket policy refusing requests not over TLS 1.2 or later; the unsigned 403 and signed 200 are F38-b WI-3.1’s; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-5. Update 2026-10-23 (F38-b WI-3.1): an object uploaded to et-dev-attachments was stored under the storage key; an unsigned GET returned 403 and a five-minute signed URL returned 200 with the same content; the object’s version was then deleted, leaving nothing; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-5. |
— | validated (F38-b WI-3.1, 2026-10-23) |
Task-role S3 permissions including listing and deleting object versions; presigned URLs signed with the role’s temporary credentials (credentials half uncertain — verify); et.files.storage.endpoint-override must be unset so DefaultCredentialsProvider is used (set → static test/test LocalStack creds). Since F38-c, also s3:GetObject on the bucket’s objects and s3:ListBucket on the bucket, which the scan-result consumer’s HeadObject existence check needs: without s3:ListBucket, S3 answers a missing key with 403, not 404, so FR-6’s drop would dead-letter instead of acknowledging (WI-1.1 plan Decision 9; spec Rev 2). LocalStack does not enforce IAM |
Needs the IAM role | 011-file-pipeline-plan.md Decisions 4, 7; S3Config.java 44–77; F38-c spec FR-6, FR-10; docs/messaging/scan-result-cloud-contract.md § “The task role” |
Infected purge succeeds; a URL issued near credential rotation still works for its 5 minutes. For the scan-result consumer: the pre-commit retry defers (an INFO Scan Result Deferred line, then the verdict), and a result for a deleted object is acknowledged with the Scan Result Dropped WARN rather than reaching the dead-letter queue Update 2026-10-21 (F38-b WI-2.1): the task role’s s3 policy allows exactly s3:ListBucket and s3:ListBucketVersions on et-dev-attachments and s3:PutObject, s3:GetObject, s3:DeleteObject and s3:DeleteObjectVersion on its objects, and the storage key through S3 only; the presigned-URL and real-call proof is F38-f’s; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-5 and AC-6. |
F38-b (role), F38-f (proof) | open |
ET_FILES_STORAGE_BUCKET (and region) on task definitions |
Required key (@NotBlank) with no default |
011-file-pipeline-findings.md F-010; plan Notes |
Dev boots with it set Update 2026-10-21 (F38-b WI-2.1): dev’s value is et-dev-attachments, the forge-et-iaac output attachments_bucket_name; F38-d sets it on the task definition (its FR-6); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-9. |
— | open |
Withdrawn 2026-10-09 — no scan runs while GuardDuty is not enabled (block note above). Alarm on INFECTED / scan failures. The signals are on main (F38-c): the Attachment Infected ERROR line (MDC attachmentId, ownerType, ownerId, scanVerdict=INFECTED) from ScanResultHandler; the Scan Result Without Verdict WARN line (MDC attachmentId, scanStatus, scanResultStatus, statusReasons) from ScanResultEventConsumer; and the counter et.attachment.scans.no_verdict, tagged scanResultStatus with the four values UNSUPPORTED, ACCESS_DENIED, FAILED and OTHER — alarm on each. The tag key is scanResultStatus, but its values are outcome names: a FAILED scan with any other result status still tags as FAILED. How the counter reaches CloudWatch is open — see the “uploads with no scan results” row below: no meter registry exports Micrometer meters yet |
Needs FS-5’s substrate and a deployed environment | 011-file-pipeline-spec.md FR-S3, Out-of-Scope row; F38-c spec FR-8, FR-11; F38-c WI-2.1 plan Decisions 2, 3, 7; forge-et-be/docs/messaging/scan-result-cloud-contract.md § “What is counted” |
Trip the INFECTED half with an EICAR upload (one ERROR line, no alarm on a second delivery), and the no-verdict half with the password-protected PDF of the F38-f no-verdict row below (one WARN, one count under UNSUPPORTED) |
F38-e (was the FS-5 cloud half) | withdrawn |
Withdrawn 2026-10-09 — GuardDuty is not enabled (block note above). A result with no verdict on real GuardDuty. Upload a file GuardDuty cannot open (a password-protected PDF) on dev; see one Scan Result Without Verdict WARN for its attachment id with scanStatus=SKIPPED and scanResultStatus=UNSUPPORTED, one count on et.attachment.scans.no_verdict under UNSUPPORTED and one on et.attachment.scans.results under UNSUPPORTED, with the attachment still PENDING and not downloadable. Read the WARN’s statusReasons as AWS actually writes it: any object key or bucket name in it is replaced by [redacted], but other object-identifying text would not be — check in particular for a URL-encoded key (%2F) or a partial key such as the UUID alone, which the literal replacement does not catch (WI-2.1 plan Risk R-4) |
Offline this is proved only against AWS’s documented sample events on LocalStack; no GuardDuty exists | F38-c spec FR-8, FR-8a, FR-11, AC-4; F38-c WI-2.1 plan Decisions 3, 6, 7 | On dev after bring-up: upload the PDF, read the WARN line and the two counts (through whatever export F38-e chose, or the task’s logs) | F38-f | withdrawn |
Withdrawn 2026-10-09 — with no protection plan there is nothing for it to detect (block note above). Proposed alarm on uploads with no scan results over a period F38-e chooses. It compares et.attachment.uploads (every committed upload sent for a scan — an upload stored NOT_SCANNED while scanning is switched off is not counted, so the alarm stays quiet then; F38-j decision 11) with et.attachment.scans.results (every scan result applied to a known attachment, tagged outcome), each summed across ECS tasks — each task counts its own work and resets on restart. After bring-up it is the only detector for a missing or prefix-scoped protection plan, which produces no event at all; an alarm on attachments left PENDING cannot do it, because UNSUPPORTED files stay PENDING for good. Every series is registered at start-up, so both read zero before the first event. The export gap: no meter registry exports Micrometer meters today (gradle.lockfile: micrometer-core only; no export in application.yml). F38-e chooses the export path: a Micrometer CloudWatch registry (a new dependency, ASK FIRST) or CloudWatch Logs metric filters on the log lines. With metric filters, use scanResultStatus / outcome as dimensions, never statusReasons, which is free text written by AWS and would make the dimension’s cardinality unbounded. This applies to the no-verdict counter of the alarm row above as well. In local and test, uploads count but scan results do not (the in-process dispatcher bypasses the consumer), so the pair means something only in a cloud shape |
No deployed environment; the counters are in-process only | F38-c spec FR-8a, FR-11, § Constraints (the FS-11 FR-C2 note); F38-c WI-2.1 plan Decisions 2, 4, 9; F38-j spec decision 11, FR-6; forge-et-be/docs/messaging/scan-result-cloud-contract.md § “What is counted” |
F38-e decides whether to build the alarm and chooses the export path; then on dev upload a file and see both counters move by one, and with the protection plan detached see uploads move with no scan result, and the alarm fire. 2026-10-02: with GuardDuty decided on, this alarm is recommended to F38-e as required rather than optional (F38-e spec Open Questions); F38-j’s switch covers turning the scanner off on purpose, and this alarm covers the protection plan going missing without it | F38-e (proposed) | withdrawn |
Narrowed 2026-10-09 (GuardDuty not enabled, block note above): F38-f proves the switched-off half only — the WARN once per task, an upload stored NOT_SCANNED and downloadable with its USER attachment.scan_skipped row and reason=upload count, and the refusal without the acknowledgement. Dev has run switched off since its first boot, so there is no protection plan, rule or queue to remove, and the switching-back-on half waits until GuardDuty is added. Switching the scanner off and back on, on dev (F38-j), by forge-et-be/docs/messaging/scan-result-cloud-contract.md § “Switching scanning off and back on (F38-j)”. Scanning is switched off with ET_FILES_SCAN_PROVIDER=none together with the acknowledgement ET_FILES_SCAN_OFF_ACKNOWLEDGED=true (without it a deployed task refuses to start, naming both keys). Each task then logs one Malware Scanning Switched Off WARN and runs the startup sweep, which moves every PENDING attachment to NOT_SCANNED with one SYSTEM attachment.scan_skipped audit row labelled attachment-scan-off-sweep each; each upload is stored NOT_SCANNED, downloadable, audited (USER attachment.scan_skipped) and counted on et.attachment.scans.skipped (reason=upload, and reason=sweep for the sweep). Switched back on, the NOT_SCANNED files are listed and rescanned by the contract’s § “Rescanning a PENDING attachment” |
The setting, the sweep and the rescan rule are backend code, proved in the test profile; removing and restoring a real protection plan, rule and queue needs the dev account |
F38-j spec decisions 4, 6, 7, FR-8 to FR-12, FR-14; forge-et-be/docs/messaging/scan-result-cloud-contract.md |
On dev after bring-up, once F38-j is on main and deployed: walk the switching-off steps (the WARN once per task, the sweep’s INFO line, no PENDING row, then the plan, the rule, the queue key and the queue removed); upload and download a file (NOT_SCANNED, one USER attachment.scan_skipped row, the reason=upload count); deploy none without the acknowledgement once and see the task refuse to start naming both keys; then walk the switching-back-on steps (a new upload goes PENDING → CLEAN; one NOT_SCANNED file is rescanned to CLEAN, with an attachment.scan_completed row starting from NOT_SCANNED) Update 2026-10-26 (F38-f WI-1.1): the WARN once per task is shown — the one running task’s stream holds exactly one Malware Scanning Switched Off line at WARN, from its startup at 2026-10-26 03:06 UTC; the upload, its audit row and count, and the refusal without the acknowledgement are still to be walked (F38-f WI-4.1 and WI-5.1); evidence: §B F38-f “F38-f evidence — AC-8 (e)”. Update 2026-10-26 (F38-f spec Rev 4): no database tunnel is built; the upload and its audit event are walked by F38-f WI-2.6 (the audit service’s DEBUG line in a short log window), the rest by F38-f WI-3.1. |
F38-f | open |
| Withdrawn 2026-10-09 — GuardDuty is not enabled; the cost annexe’s GuardDuty line already says so. Running-cost line for GuardDuty with current rates confirmed (optional AWS Budgets guardrail) | Owed with F38-b’s resources (GuardDuty decided on, 2026-10-02) | 011-file-pipeline-spec.md OQ-B, Constraints |
Add to .forge/client/sow-cost-annexe.md |
— | withdrawn |
Cloud-shape integration tests (CloudShapeIT, MailboxCloudShape*IT, SeamGuardIT, ScanResultQueueGuardIT) boot with the real GuardDuty provider (which makes no AWS call), the real visibility check (RoutingRecordVisibilityCheck), fake mailbox / key beans and Testcontainers LocalStack. Since F38-c no fake set supplies a scanner, and since F5-f none supplies a visibility check, so these tests prove the guards and the two file-pipeline seams’ wiring — never GuardDuty scanning, EventBridge, IAM or a deployed boot |
By design | forge-et-be/src/test/.../support/CloudSeamFakes.groovy; SeamGuardIT.groovy class block (“Cannot prove that GuardDuty scans anything, or that a deployed task boots”); F38-c plan WI-1.2 Decision 3; F5-f WI-1.2 plan D6, D7 |
Nothing to change; this row exists so nobody reads a green CloudShapeIT as cloud proof |
— | open |
F4-a — Pipedrive import engine (dry-run report)
Spec FR-G1 names one cloud-gated item. Everything else in F4-a — reading the exports, the mapping, the
comparison against Postgres, the report, the audit event, the 30-second bound on the export-sized set — runs
and is proved locally (forge-et-be ./gradlew check; WI-2.1 plan ## Progress).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The dry run on a deployed environment: POST /api/v1/migration/pipedrive/dry-run on dev behind the ALB, with its request-size limit and its 60-second idle timeout; later on production, with the production import owner (Kavindu Anthony’s CRM user) |
No AWS account yet (§A-5); locally the endpoint runs on a laptop with no ALB in front of it | F4-a spec FR-G1, NFR-1; WI-2.1 plan subtask 11 | Run the dry run on dev with the same three files as the last local run (a developer’s machine, counts only — never the report itself, spec NFR-2); check that the report’s summary counts match the local run and that the response arrives inside the ALB idle timeout. Also confirm the ALB passes a 27 MB multipart body (the backend’s max-request-size) |
F38 / Dinil | open |
F4-d — Pipedrive migration write run
Spec FR-G1 names two cloud-gated items. Everything else in F4-d — the write run’s checks, its one transaction and
lock, the group writes and tags, the digest, the report, the audit events and the timing on the export-sized set —
runs and is proved locally (forge-et-be ./gradlew check; the F4-d WI-2.1 and WI-2.2 plans’ ## Progress, which
holds the local timing reading).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The write run on dev: POST /api/v1/migration/pipedrive/runs behind the ALB, with its 60-second idle timeout and its request-size limit; and the database’s lock table, because a run holds one advisory lock per tagged record it did not create, plus a row lock per account and contact it merges into or updates, until it commits (since WI-2.1 revision 2’s bulk attach a first load into an empty CRM takes almost none; a final run’s count is its merges and updates), and RDS’s lock table is max_locks_per_transaction × max_connections entries. Locally the first-load write run took 3.2–4.7 s (2026-10-24, 2,339 tagged records) |
No AWS account yet (§A-5); locally the run answers on a laptop with no ALB in front of it, against the Testcontainers Postgres’s default lock table | F4-d spec FR-G1, NFR-1; F4-d WI-2.2 plan Decision 16, ## Risks; F4-d WI-2.1 plan ## Risks (lock-table exhaustion) |
With the same synthetic or redacted files as the last local run (counts only — never the report, spec NFR-2): a dry run, then a write run with its digest, on dev; check the run’s Written counts match the local run’s and the response arrives inside the ALB idle timeout; then a second dry run and write run, every row unchanged. Record dev’s max_locks_per_transaction and max_connections beside the run’s tagged-record count. If the run does not fit the timeout, spec Q-2’s fallback (a background job) is a spec revision |
F38 / Biseni | open |
The production runs — the first load and the final run — with the production import owner (Kavindu Anthony’s CRM user, whose role holds settings.manage, account.manage, account.all, lead.view, lead.all, and lead.manage for F4-b’s lead writes) |
The production environment does not exist yet (§A-5) | F4-d spec FR-G1, Q-9 (the default, which Tharinda confirms — owed.md) |
The delivery team runs the dry run with Kavindu Anthony on the day’s fresh exports, he checks the report, then runs the write run with its digest; after the first load, a dry run shows every created record unchanged. Counts only in any note | Delivery team with Kavindu Anthony | open |
F4-b — Pipedrive migration: lead writes
Spec FR-21 names three cloud-gated items. Everything else in F4-b — the lead writes of the write run, its permission keys, the digest over the leads, the over-limit refusal, the changed-in-the-CRM rule, the report and the audit events, and the timing at the export’s size — runs and is proved locally (forge-et-be ./gradlew check; the F4-b WI-1.1 and WI-1.3 plans’ ## Progress, which hold the local timing readings). Items (a) and (b) run on dev in one run, after F4-b’s wave is on dev; the leads they create queue real Bedrock calls, so the run needs the lead’s OK (.claude/CLAUDE.md § Boundaries › Ask first). Counts only in any note — never the report or a cell value (F4-d spec NFR-2).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) The write run with leads on dev — POST /api/v1/migration/pipedrive/runs behind the ALB with files of the export’s size (692 leads, 4,235 organisations, 4,977 people), answering inside the ALB’s 60-second idle timeout; and the database’s lock table, because the run locks every lead it changes, and every row those writes lock, up front in id order until it commits (NFR-1), and RDS’s lock table is max_locks_per_transaction × max_connections entries. The local reading at the export’s size is in the F4-b-WI-1.3 plan’s ## Progress (PipedriveRunTimingIT, bound 30 s) |
Only a deployed environment has the ALB in front of the backend and RDS’s lock table; locally the run answers on a laptop with no ALB, against the Testcontainers Postgres’s default lock table | F4-b spec FR-21 (a), NFR-1, NFR-3, AC-15; F4-d spec FR-G1; §B F4-d’s dev row | With the lead’s OK (item (b)) and the same synthetic or redacted files as the last local run: a dry run, then a write run with its digest, on dev; check that the run’s lead counts (created, updated, unchanged, changed in the CRM, refused, missing) match the local run’s and that the response arrives inside the ALB idle timeout, and record the time it took. Then a second dry run and write run on the same files: every lead unchanged (NFR-3). Record dev’s max_locks_per_transaction and max_connections beside the run’s lead, account and contact counts. If the run does not fit the timeout, F4-d spec Q-2’s fallback (a background job) is a spec revision |
F38 / Dinil | open |
(b) The triage backlog on dev — the time to triage the leads a first load of the export’s size creates, under dev’s pacer: from the write run’s commit until the last of its created leads is triaged. Expected (NFR-2): about 1.5 hours with F43’s 60-second wait for a background slot, at Haiku 4.5’s 10 requests a minute; about 4 hours on an environment without F43’s wave 1, at F42’s defaults; or about 1.5 hours with ET_LEADTRIAGE_SWEEPBATCH raised to about 600 for the run. A lead a person types in during that time waits behind the backlog |
Only real Bedrock enforces the account’s quota, and only the deployed pacer and triage queue drain a backlog; locally triage runs on the AI stub, and the pacer’s waits are proved on a fake clock | F4-b spec FR-21 (b), FR-19, FR-20, NFR-2, AC-14, AC-17; F43 spec FR-D4, decision 3; F42 spec W-5; §B F42 item 1 and §B F43 item 1 (the same log reads) | Read from item (a)'s first-load write run, with no second load. Before the run, the lead decides whether lead research stays on for it: left on, F11-c’s hourly sweep researches the created leads at about 50 an hour (about 14 hours), one Exa search each of dev’s credit (.forge/owed.md, “Keep a UAT round from spending dev’s Exa credit”), and with F11-g’s ranking the run costs about $35 of AI (NFR-2); triage alone is about $2. No live evaluation runs and nothing is deployed to dev until the backlog has drained. Then read the backend’s AI Call Completed and AI Call Rate Limited lines for lead-triage from the run’s commit (the time of its migration.applied event) until each lead the run created — found by the run id, FR-20 — has a triage result: the time of the last, the count, any lead left to a later sweep, and the total costUsd. Record the pacer settings dev ran with (whether F43’s wave 1 is on dev, and ET_LEADTRIAGE_SWEEPBATCH), and log the measured time beside the expected figure in this register’s ## Log |
Dinil (F4-b) | open |
© The production first load and final run, with the lead writes — the runs of §B F4-d’s production row, now also writing leads: the caller holds F4-b FR-3’s keys (settings.users.manage, settings.roles.manage, account.view, account.all, account.manage, lead.view, lead.all, lead.manage), and the import owner’s role holds lead.manage and lead.all (a write run refuses an owner without them); each run is made out of working hours, with no deploy and no live evaluation run while its triage backlog drains (item (b)'s figure) |
The production environment does not exist yet (§A-5) | F4-b spec FR-21 ©, FR-3, FR-19, NFR-2, W-3, § Dependencies; F42 spec FR-H4; F4-d spec FR-G1, Q-9; .forge/owed.md “Confirm who runs the production Pipedrive runs, and as which import owner” and “Decide how F11-d’s duplicate rule treats the final Pipedrive run” |
The runs follow §B F4-d’s production row (Kavindu Anthony checks each dry run’s report on the day’s fresh exports before its write run). Before the first load, F42’s dev quota checks (F42 FR-H4 items 1 and 3) have passed (§B F42), and the dry run’s import-owner section shows no CANNOT_REASSIGN warning for the chosen owner. Before the final run, the duplicate-rule row in owed.md is settled (spec W-3), and the final run’s dry run gives the number of leads it will update, from which its AI cost follows (about 5 US cents a lead, NFR-2) — record both here. After the first load, a dry run shows every created lead unchanged. The first load’s one-off AI cost (about $35), on top of the expected ~$26–31 a month of AI usage, takes that month over the running-cost annexe’s recommended $50 monthly AI budget and stays under its $150 billing alert (NFR-2, .forge/client/sow-cost-annexe.md). Counts only in any note |
Delivery team with Kavindu Anthony | open |
F5-a — Sign in with Microsoft and sign out (done)
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
| Real Microsoft (Entra) OAuth2 authorization-code round trip | Needs the Entra app registration + a reachable reply URL; no local Entra | AD #28; F5-a spec (AC-4 asserts only the full-page nav target locally); W-1 / R-FND-003; forge-et-fe/e2e/browser/auth/oauth-initiation.spec.ts aborts the /oauth2/authorization/** navigation |
Sign in as a real Empire M365 user end-to-end; the pre-created user’s identity link is written (no user is auto-provisioned — AD #34), a sign-in success event is written to the audit trail (once F39 is on main), and /me returns the expected role and permissions |
— Laptop, not dev (2026-09-29, Nanndish, F5-i-WI-2.1 R-1): real Microsoft sign-in and sign-out as CRM01 completed, with identity linking and sign-in/sign-out audited. Still owed on dev. | open |
from-route preservation across the real full-page redirect, on first login and on the session-expiry path (AC-8’s “your session expired” notice then return to the page). The carrier is app code, not a cloud item (2026-09-30): the route was kept only in router location state, which the full-page redirect drops, so it is lost today. That is BUG-054 — the fix (forge-et-fe#101) stores the validated path in sessionStorage before the redirect and restores it once after /me succeeds. It is covered by unit tests and the stub-path browser suite. Not yet run: the real-Entra check in laptop mode (F5-i), where a sign-in from a deep link lands back on it. It needs a person signed in to a test Microsoft account, so it is a manual step for Tharinda after #101 merges. What stays cloud-gated is the confirmation on dev, with the real Entra round trip through CloudFront and api.* |
The dev confirmation needs a deployed frontend and backend behind the real reply URL (§A-5) | F5-a WI-1.2 Decision 7 / Risk; spec FR-3 / AC-5, FR-6 / AC-8; WI-2.1 Risk W-1; bugs.md BUG-054 |
Once BUG-054 is fixed: on dev, after a real login from a deep link, the user lands back on the attempted protected route; after a real expiry and re-login, the notice shows and the route is restored | BUG-054 (code); F38 (dev check) | open |
The real ?login=failed redirect when Microsoft succeeds but the CRM refuses the account (unknown or unlinkable user) |
AC-10 checks only that the login page reacts to the query parameter | spec FR-8 / AC-10; F5-a-Wave-1-contract.md § OAuth2 flow (cloud only); BUG-037 (the redirect names /login, because the root dropped the marker) |
Sign in with a tenant account that has no CRM user (a deactivated user’s refusal is the §B F5-c row); land on /login?login=failed with one vague “couldn’t sign you in” message; URL and page reveal no reason; the refusal is written to the audit trail (once F39 is on main). The redirect is a deep link, so it needs the CloudFront deep-path rule of F38 AC-14. Laptop evidence (2026-09-29, laptop, not dev): a real refusal of CRM05@empire.lk against the Empire tenant wrote auth.sign_in_refused; after BUG-037, the backend’s refusal redirect showed the message in a fresh browser |
— Laptop, not dev (2026-09-29, Nanndish, F5-i-WI-2.1 R-2): CRM02 signed in at Microsoft, was refused by the CRM, landed on ?login=failed, and auth.sign_in_refused was audited. Still owed on dev. | open |
The real ?login=unavailable redirect when the backend cannot sign the Entra client assertion (certificate expired, not yet valid, unreadable or not matching its key). Locally each half is proved on its own: EntraLoginFailureSpec (forge-et-be#99) drives the failure handler with a built exception, and LoginRoute.test.tsx and e2e/browser/auth/login-page.spec.ts (forge-et-fe#75) open /login?login=unavailable directly. No test runs the full path, because no local sign-in builds a client assertion |
Needs the Entra registration and a certificate pair in Secrets Manager; the stub sign-in never signs anything | F5-a spec FR-8 / AC-10b (Rev 5); forge-et-be auth/config/EntraLoginConfig.java (unsignableAssertion(), failureHandler()) |
On dev, set the certificate secret to a pair that does not match (or an expired test certificate) and start a sign-in: the browser lands on /login?login=unavailable (the path since BUG-037) with the “Sign-in is unavailable right now” message, never a 500; the backend logs one Entra Sign-In Refused - Client Assertion Unsignable ERROR line; no request reaches Entra’s token endpoint. Restore the real secret and confirm sign-in works again. The F38-e alarm on EntraKeyMaterialException (§B FS-6 certificate-expiry row) should fire from the same test. Update 2026-10-23 (F38-e spec decision 1): dev has no such alarm, so on dev the test checks the ERROR line only; the alarm is checked where UAT or production builds it |
F38 | open |
et.security.bootstrap-admin-email); FirstAdministratorBootstrap is inert when the key is unset, so without it nobody can sign in |
Needs the task-definition secret set on a truly empty DB | F5-a WI-1.3 (app code done, unit/IT green); FS-6 R11; FirstAdministratorBootstrap.java 68–71; AD #34 |
FS-1 cloud half | withdrawn | |
VITE_API_BASE_URL, the frontend’s build-time API address — the environment’s api.* address under AD #33 (https://api.crm.empire.lk, https://api.uat.crm.empire.lk, https://api.dev.crm.empire.lk) |
The CloudFront frontend needs the right value per environment; no integration test can catch a regression | F5-a-WI-1.1-session-http-foundation.md Decision 5, Risk “Cross-host forward-compat”; forge-et-fe/src/lib/api/apiClient.ts 122 |
Set per environment; /api/v1/me, sign-out and one write reach the API with cookies included |
FS-1 cloud half | open |
| The smoke suite’s session on dev — captured by a person’s Microsoft sign-in each run and deleted afterwards (F38-g FR-2) | The captured cookies are for localhost:5173 via the stub login and dev proxy |
F5-a WI-1.4 Risk; forge-et-fe/e2e/fixtures/auth.setup.ts 23–32 |
Validated 2026-10-22 (F38-g-WI-2.1 R-3 to R-5): the session file was written, used and deleted on every run, interrupted runs included; record .forge/plans/F38-g/F38-g-WI-2.1-record.md |
F38-g | validated |
Playwright E2E storageState regenerated for the cloud origin — the full tier’s session against uat |
The captured cookies are for localhost:5173 via the stub login and dev proxy |
F5-a WI-1.4 Risk; forge-et-fe/e2e/fixtures/auth.setup.ts 23–32 |
Regenerate the shared auth storageState against the cloud origin for the standing tiers (a real-user or test-account path replaces the stub — on dev, the seeded IT Admin CRM01@empire.lk or a test account holding a test role, since F5-k retires the superadmin sessions the local suites use today). Update 2026-10-19 (F38 Rev 21): on dev the smoke suite captures its own session each run, by a person’s Microsoft sign-in as the reserved view-only smoke account, and deletes it afterwards (F38-g FR-2); the shared storageState is not regenerated for dev. The full tier’s session against uat is F38-h’s |
F38-h | open |
Role-scoped navigation with a real limited-role user — the stub only resolves seeded users, so F5-b’s AC-4/AC-12 fake /me permissions in the browser (partly a seed-data limit) |
No real limited-permission user can sign in locally | F5-b-WI-2.1-plan.md § Notes “limited-role fixture” |
Sign in as a real Entra user holding a limited role; hidden nav entries and the fallback landing page behave as specified | — | open |
F5-b — App frame and navigation shell (done)
The Wave-1 note that its browser tests are “not run by any PR lane (deferred to the FS-1 cloud
half)” is the FS-4 full-tier row (“The full E2E tier on the UAT tag”). The deployed-bundle version check is the FS-1 row. The
limited-role fixture row sits under F5-a. No further F5-b rows (checked spec, findings,
plans/F5-b/**, 2026-09-23).
F5-c — Users: list, create, role, manager, deactivate (done 2026-10-06 — spec Rev 7; every row below is still owed on dev)
Pre-registered from the approved plans, which each call the live link cloud-gated
(F5-c-decomposition-plan.md Notes; F5-c-WI-1.1-plan.md); the shipping PRs refine them.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
An administrator-created user links to their Microsoft account on first real sign-in — the (provider, subject) link by email match (FS-6 FR-A3) |
Only a real Entra sign-in produces a real sub; locally WI-1.1 proves only that the created user is active and that a stub sign-in by that email resolves to them, with /me showing their role (UserControllerIT) |
F5-c-user-management-spec.md Rev 5 FR-4 / AC-5; F5-c-WI-1.1-plan.md (re-planned 2026-10-03, ## Approach out-of-scope note and subtask 9); F5-c-decomposition-plan.md Notes (“AC-5 … partly cloud-gated”); FS-6 FR-A3 |
On dev, sign in as the migration-seeded IT Admin and create a user for a test mailbox account that has no CRM user yet (for example CRM02@empire.lk) through POST /api/v1/users or the WI-1.2 Users screen; then sign in as that account with Microsoft: the user_identity row is written, a user.identity_linked audit event appears for the new user, and /me returns the new user’s role |
— | open |
| A deactivated user is refused by the real Microsoft sign-in, and their live sessions end | Refusal on the real callback path; session kill with more than one ECS task sharing the JDBC store | F5-c spec (deactivate blocks sign-in); F5-a spec FR-8 (?login=failed); AD #34 (deactivation deletes the user’s session rows by the principal-name index); F5-c-WI-3.1-plan.md D5 (the spring_session rows are deleted by principal_name inside the deactivation transaction; locally UserActivationIT proves it for a stub sign-in) |
Deactivate a signed-in test user on dev: their next request is 401; a fresh Microsoft sign-in lands on the login page with the one vague message. The user’s sessions on every ECS task end at once: sign the user in twice, through two browsers, while dev runs two tasks, deactivate them, and both browsers’ next request is 401 |
— | open |
| After an administrator changes a user’s email, the user’s next real Microsoft sign-in by the new address links again, the old address is refused, and the mailbox connect works only after that sign-in | Only a real Entra sign-in produces a real sub and a verified email; locally WI-2.1 proves only that the email change deletes the user’s user_identity rows with one user.identity_unlinked each (a change of case keeps them) and that a stub sign-in by the new address resolves to the same user (UserControllerIT, UserAuditHttpIT, RepositoryUserAdministrationSpec). The mailbox connect (F20-a) looks the Microsoft subject up through the link, so it is refused between the change and the re-sign-in |
F5-c-user-management-spec.md FR-4a / AC-17; F5-c-WI-2.1-plan.md D6, ## Risks (the mailbox paragraph) and subtask 9; F5-c-WI-3.1-plan.md D8, D9; F5-c-Wave-2-contract.md § Field Definitions (email); FS-6 FR-A3 |
On dev, as the IT Admin, create a user for a test mailbox account (for example CRM03@empire.lk) and have it sign in once with Microsoft (user.identity_linked). Change its email through PATCH /api/v1/users/{id} (or the WI-2.2 edit) to a second test mailbox address that has no CRM user (for example CRM04@empire.lk): the user_identity row is gone and user.identity_unlinked is recorded. Before the user signs in again, a mailbox connect is refused (no Microsoft identity). Then a Microsoft sign-in as the new address writes a new user_identity row and user.identity_linked for the same user id, and /me answers that user; a Microsoft sign-in as the old address is refused (?login=failed); after the re-sign-in, the mailbox connect succeeds. The email change releases a connection made before it (mailbox.released, status_reason = OFFBOARDED, its Graph subscription deleted), and the user connects their mailbox again after signing in with the new address (spec Rev 7, AC-20; WI-3.1 D8, D9) |
— | open |
| Deactivation and an email change release the user’s live mailbox connection: the Graph subscription is deleted and the stored tokens discarded | Only provable against a live mailbox; locally WI-3.1 proves the connection row released with mailbox.released in the same transaction, against the stub provider (UserActivationIT, UserControllerIT, UserAuditHttpIT) |
F5-c-user-management-spec.md FR-7, FR-4a / AC-8, AC-20; F5-c-WI-3.1-plan.md D8; F20-a FR-8 |
On dev, a test mailbox user (for example CRM03@empire.lk) connects their mailbox; the IT Admin deactivates them: the connection is DISCONNECTED / OFFBOARDED with its token columns empty, the Graph subscription is gone (no further change notification for that mailbox), and mailbox.released is recorded; reactivate, sign in and reconnect succeeds; repeat with an email change instead of a deactivation, and with a role edit that removes email.manage from the user’s role (F20-i WI-1.1: the connection is DISCONNECTED / PERMISSION_REMOVED, disconnected_by the administrator, with the same token check; this release makes no Graph call, so the subscription is left to expire, and no change notification for that mailbox is processed after the release — the lead’s decision of 2026-11-02 in the WI-1.1 plan’s ## Notes). A connect that completes in the same instant as the deactivation or the email change waits for it, or the change waits for the connect, on a lock on the user’s row (BUG-100, fixed in forge-et-be#235 and proven locally by MailboxConnectOwnerChangeRaceIT); nothing about that race needs the cloud |
— | open |
F5-c was re-scoped on 2026-09-24 (AD #34): it gates on access.manage, drops the system-role flag, writes audit events, and refuses deactivation for oneself, for the last admin, while the user has direct reports and while they own records. Neither of the first two rows above changes; the re-scoped spec refines them.
F5-f — Ownership and sharing substrate (wave 1 builds the real RecordVisibilityCheck)
The spec’s one cloud-gated item (FR-C1). It replaces the F5-f “nothing cloud-gated” row and the F5-f backlog row (2026-10-02).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The cloud-profile boot passes SeamGuard with the real RecordVisibilityCheck — RoutingRecordVisibilityCheck, in every profile, routing account and lead to AccountRecordVisibility and LeadRecordVisibility, which ask the one visibility rule; any other owner type is refused (A-1’s visibility half) |
No deployed environment; offline, SeamGuardIT and MailboxCloudShapePositiveBootIT prove only the wiring a cloud task would boot with |
F5-f spec FR-C1, AC-17; F5-f WI-1.2 plan D1, D7; forge-et-be attachment/service/RoutingRecordVisibilityCheck.java |
On dev the backend starts with no SeamGuard failure in its log; signed in as CRM01, an upload to an account the user reaches is 201 and one to an account they do not reach is 404 |
F38-f | open |
The rest of the substrate — record_share, the one visibility rule and the user directory — runs and is
proved on the local Postgres, with no AWS dependency.
F5-j — Cross-site CSRF cookie (AD #33) (merged to main 2026-09-27: forge-et-be#78, forge-et-fe#59)
Spec FR-12 names three things only a deployed environment can prove. Two are folded into existing
rows: the shared-domain cookie read and echoed across crm.<env> → api.<env> is the §B FS-6
CSRF row (FR-12 (a)), and the real cross-site preflight with Access-Control-Max-Age: 3600 is the
§B FS-6 CORS row (FR-12 ©). The third is this block’s row. Everything else in F5-j runs and is
proved locally with the settings unset or set in a test context.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The deploy workflows set the CSRF cookie settings on both sides, with the same name in each environment: ET_SECURITY_CSRF_COOKIE_DOMAIN and ET_SECURITY_CSRF_COOKIE_NAME on the backend task definition, VITE_CSRF_COOKIE_NAME on the forge-et-fe build (values in §D) |
Neither the task definition nor the deploy workflows exist yet. vite build refuses a cloud build with no frontend name, and the backend refuses a domain with no name — but nothing local can catch XSRF-TOKEN-DEV on one side and XSRF-TOKEN-UAT on the other, and a backend with neither key set starts cleanly as host-only XSRF-TOKEN. A name set with no domain also starts cleanly (FR-5 does not refuse it; lead decision 2026-09-27 at /forge-review-pr, to keep a renamed cookie testable on one origin): deployed, it is a host-only cookie on api.* that the SPA cannot read, so every write fails. Both keys must be set together in every cloud environment |
F5-j spec FR-12 (b), FR-9, FR-5; plan Risks (“A mismatched name between the two sides is invisible until deploy”) | On each environment: the backend task definition carries both keys (plain values, not secrets — AD #22) and the frontend build carries VITE_CSRF_COOKIE_NAME; a response from api.<env> sets the cookie under the name the bundle reads (grep -l <name> dist/assets/*.js), and one write succeeds from the browser |
F38 / FS-1 cloud half | open |
F5-k — Access-model realignment: minimal catalog, no superadmin, first admin by migration
The seeded administrator is the first thing any cloud environment needs, so these rows were
pre-registered from AD #34 and are now fixed by the approved spec
(.forge/specs/F5-k-access-model-realignment-spec.md FR-7, FR-8) and built by F5-k WI-1.1. The
names are: the application property et.security.seed-admin-email, bound from the task-definition
variable ET_SECURITY_SEED_ADMIN_EMAIL by relaxed binding, and handed to the migration
(V28__access_model_realignment.sql) as the Flyway placeholder seed_admin_email by
FlywayPlaceholderConfig. Locally the local profile document carries crm01@empire.lk and the test
suite its own .example value; no cloud value exists in any committed file.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
ET_SECURITY_SEED_ADMIN_EMAIL (→ et.security.seed-admin-email → placeholder seed_admin_email) has a value on dev and UAT: crm01@empire.lk — one of Empire’s test mailboxes, not a person — set as a plain task-definition value (it is not secret) |
No task definitions exist; locally the local profile document and the test suite supply their own values |
AD #34; F5-k spec FR-7, FR-8; environments-and-identity.md §5 (test-accounts row) |
On the first dev deploy (and each uat apply) the migration seeds one IT Admin role holding access.manage and one active user crm01@empire.lk, and V54 then moves that grant to settings.manage (AD #34 refined 2026-10-26); sign in as CRM01 through the real Microsoft sign-in; /me returns settings.manage and no access.manage. Update 2026-11-19 (F38-h WI-5.2): on UAT, the first start of v0.1.0-rc1 seeded IT Admin and crm01@empire.lk from td.uat.json’s plain value; CRM01 signed in through Microsoft on 2026-11-17 and /api/v1/me returned role it_admin with settings.account_tags.manage, settings.contact_tags.manage, settings.email_templates.manage, settings.lead_tags.manage, settings.roles.manage and settings.users.manage — the six Settings page keys since V58 (AD #41), not settings.manage; record .forge/plans/F38-h/F38-h-WI-5.2-record.md R-4. |
F38 | open |
The production value of ET_SECURITY_SEED_ADMIN_EMAIL — the administrator’s email Empire confirms — secrets reference; never in gitforge-et-be .aws/td.prod.json, in git once confirmed) |
No production account or Secrets Manager; the address is not yet confirmed — waiting on Kavindu Anthony | AD #34; F5-k spec FR-8; environments-and-identity.md secrets table and §5 (the production administrator’s email row, agenda item 7) |
Confirm the address with Kavindu; ET_SECURITY_SEED_ADMIN_EMAIL; the first production boot seeds the IT Admin user with that address and that person signs in. Do this before the first production deploy (checklist step 28) |
Tharinda, with Kavindu Anthony (Empire IT) to confirm the address | open |
With ET_SECURITY_SEED_ADMIN_EMAIL unset, or not an email address, the backend refuses to start, naming et.security.seed-admin-email — before any migration runs; the migration’s own check on seed_admin_email is the second line. The loud failure AD #34 relies on, instead of an environment nobody can sign in to |
The refusal only matters on a real deployment with a task definition that might be missing the value; locally EtPropertiesSpec and AccessModelMigrationIT prove both refusals |
AD #34 (accepted trade-off: production cannot start until its admin email is supplied); F5-k spec FR-8, AC-11 | Verify on the first deploy of a fresh database — before the value is set on dev, or on a fresh uat apply: the task fails to start, the log names et.security.seed-admin-email, and ECS marks the deploy failed (the previous task, if any, keeps serving); with the value set, the migration runs |
F38 | open |
On a fresh cloud database the only role is IT Admin and the only user is the seeded admin — none of the laptop sample roles and users from R__local_seed.sql exist |
The sample dataset is local-only by construction; only a real empty cloud database shows it stayed out | AD #34 (the laptop sample dataset stays local-only); F5-k spec FR-16; coding-standards.md §5 (fixtures unreachable outside local/test) |
After the first dev migrate, list the roles and users: exactly IT Admin and crm01@empire.lk. Update 2026-10-22 (F38-g-WI-2.1 R-1): dev’s role and user lists on 2026-10-21, before WI-2.1 added the smoke role and changed CRM05’s role — dev no longer held only the IT Admin role and crm01@empire.lk: teammates had added five roles (All Permissions, All Permissions 2, All Permissions role, test 26, test 44) and three users (CRM02 with IT Admin, CRM03 and CRM05 with All Permissions) for testing; record .forge/plans/F38-g/F38-g-WI-2.1-record.md. After that day dev no longer shows the fresh state. Update 2026-11-19 (F38-h WI-5.2): shown on UAT’s fresh database — on 2026-11-17 the Roles screen listed only IT Admin and the Users screen only crm01@empire.lk, before anything was entered; record .forge/plans/F38-h/F38-h-WI-5.2-record.md R-4 and its screenshots. |
F38 | validated |
BUG-022 on the real consent path — a user whose role lacks email.manage cannot connect a mailbox. Shipped by forge-et-be#122 rather than F5-k WI-2.1; this is the row F5-k FR-22 asks for |
The offline proof (MailboxAuthorizationAccessIT, MailboxConnectFlowIT, LaptopModeIT) runs against a faked token-response client or MockMvc; only a deployed environment’s real Entra exchange shows the refusal holds end to end there |
F5-k spec FR-5, FR-22, AC-5; .forge/bugs.md BUG-022 |
On dev, signed in as CRM01 while IT Admin holds no email.manage: GET /oauth2/authorization/microsoft-mailbox answers 403 FORBIDDEN with no Location, so nothing reaches Microsoft; a callback to /authorized/microsoft-mailbox after the key is removed mid-consent lands on ?connect=failed and writes no mailbox_connection; with the key granted, the consent completes on ?connect=success |
F38 | open |
The lost-access path across sites — the SPA re-reads /me after a 403 from any /api/v1 call (a CSRF refusal included) and on window focus, and shows the no-access state without signing the user out |
Locally every call is same-origin behind the Vite proxy. On a deployed environment the SPA calls api.* cross-site (AD #33), so a 403 sent without CORS headers (a CSRF refusal answered before the CORS filter, for instance) reaches the SPA as a network error, not a 403, and the re-read never starts; the §B FS-6 CSRF and CORS rows test writes that succeed, not refusals that must stay readable. The browser proof of losing a key (AC-24) also uses the local grant stub, which no deployed environment has |
F5-k spec FR-19, AC-23, AC-24; F5-k Wave-2 contract § Envelope & Convention Rules; AD #33; .forge/plans/F5-k/F5-k-WI-3.1-plan.md Decision 13 |
On dev: with the Email screen open as a user whose role holds email.manage, remove the key through F5-d’s role editor, focus the window, and check that the Email entry goes and the no-access state shows, with no sign-out. Then, from that browser’s console, send a POST to an /api/v1 write with a wrong X-XSRF-TOKEN and credentials: 'include', and check that the answer is a readable 403 carrying Access-Control-Allow-Origin for the CRM origin, and that a GET /api/v1/me follows |
F38 | open |
Not cloud-gated, recorded so it is not lost (FR-9, proved locally by AccessModelMigrationIT,
AC-12): a later change to ET_SECURITY_SEED_ADMIN_EMAIL does not fail Flyway’s checksum
validation on restart — Flyway checksums a versioned migration before it replaces placeholders — and
the database migrates cleanly; but the migration has already run, so the seeded user keeps the email
it was seeded with. A production address corrected after the first deploy is therefore changed on the
user record (F5-c’s user edit), not by changing the variable, and needs no new migration.
F11-b — Lead inbox: list, filters, add/edit
One row, found by the wave-1 review of forge-et-be PR #202 (2026-10-05). It replaces the F11-b
“nothing cloud-gated” row.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
ALB access logs and the lead search text — GET /api/v1/leads?q=… carries the search box text in the query string, which can be a person’s name. The backend keeps it out of every log line (bound parameter, INFO caps on DispatcherServlet and FilterChainProxy, LeadLogHygieneIT), but an ALB access log, if enabled, records the full request line |
No ALB exists yet; whether forge-et-iaac turns access logs on, and where they go, is not decided |
F11-b spec NFR-6; forge-et-be PR #202 review (backend-reviewer Nit 8) |
When the ALB is built: decide with the lead whether access logs are on. If they are, either accept that the query string is logged (record the decision, its retention and who can read the bucket) or exclude it; then check one ?q= request on dev against the log |
F38 | open |
The rest of F11-b — the lead list, the tag list, the seed fixtures and the volume fixture — runs and is proved on the local Postgres with the stub login, with no AWS, Entra, Bedrock or SES call (spec FR-H5).
F11-c — Lead research job
The eight items F11-c spec FR-H3 names (AC-31), added by F11-c-WI-1.3. Every backend AC runs on the search stub, the AI
stub and LocalStack (forge-et-be LeadResearch*IT, and the Search*IT specs of the search seam); these are what only a
cloud environment can prove. An environment without the queue starts with no lead-research listener (the
nullable-queue pattern, LeadResearchCloudShapeIT): a created lead logs Lead Research Not Enqueued at WARN, the
hourly sweep logs Lead Research Sweep Skipped, the lead view’s Research panel reads Researching this lead… and then
Still researching — check back shortly., and every other lead screen works as before. An environment without the Exa
key starts too (spec Rev 1): every search answers NOT_CONFIGURED, the panel reads Research isn’t available right
now, and the sweep queues nothing. The DevOps request is in .forge/owed.md (Shyamin Ayesh), and the task-definition
lines in a follow-up forge-et-be PR (Dinil).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
1. Dev’s Exa key in dev’s Secrets Manager, the task definition’s secrets reference to ET_SEARCH_EXA_APIKEY, and a budget set on the dev key in Exa’s dashboard |
The key exists only in Altrium’s Exa account; locally and in tests the stub answers and needs no key. ExaSearchProvider without a key logs one startup ERROR line naming the variable and every search is NOT_CONFIGURED |
F11-c spec FR-Q2, FR-H3 item 1, NFR-6; lead-research-source-options.md §13; AD #22 |
Store the key in Secrets Manager (DevOps, forge-et-iaac), reference it from .aws/td.dev.json, set the Exa dashboard budget; restart: the startup log has no ET_SEARCH_EXA_APIKEY ERROR line and a created lead’s search reaches Exa (a search_call row with outcome SUCCESS and http_status 200) |
Shyamin Ayesh (key, secret, reference); Dinil (Exa budget) | open — the Exa budget only. 2026-10-23, dev: the secret (forge-et-iaac#8) holds the key and forge-et-be#307 references it; the backend:62 task started at 11:22 UTC with no ET_SEARCH_EXA_APIKEY ERROR line, and lead 12’s search logged Search Call Completed - [searchCallId=1, purpose=COMPANY, outcome=SUCCESS] [resultCount=5, httpStatus=200] |
2. The lead-research queue and its dead-letter queue in forge-et-iaac (redrive maxReceiveCount 3; visibility timeout 270 s — one run can take up to about 181 s at the 70 s AI pacing cap, since F36-c’s plan-research searcher shares the search rate limiter, so a shorter timeout lets SQS hand the same message to a second consumer, which pays for a second search and AI call; the listener asks for 270 s on every receive (LeadResearchJobListener.VISIBILITY_SECONDS, messageVisibilitySeconds), which holds whatever the queue’s default is, so the queue’s own 270 s is the backstop and keeps the estate consistent; the local queue gets it from QueueProvisioner.LEAD_RESEARCH_VISIBILITY_TIMEOUT, and forge-et-be docs/search/README.md § How long one research run can take holds the arithmetic), the backend task role’s send, receive, delete and get-attributes rights on the queue, and ET_MESSAGING_LEADRESEARCHQUEUE on the task definition |
LeadResearchJobListener exists only when the key is set and LeadResearchEnqueuer sends nothing without it, so a task definition without it boots healthy and researches nothing; the queue pair is QueueProvisioner’s against LocalStack only. The spec puts ET_MESSAGING_LEADRESEARCHQUEUE / _DLQ in the task definition; only the queue key is needed there — ET_MESSAGING_LEADRESEARCHDLQ is read only by the local provisioner and stays unset in cloud, where the DLQ is Terraform’s (F11-c WI-1.3 plan Decision 7, as for F11-d) |
F11-c spec FR-J1, FR-J4, FR-H3 item 2; forge-et-be EtProperties.Messaging#leadResearchQueue, LeadResearchJobListener.java, LeadResearchCloudShapeIT |
Terraform the queue, DLQ and redrive; grant the task role; then set the key (a queue name set before the queue exists fails the dev deploy); restart: a create logs no Lead Research Not Enqueued, and a lead created on dev gets a lead_research row |
Shyamin Ayesh (queue, DLQ, grants); Dinil (the key in .aws/td.dev.json) |
validated (2026-10-23, dev) — forge-et-iaac#8 and forge-et-be#307; lead 12, created at 11:39:41 UTC, logged Lead Research Enqueued and then Lead Research Completed - [leadId=12, state=done, outcome=SUCCESS, stage=ai, … written=true] |
3. The EventBridge schedule lead-research-sweep at rate(1 hour) targeting the scheduler queue, with the payload {"jobKind": "lead-research-sweep", "idempotencyKey": "lead-research-sweep#<aws.scheduler.scheduled-time>"} |
No schedule exists; the sweep is proved by calling its handler directly (LeadResearchSweepIT) |
F11-c spec FR-J5, FR-H3 item 3; forge-et-be docs/messaging/scheduler-cloud-contract.md § Lead research sweep |
Create the schedule in forge-et-iaac per the scheduler contract; after the next hour, the task log has one Lead Research Sweep Completed line naming the firing’s key (or Lead Research Sweep Skipped with the reason while the key or the queue is missing) |
Shyamin Ayesh | validated (2026-10-23, dev) — the schedule exists (forge-et-iaac#8: group et-dev, rate(1 hour), ENABLED); its first firing after the deploy logged Lead Research Sweep Completed - [idempotencyKey=lead-research-sweep#2026-10-23T12:08:02Z, selected=9, queued=9, backlog=0] at 12:08:07 UTC, and all nine queued leads logged Lead Research Completed by 12:08:23 UTC |
4. A live research run on dev — a lead for a real, well-known public company reaches done with working source links, a lead for an invented company reaches nothing_found, and the time from intake to the row being written is recorded against NFR-5’s 2 minutes |
Every test runs on the search stub and the AI stub. Limit: the 429 and 402 paths are proved only against a mocked HTTP server (ExaSearchProvider’s specs, F11-c WI-1.2), because forcing a rate limit or spent credit on a live account is not acceptable |
F11-c spec FR-H3 item 4, NFR-5, AC-1 | With items 1–3 and 6 in place: create the two leads through the form; read GET /api/v1/leads/{id}/research; open each source link; record the two intake-to-row times; record that the 429 / 402 paths were not forced Update 2026-11-21 (F38-f WI-2.1): the source-link check is done — a new Twinings lead (142, twinings.co.uk) reached done 5.7 s after being queued (Lead Research Enqueued 21:20:13.289 to Lead Research Completed 21:20:19.009 UTC), and each of its three source links (twinings.co.uk/, the contact page, the 216 Strand flagship page) was opened and loads with text that supports the facts citing it; two invented companies reached nothing_found; evidence: §B F38-f “F38-f evidence — AC-2 and AC-3”. |
Dinil | open — the source-link check is left. 2026-10-23, dev: the well-known company (lead 12, Twinings, twinings.co.uk) reached done 6.6 s after intake (Lead Created 11:39:41.59 to Lead Research Completed 11:39:48.16 UTC; the Exa call took 2.7 s), against NFR-5’s 2 minutes. The invented-company case was met by dev’s older test leads, whose company names are made up: the 12:08 UTC sweep researched leads 2, 3, 4, 5, 6, 8 and 11 to nothing_found (outcome=SUCCESS, stage=ai) and leads 7 and 10, which have no company name and no website, to no_input with no search. Those runs came from the sweep, not from intake, so no intake-to-row time was measured for that case; each took about 3 s. The 429 / 402 paths were not forced |
| 5. The FR-A8 evaluation set on dev, on Haiku 4.5 | LeadResearchEvaluationIT is the stub run only; LeadResearchEvaluationCaseRunner takes any AIProvider, and LiveEvaluation runs it on Bedrock |
F11-c spec FR-A8, FR-H3 item 5, AC-22; forge-et-be docs/ai/README.md § Evaluation |
Ask the lead first (a Bedrock call), then AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=lead-research -PconfirmBedrockCalls=true -Pmodel=us.anthropic.claude-haiku-4-5-20251001-v1:0; the run must keep no finding from another firm’s result, cite no result it did not mark yes, and keep no forbidden fact; record the report and its cost |
Dinil | open |
6. The dev model override while Sonnet 5.5 is refused — ET_AI_FEATURES_LEADRESEARCH_MODEL=us.anthropic.claude-haiku-4-5-20251001-v1:0 in forge-et-be .aws/td.dev.json (and on a UAT instance applied before the quota is raised); §D carries the row; removed once item 7 passes. 26 Oct: in every environment, production included, until the lead moves research off Haiku 4.5 on item 7’s evaluation (ai-provider-strategy.md §5 row 10, §4d). Update 2026-11-04 (F43-WI-4.2): since F43’s wave 4 the override names Haiku 4.5’s Global profile, global.anthropic.claude-haiku-4-5-20251001-v1:0, so lead research shares the Global profile’s request quota with lead triage only (F43 FR-R3; §D); its reason, Sonnet 5.5’s refusal, is unchanged. |
application.yml defaults the feature to Sonnet 5.5’s Global profile (AD #19), which the account refuses, so without the override every research call on dev fails unavailable |
F11-c spec FR-A1, FR-H3 item 6; AiPropertiesBindingSpec (AC-34: the variable binds to the Haiku 4.5 id) |
A research run on dev records an ai_invocation row with the Haiku 4.5 profile, feature_key lead-research and SUCCESS; tell the DevOps team the entry is there so a direct push to the file keeps it (.forge/owed.md) |
Dinil | open — set on dev by forge-et-be#307 (2026-10-23); lead 12’s run succeeded at its AI stage (stage=ai, aiInvocationId=34), which the refused Sonnet 5.5 default could not have done, but the ai_invocation row itself is not yet read |
| 7. Sonnet 5.5 on dev once its quota is raised — the Global profile answers from the dev task role, the task role’s policy allows the profile’s Regions, the FR-A8 evaluation passes on Sonnet 5.5 (OQ-15 was settled on 2026-11-04, so there is nothing to record about it) | The account refuses Sonnet 5.5 (quota 0, §C-5) | F11-c spec FR-H3 item 7, decision 6; §C-5 | Once the quota is raised: run item 5 on global.anthropic.claude-sonnet-5-5, then remove item 6’s override — before production runs the default |
Dinil | open |
8. Production: Empire’s own Exa account and key, its limits set for production (ET_RESEARCH_MONTHLYSEARCHLIMIT, ET_RESEARCH_BACKLOGMONTHLYSEARCHLIMIT), and Exa’s pricing and terms re-read before go-live |
Production does not exist; dev uses Altrium’s account and its free credit | F11-c spec FR-H3 item 8, Context; lead-research-source-options.md §13; .forge/owed.md (Empire’s Exa account) |
Before the production deploy: Empire opens its Exa account; its key goes into production’s Secrets Manager; the limits are set from the production free credit; Exa’s pricing page and terms are re-read and any change raised with the lead | Tharinda (with Empire); Shyamin Ayesh (secret) | open |
F11-i — Person-search switch on the Settings page
The two items F11-i spec FR-H3 names (AC-18), added by F11-i-WI-1.1. Every backend AC runs on the search stub and
LocalStack (forge-et-be LeadResearchSettingControllerIT, LeadResearchSwitchIT, LeadResearchPersonSearchIT);
these are what only a cloud environment can prove. The switch is one row, lead_research_setting, which
V78__lead_research_setting.sql inserts on in every environment, and its page’s key
settings.lead_research.manage ships held by no role (AD #30), so a new environment researches with the person search
on (spec decision 8), and only someone holding Roles can grant the key.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
1. The switch on dev, and the person search following it — the row exists with person_search_enabled = true after the deploy; a lead with a company and a contact name created on dev writes a PERSON row in search_call beside its COMPANY row; after the switch is turned off on dev’s Settings → Lead research, the next such lead writes none |
Locally and in tests the search stub answers; only dev reaches Exa with the person query, and dev’s row is written by its own migration run | F11-i spec FR-H3 item 1, FR-S1, FR-P1, AC-8 | After the deploy: SELECT * FROM lead_research_setting shows one row, on, version 0; grant the key to a dev role (.forge/owed.md’s grant row); create a lead with a company and a contact name, read its search_call rows (COMPANY and PERSON); turn the switch off on the page; create a second lead and read its rows (COMPANY alone); turn it back on if dev should keep it on |
Dinil | open |
2. A task definition that still sets ET_RESEARCH_PERSONSEARCH_ENABLED starts normally and the value is ignored, and the variable is removed from forge-et-be .aws/ task definitions where present |
ResearchProperties no longer declares the key (F11-i spec FR-P2); SearchPropertiesBindingSpec and LeadResearchPersonSearchIT prove it binds and is ignored, but only a deploy proves a task definition that carries it — such as the one Biseni’s unmerged feature/F11-c-person-search-dev would set — still starts |
F11-i spec FR-H3 item 2, FR-P2, AC-10 | git -C forge-et-be grep -n PERSONSEARCH origin/main -- .aws returns nothing (remove the line by PR if it does); on the dev deploy after F11-i merges, the task starts healthy whether or not the variable is set; close feature/F11-c-person-search-dev as superseded |
Dinil | open |
F11-d — AI summary and spam/duplicate detection
The seven rows F11-d spec FR-H3 names (AC-32), added by F11-d-WI-1.1. Every backend AC runs on the stub and LocalStack
(forge-et-be LeadTriage*IT); these are what only a cloud environment can prove. Since 2026-10-20 dev has the
lead-triage queue, its schedule and the task-definition key (rows 1 and 2). An environment without them starts with
no lead-triage queue (the nullable-queue pattern): a created lead logs Lead Triage Not Enqueued at WARN, the hourly
sweep logs Lead Triage Sweep Skipped, the lead view’s panel reads Preparing the summary… and then Still preparing —
check back shortly., and every other lead screen works as before. What is left on dev is in .forge/owed.md (Dinil).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
1. The lead-triage queue and its dead-letter queue in forge-et-iaac (redrive maxReceiveCount 3, the et.messaging.* default; the queue’s own visibility timeout is overridden on every receive by et.lead-triage.visibility-timeout since F43, so it sets no floor), the backend task role’s send, receive, delete and get-attributes rights on the queue, and ET_MESSAGING_LEADTRIAGEQUEUE on the task definition |
LeadTriageJobListener is @ConditionalOnProperty on the key and LeadTriageEnqueuer sends nothing without it, so a task definition without it boots healthy and triages nothing; the queue pair is QueueProvisioner’s against LocalStack only. The DLQ key (ET_MESSAGING_LEADTRIAGEDLQ) is read only by the local provisioner; in cloud the DLQ is Terraform’s and the key stays unset |
F11-d spec FR-J1, FR-H3 item 1; F11-d WI-1.1 plan Decision 19; forge-et-be EtProperties.Messaging#leadTriageQueue, LeadTriageJobListener.java, LeadTriageCloudShapeIT |
Terraform the queue, DLQ and redrive; grant the task role; set the key; restart: the startup log has no Lead Triage Not Enqueued on a create, and a lead created on dev gets a lead_triage row. Written 2026-10-20: the queue, DLQ and grants in forge-et-iaac#4 (stacked on #3), the key in draft forge-et-be#283 (merges after #4 is applied). Update 2026-10-20 (Claude, for Tharinda): #3 and #4 merged into forge-et-iaac dev and applied (run 39475850266, 18:35 UTC); read back in the account: et-dev-lead-triage with VisibilityTimeout 30, redrive to et-dev-lead-triage-dlq at maxReceiveCount 3, SSE on; the task role’s sqs and sqs-send policies name it. forge-et-be#283 merged (e9a18c0) and deployed: backend:44 carries ET_MESSAGING_LEADTRIAGEQUEUE=et-dev-lead-triage, and its task started at 19:38 UTC with no lead-triage WARN. Left: a lead created on dev gets a lead_triage row (needs a signed-in user) Update 2026-11-19 (Dinil): a lead created on dev got its AI summary; the create check is done. Update 2026-11-19: the Terraform queue’s own visibility timeout is now 150 s, matching et.lead-triage.visibility-timeout — forge-et-iaac#24, applied on dev by dev.yaml run 44488259851 (visibility_timeout_seconds 30 → 150, in place). |
Dinil (a lead created on dev) | validated 2026-11-19 |
2. The EventBridge schedule lead-triage-sweep at rate(1 hour) targeting the scheduler queue, with the payload {"jobKind": "lead-triage-sweep", "idempotencyKey": "lead-triage-sweep#<aws.scheduler.scheduled-time>"} |
No schedule exists in any account; the sweep is proved by calling its handler directly (LeadTriageSweepIT) |
F11-d spec FR-J7, FR-H3 item 2; forge-et-be docs/messaging/scheduler-cloud-contract.md § Lead triage sweep |
Create the schedule in forge-et-iaac per the scheduler contract; after the next hour, the task log has one Lead Triage Sweep Completed line naming the firing’s key. Written 2026-10-20 in forge-et-iaac#4 (base/scheduler/locals.tf). Update 2026-10-20 (Claude, for Tharinda): applied; get-schedule shows lead-triage-sweep ENABLED, rate(1 hour), target et-dev-scheduler, role et-dev-scheduler, and the payload above. Its 19:34:58 UTC firing reached a task without the key and logged Lead Triage Sweep Skipped; the 20:34:58 UTC firing, on backend:44, logged Lead Triage Sweep Completed - [idempotencyKey=lead-triage-sweep#2026-10-20T20:34:58Z, aiEnabled=true, selected=0, queued=0]; the scheduler DLQ is empty |
DevOps (Shyamin Ayesh) to apply, raised by Dinil | validated (2026-10-20, dev) |
3. The ET_AI_FEATURES_LEADTRIAGE_MODEL and _ENABLED rows in §D, and the price row the model needs |
application.yml sets the model (the Haiku 4.5 us. profile, whose price row exists), so a task definition sets neither; the names are proved by AiPropertiesBindingSpec, not on a running task Update 2026-11-04 (F43-WI-4.2): since F43’s wave 4 dev’s task definition sets ET_AI_FEATURES_LEADTRIAGE_MODEL to Haiku 4.5’s Global profile, and UAT and production take it from §D (F43 FR-R3); application.yml’s default stays the us. profile. |
F11-d spec FR-A1, FR-H3 item 3, AC-9 | Leave both unset on dev; if a task definition ever sets the model, set a matching price row (§D’s list rule) and confirm the running task reads it | F38 | open |
4. A real Converse call for lead-triage from the dev task role on Haiku 4.5, through the extract operation, its cost read from ai_invocation.cost_usd |
Every test runs on the stub; dev Bedrock serves Haiku 4.5 (§C-5) but no lead has been triaged there | F11-d spec FR-H3 item 4; F12 block row 1 (the same profile and task-role policy) | Create a lead on dev with an inquiry; its ai_invocation row shows bedrock, the profile id, feature_key lead-triage, non-zero tokens and SUCCESS; record the cost per call Update 2026-11-21 (F38-f WI-2.1): eleven leads created through the app on dev were triaged from the task role on Haiku 4.5’s Global profile, every call SUCCESS with non-zero tokens, 1 070–1 531 ms, $0.002374–$0.002766 a call (total $0.028382), each job done within 1.6 s of being queued; the three evaluation cases met their spam flag and summary terms; evidence: §B F38-f “F38-f evidence — AC-2 and AC-3”. |
Dinil | open |
| 5. The FR-A7 evaluation run on the real model | LeadTriageEvaluationIT is the stub run only; LeadTriageEvaluationCaseRunner takes any AIProvider |
F11-d spec FR-A7, FR-H3 item 5, AC-14; forge-et-be docs/ai/README.md § Evaluation |
With the lead’s OK, run ./gradlew liveEvaluation -Pfeature=lead-triage -PconfirmBedrockCalls=true (the F12 block’s row 4 runner, forge-et-be#289); the run must mark no genuine inquiry as spam, catch at least 90% of the spam cases and return no summary with a forbidden word; record the report. Run it on prompt lead-triage/v2 (draft forge-et-be#281, whose merge waits on this run; its new case instruction-shaped-email-domain must be caught). First run, 2026-10-21 05:49 UTC, v2 on Haiku 4.5, the lead’s OK relayed by Dinil: no result — Bedrock throttled the unpaced run (the dev account allows 10 Haiku 4.5 requests a minute), the circuit opened and 10 of 25 cases did not run, at a cost of US$0.04. Of the 15 that ran, no genuine inquiry was marked spam and no summary had a forbidden word. Re-run once forge-et-be#290 (pacing) merges; the quota itself is in owed.md. Paced runs, 2026-10-21, Haiku 4.5, all with the lead’s OK relayed by Dinil (reports on forge-et-be#281): v2 (07:02 UTC, US$0.066, 25 of 25 calls succeeded) caught all 11 spam cases, the instruction-shaped domain included, but marked migrated-no-inquiry spam, so the prompt became v3 (F11-d spec Rev 2). v3 run A (08:02 UTC, US$0.060): no genuine inquiry marked spam, 11 of 11 spam caught, no forbidden word; 3 calls failed with Bedrock ServiceUnavailableException. v3 run B (08:13 UTC, US$0.063): no genuine inquiry marked spam, 9 of 11 spam caught, no forbidden word; the 2 not caught are the 2 failed calls (ServiceUnavailableException, ApiCallTimeoutException). The review of forge-et-be#281 then found v3’s no-inquiry rule left out the email domain; v3 was reworded and the set gained two spam leads with no inquiry text (27 cases, 13 spam). Final v3 (08:42 UTC, US$0.074, 27 of 27 calls succeeded, p95 2.2 s): meets the bar — no genuine inquiry marked spam, 12 of 13 spam caught (92%), no forbidden word. The one missed, no-inquiry-instruction-shaped-domain, is a known limit accepted by Dinil and logged in owed.md for a later version. |
Dinil | validated |
| 6. A lead created on dev shows its summary within 60 seconds at p95 with an idle queue | Timing exists only with the real queue, task and model; locally the stub answers within 10 seconds | F11-d spec NFR-1, FR-H3 item 6 | Create ten leads on dev through the form; time from create to the triage GET answering done; the 95th percentile is under 60 s |
Dinil | open |
7. A dead-letter alarm on the lead-triage DLQ, alongside the existing ones |
The alarms are FS-5’s cloud half; a poison message reaching the DLQ is the only signal that triage stopped for a lead | F11-d spec FR-H3 item 7; F38 | Wire the alarm on the DLQ’s ApproximateNumberOfMessagesVisible; send one malformed message to the queue on dev and see the SNS email after the third receive. Update 2026-10-23 (F38-e spec decision 1): not built in dev; UAT and production decide, and the trip test moves with it Update 2026-10-26 (F38-h WI-3.1): F38-h’s uat overlay is built with no alarm (the lead, 2026-10-25; F38-h Decomposition Plan § Notes); UAT’s set comes as a later F38-h work item once the .forge/owed.md row “Decide the UAT and production alarm sets” is decided. The row stays open. Update 2026-11-08 (the review of forge-et-iaac#18): superseded by F38-i spec decision 26, which settled the alarm sets — no dead-letter alarm in any environment, no alarm in UAT, and in production an alarm on the age of the oldest message on each main queue (F38-i WI-5.1), whose list must name the lead-triage queue. |
F38-h, F38-i | closed 2026-11-08 — superseded by F38-i decision 26 |
F11-g — AI lead ranking and hot/cold
The seven rows F11-g spec FR-H3 names (AC-28), and one for the SQL fingerprint’s locale, added by F11-g-WI-1.1. Every
backend AC runs on the stub and LocalStack (forge-et-be LeadRanking*IT, LeadEngagementIT); these are what only a
cloud environment can prove. Until the queue, its key and the schedule exist on an environment, it starts with no
lead-ranking queue (the nullable-queue pattern, LeadRankingCloudShapeIT): a research write logs Lead Ranking Not Enqueued at WARN, the hourly sweep logs Lead Ranking Sweep Skipped, the lead view’s Priority panel reads
Ranking… and then Still working — check back shortly., the hot / cold status shows from the merge (it is computed on
read, never queued), and every other lead screen works as before. The DevOps request is in .forge/owed.md (Dinil).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
1. The lead-ranking queue and its dead-letter queue in forge-et-iaac (redrive maxReceiveCount 3, the et.messaging.* default; the queue’s own visibility timeout is overridden on every receive by et.lead-ranking.visibility-timeout, 150 s), the backend task role’s send, receive, delete and get-attributes rights on it, and ET_MESSAGING_LEADRANKINGQUEUE on the task definition |
LeadRankingJobListener is @ConditionalOnExpression on the key and LeadRankingEnqueuer sends nothing without it, so a task definition without it boots healthy and ranks nothing; the queue pair is QueueProvisioner’s against LocalStack only. The DLQ key (ET_MESSAGING_LEADRANKINGDLQ) is read only by the local provisioner; in cloud the DLQ is Terraform’s and the key stays unset |
F11-g spec FR-J1, FR-H3 item 1; F11-g WI-1.1 plan Decision 28; forge-et-be EtProperties.Messaging#leadRankingQueue, LeadRankingJobListener.java, LeadRankingCloudShapeIT |
Terraform the queue, DLQ and redrive; grant the task role; set the key; restart: a research write on dev logs Lead Ranking Enqueued, not Not Enqueued, and the lead gets a lead_ranking row. Met 2026-11-09: forge-et-iaac#18 applied on dev (the queue et-dev-lead-ranking, its DLQ and the task role’s rights), and forge-et-be#375 deployed as backend:120 with ET_MESSAGING_LEADRANKINGQUEUE=et-dev-lead-ranking (read with aws ecs describe-task-definition). The first sweep (row 2) sent 77 leads through the queue: 77 Lead Ranking Enqueued and 77 Lead Ranking Completed lines between 09:15 and 09:25 UTC, each written=true. The enqueue came from the sweep, not a research write; both use the same LeadRankingEnqueuer. |
Dinil (with Shyamin Ayesh, .forge/owed.md) |
closed 2026-11-09 — the queue carries rankings on dev |
2. The EventBridge schedule lead-ranking-sweep at rate(1 hour) targeting the scheduler queue, payload {"jobKind": "lead-ranking-sweep", "idempotencyKey": "lead-ranking-sweep#<aws.scheduler.scheduled-time>"} |
The sweep is a scheduler jobKind (AD #12), never @Scheduled; without the schedule a failed ranking is never retried and leads created before F11-g shipped stay unranked |
F11-g spec FR-J6, FR-H3 item 2; forge-et-be docs/messaging/scheduler-cloud-contract.md § Lead ranking sweep; LeadRankingSweepJobHandler |
Create the schedule in forge-et-iaac; after the next hour the backend log has one Lead Ranking Sweep Completed line naming the firing’s key, with currentInJava=0. Met 2026-11-09: the schedule was created by forge-et-iaac#18’s apply at 08:15 UTC, and its first firing logged Lead Ranking Sweep Completed - [idempotencyKey=lead-ranking-sweep#2026-11-09T09:15:26Z, aiEnabled=true, selected=77, queued=77, currentInJava=0]. |
Dinil (with Shyamin Ayesh) | closed 2026-11-09 — the first hourly firing ran |
3. The ET_AI_FEATURES_LEADRANKING_MODEL and _ENABLED rows in §D |
application.yml sets the model (Haiku 4.5’s us. profile, whose price row exists); dev’s task definition puts background work on Haiku 4.5’s Global profile (F43 FR-R3), which only a deployed task shows. The names are proved by AiPropertiesBindingSpec, not on a running task |
F11-g spec FR-A1, FR-H3 item 3, AC-11 | Set ET_AI_FEATURES_LEADRANKING_MODEL=global.anthropic.claude-haiku-4-5-20251001-v1:0 on dev’s task definition, as for lead-triage and lead-research; read the running revision back (aws ecs describe-task-definition, a read) and one AI Call Completed line for lead-ranking naming the Global profile. Met 2026-11-09: backend:120 sets ET_AI_FEATURES_LEADRANKING_MODEL=global.anthropic.claude-haiku-4-5-20251001-v1:0 (forge-et-be#375, read with aws ecs describe-task-definition; _ENABLED is unset, so the default true holds, and the sweep logged aiEnabled=true), and every one of the 76 AI Call Completed lines for lead-ranking names provider=bedrock, model=global.anthropic.claude-haiku-4-5-20251001-v1:0. |
Dinil (with Shyamin Ayesh) | closed 2026-11-09 — the Global profile runs on dev |
4. A real Converse call for lead-ranking from the dev task role, and its cost per call |
The stub records zero tokens and cost; only Bedrock gives the real ai_invocation.cost_usd |
F11-g spec FR-H3 item 4; NFR-6 | After rows 1 and 3, create a lead on dev, wait for its ranking, and read its ai_invocation row (feature_key = 'lead-ranking'): outcome SUCCESS, tokens and cost_usd recorded; note the cost here against .forge/client/sow-cost-annexe.md. Met 2026-11-09, from the first sweep’s 77 leads rather than a new lead (read from the backend’s log lines, which carry the ai_invocation row’s id, tokens and cost): 73 rankings SUCCESS, 1 NO_INPUT with no AI call, 3 INVALID_OUTPUT; 76 calls (attempts=1 71, attempts=2 5); about 1,700 input and 180 output tokens a call; cost USD 0.0028 a call on average (most 0.0049), USD 0.22 for all 77; latency median 2.2 s, longest 56.9 s, inside the 60 s pacing wait. The cost annexe has no line of its own for ranking. To watch: 8 of the 81 Bedrock attempts were INVALID_OUTPUT (NO_TOOL_USE stopReason=end_turn: Haiku 4.5 answered without calling the ranking tool, which toolChoice auto allows, BUG-248), and 3 leads still failed on the retry; the hourly sweep ranks them again. The rate is in .forge/owed.md, “Watch lead ranking’s no-tool replies on dev”. |
Dinil | closed 2026-11-09 — real calls and their cost recorded |
| 5. The live evaluation run on dev, meeting F11-g FR-A8’s bar | The stub answers P2 to every case, so the stub run proves the runner and scorer only (LeadRankingEvaluationIT) |
F11-g spec FR-A8, FR-H3 item 5, AC-15; forge-et-be docs/ai/README.md § Evaluation |
AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=lead-ranking -PconfirmBedrockCalls=true (costs money — ask the lead first); record the report: no case two levels off, at least 75% exact, no research-group member ranked lower without research, no forbidden word. (Rev 6) From F36-c’s wave 3 on dev, this command runs the lead-ranking-v2 set, so its Haiku 4.5 run is recorded once, under §B F36-c item (k), and that record covers this row’s Haiku 4.5 run. Before wave 3 is on dev, this row stands as written |
Dinil (the lead approves the run) | open |
| 6. A lead created on dev is ranked within 60 seconds of its research completing, at p95 | Locally the stub ranks within 10 s (NFR-1, the WI-1.1 smoke); dev’s queue, pacing and Bedrock latency are only measurable there | F11-g spec NFR-1, FR-H3 item 6 | Create five leads on dev; for each read lead_research.researched_at and lead_ranking.ranked_at; record the five gaps and their maximum here |
Dinil | open |
7. A dead-letter alarm on the lead-ranking DLQ, alongside the existing ones (F38) |
A message that fails three times lands in the DLQ silently without it | F11-g spec FR-H3 item 7; F38’s alarm topic | Add the alarm in forge-et-iaac with the other DLQ alarms; confirm it is in OK. Update 2026-11-08 (the review of forge-et-iaac#18): superseded by F38-i spec decision 26 — no dead-letter alarm in any environment (dev has none, F38-e decision 1), no alarm in UAT, and in production an alarm on the age of the oldest message on each main queue (F38-i WI-5.1), whose list must name the lead-ranking queue. A failed ranking is logged at ERROR by the job framework, which production’s ERROR log alarm reports. |
Dinil (with Shyamin Ayesh) | closed 2026-11-08 — superseded by F38-i decision 26 |
| 8. The SQL fingerprint’s locale on dev’s database | LeadRankingSql.websiteDomain uses lower() and \w, which follow the database’s lc_ctype; LeadRankingFingerprintIT holds the SQL and Java copies equal only on the Testcontainers image (postgres:18-alpine). A different lc_ctype could make the list show a ranked lead as unranked |
F11-g WI-1.1 plan Risks (the two fingerprint copies) | Read SHOW lc_ctype on dev’s database and on the test image and record both; or, after row 2, confirm every Lead Ranking Sweep Completed line reports currentInJava=0 and no Lead Ranking Sweep Fingerprints Disagree WARN appears for a week. 2026-11-09: the first sweep reported currentInJava=0 with no such WARN; the week runs to 2026-11-16. |
Dinil | open |
F11-h — AI-drafted follow-up with Approve & send
The nine items F11-h spec FR-H3 names (AC-33, WI-1.1’s clause), added by F11-h-WI-1.1. Every F11-h AC is asserted
against the AIProvider stub, F12’s fake Bedrock SDK client and F20-a’s local mailbox stand-in; these are what only
real Bedrock, a real mailbox and a deployed environment can prove. The draft runs on the same seam, runner and record
as F12’s extraction, so F12’s rows 2, 6 and 7 (the cloud boot, the alarms, the overrides) cover it too.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
1. A real Converse call for email-draft from the dev task role — on Haiku 4.5 through dev’s override now (item 7), on Sonnet 5.5 once its quota is raised — answering inside the 9 s deadline at p95 over the evaluation set, with the cost per call read from ai_invocation.cost_usd and compared with proposal §A.3.2’s 1.5 US cents |
Every test runs on the stub or the fake SDK client; the account refuses Sonnet 5.5 (quota 0, §C-5) | F11-h spec FR-H3 item 1, FR-B1, NFR-1, decision 3; forge-et-be ai/service/BedrockAiProvider.java (draft) |
On dev, draft from a lead; the ai_invocation row shows bedrock, the configured profile id, non-zero tokens and SUCCESS; read p95 from et.ai.latency{feature=email-draft} over item 2’s run and the cost from cost_usd; if Sonnet 5.5 cannot answer inside the deadline, reconsider streaming (spec decision 3) Update 2026-11-21 (F38-f WI-2.1): the Haiku 4.5 half — three drafts on dev from leads holding the evaluation set’s lead-reply-full, lead-price-request and lead-injection, through dev’s us. Haiku 4.5 override, each SUCCESS in 1 161–1 337 ms at $0.003186–$0.003389 a call (about 0.33 US cents, against §A.3.2’s 1.5); no draft held a forbidden term or a price, and none was approved or sent; the Sonnet 5.5 half and the p95 over item 2’s run are left; evidence: §B F38-f “F38-f evidence — AC-2 and AC-3”. |
Dinil (F11-h); when dev Bedrock answers (F38) | open |
| 2. The FR-B6 evaluation run on dev — on Haiku 4.5 now, and on Haiku 5.5 and Sonnet 5.5 once their quotas are raised (§C-5) — with the per-case results, a person’s tone and fit rating (at least 4 of 5 on average), and the model kept for UAT | DraftEvaluationCaseRunner takes any AIProvider, but only the stub run (EmailDraftEvaluationIT) runs in the suite; a real run needs Bedrock in the dev account |
F11-h spec FR-B6, FR-H3 item 2, decision 2; forge-et-be docs/ai/README.md § Evaluation |
With the lead’s OK, run ./gradlew liveEvaluation -Pfeature=email-draft -PconfirmBedrockCalls=true -Pmodel=<profile id> (F12 row 4’s runner, forge-et-be#289) on each model, record which pass every check and the rating; a move of drafts to a Haiku model needs the lead’s sign-off on AD #19 and a proposal post-issue note (.forge/owed.md) |
Dinil, with Tharinda | open |
3. The manual paths on dev — ET_AI_FEATURES_EMAILDRAFT_ENABLED=false answers 503 AI_DRAFT_UNAVAILABLE and the panel and the compose button show the manual message; a wrong model identifier fails as MODEL_UNAVAILABLE to the same path |
Proved on the stub (AiDraftDisabledIT, AiDraftControllerIT) |
F11-h spec FR-H3 item 3, NFR-6, AC-5, AC-30 | Set the switch off and restart: Draft follow-up shows the unavailable message, Approve & send still sends and approves; set the model to a non-existent id (with a matching price list) and restart: the row ends MODEL_UNAVAILABLE Update 2026-11-21 (F38-f WI-2.2): with ET_AI_FEATURES_EMAILDRAFT_ENABLED=false deployed on dev (backend:179), Draft follow-up showed “AI drafting is unavailable right now. You can write the email yourself.” and the call was recorded outcome=DISABLED with no Bedrock attempt; Compose Email still opened the compose form; once main was deployed again the draft answered SUCCESS. This run exercised Draft follow-up only, not the compose button’s AI draft or Approve & send, and made no wrong-model-id check; evidence: §B F38-f “F38-f evidence — AC-4”. |
F38 | open — the enabled: false half of Draft follow-up met 2026-11-21 (F38-f WI-2.2); the compose button, Approve & send and the wrong-model-id half open |
4. A draft, edited and sent with Approve & send from dev as CRM01 to an external mailbox — the email arrives, the lead is approved, and the email_send row has ai_drafted true and edited_by_human matching the edit |
The stand-in mailbox sends nothing; no Bedrock locally | F11-h spec FR-H3 item 4, AC-23 | On dev, draft on a new lead, edit, send through Approve & send; check the external inbox, the lead’s status and the email_send row’s two flags |
Dinil (F11-h) | open |
5. Sonnet 5.5 on dev once its quota is raised — the Global profile global.anthropic.claude-sonnet-5-5 answers, its et.ai.prices row ($2 / $10 per million tokens) matches Bedrock’s price that day, and the task role’s policy allows the profile’s Regions (OQ-15 was settled on 2026-11-04) |
The account refuses Sonnet 5.5 (quota 0, §C-5); the Global route’s Regions are AWS’s | F11-h spec FR-H3 item 5, FR-B3, decision 9; forge-et-be application.yml et.ai |
When AWS raises the quota: remove dev’s override (item 7), draft once, compare the price row with Bedrock’s pricing page and correct application.yml if it differs, and confirm the task role’s policy |
Dinil, with the DevOps team (Shyamin Ayesh) | open |
| 6. No lead, inquiry or email text in the task’s log at DEBUG during a draft | EmailDraftLogHygieneIT sweeps the local log at DEBUG; only a real SDK call over the real HTTP client shows what reaches the task’s log |
F11-h spec FR-H3 item 6, NFR-2, AC-28; F12 row 8 | On dev, raise root to DEBUG through the loggers actuator, draft from a lead whose inquiry holds a coined word and reply to a message holding another; grep the task’s log for both; restore the level |
Dinil (F11-h) | open |
7. The email-draft model override while Sonnet 5.5 is refused — the task definition of every environment applied while the account refuses Sonnet 5.5 (dev now, and the temporary UAT instance if it is applied before the quota is raised) sets ET_AI_FEATURES_EMAILDRAFT_MODEL=us.anthropic.claude-haiku-4-5-20251001-v1:0; the variable is removed once item 5 passes. §D carries the row. 26 Oct: production included, and removed only when the lead moves drafts off Haiku 4.5 on the evaluation’s result (ai-provider-strategy.md §5 row 10, §4d). |
application.yml defaults the feature to Sonnet 5.5 (AD #19), which the account refuses, so without the override every draft on dev fails to the manual path; dev’s entry is in forge-et-be .aws/td.dev.json since forge-et-be#280 (merged 2026-10-20), and a draft on dev also needs the task role’s Bedrock grant, which forge-et-iaac#3 applied on 20 Oct (the bedrock policy on et-dev-backend-task allows bedrock:InvokeModel on the us. Haiku 4.5 profile and its foundation model in the three US Regions, read 21 Oct) |
F11-h spec FR-H3 item 7, decision 2; Decomposition Plan ## Wave Ship Plan C3; AiPropertiesBindingSpec (the variable binds) |
A draft on dev succeeds and its ai_invocation.model_id is the Haiku 4.5 profile; tell the DevOps team the entry is there so a direct push keeps it (.forge/owed.md); remove the variable once item 5 passes |
Dinil | open — set on dev, not yet proved |
8. The draft client’s timeout sits above dev’s deadline — the frontend’s draft request timeout (12 s, spec NFR-1) is above dev’s et.ai.call.deadline and the measured end-to-end time of a draft on the configured model |
The constant assumes the 9 s default deadline plus the server’s own work; every test answers in under a second | F11-h spec FR-H3 item 8, NFR-1, AC-34; F12 row 9 | Read dev’s deadline and item 1’s measured p95; confirm 12 s is above both with room for the round trip; raise the constant with the deadline if it is raised | Dinil (F11-h) | open |
9. A matched send’s text reaches a second draft — after item 4’s real send is matched to its Sent Items copy on dev, a second draft from the same lead carries that email’s text in its input and has purpose FOLLOW_UP |
The stand-in mailbox adds no Sent Items copy; locally the path runs only on seeded rows (spec NFR-9: AiDraftInputIT) and on a seeded copy through the real ingest (AiDraftControllerIT) |
F11-h spec FR-H3 item 9, FR-D2, decision 7 | After item 4, wait for the copy to match (email_send.email_message_id set), draft again from the lead: purpose is FOLLOW_UP and the provenance label counts one sent email |
Dinil (F11-h) | open |
F12 — AI inquiry extraction
The seven rows F12 spec FR-H5 names (AC-24), added by F12-WI-1.2. Every F12 AC is asserted against the
stub or a fake Bedrock SDK client; these are what only real Bedrock can prove. What the backend expects
of the cloud is forge-et-be/docs/ai/bedrock-cloud-contract.md.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
1. A real Converse call to Haiku 4.5 through us.anthropic.claude-haiku-4-5-20251001-v1:0 from the ECS task role, answering inside the 9 s deadline; the task role’s policy allows the inference profile and the model in us-east-1, us-east-2 and us-west-2; Haiku 4.5 answers through the tool under toolChoice auto (the tool was forced until forge-et-be#316, BUG-248; row 10) |
Every test runs on a fake SDK client (FakeBedrockRuntimeClient); no account had Bedrock when F12 was built |
F12 spec FR-B2, FR-H5 item 1, AC-2b; F12 WI-1.2 plan Decisions 3–6; forge-et-be ai/service/BedrockAiProvider.java, ai/BedrockClientConfig.java |
On dev, POST /api/v1/leads/extractions with a real inquiry returns an extraction; the ai_invocation row shows bedrock, the profile id, non-zero tokens and SUCCESS; the reply was a toolUse block (no INVALID_OUTPUT with NO_TOOL_USE in the WARN log); the task role’s policy names the inference profile and the foundation model in all three Regions (the resource shape is in the backend contract, from AWS documentation not re-read at implementation) Update 2026-10-21 (F38-b WI-2.1): the task-role-policy half is met — the task role’s bedrock policy allows only bedrock:InvokeModel on the us. Haiku 4.5 and global. Sonnet 5.5 inference profiles, Haiku 4.5’s foundation model in us-east-1, us-east-2 and us-west-2, and Sonnet 5.5’s in any Region; the real call from the task is F38-f’s; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-11. Update 2026-11-04 (F43-WI-4.2): since forge-et-iaac#16 was applied on 2026-11-04 (dev.yaml run 41882320196), the task role’s policy grants Haiku 4.5’s foundation model as arn:aws:bedrock:*::foundation-model/anthropic.claude-haiku-4-5-20251001-v1:0, which matches all three Regions, in place of three per-Region ARNs (F43-WI-3.1 D-2); the check reads that wildcard. Update 2026-11-21 (F38-f WI-2.1): the call half is met — an Extract on dev was answered by Haiku 4.5 through the us. profile from the backend task role (AI Call Completed invocation 550, SUCCESS, 2 090 ms, attempts=1; CloudTrail Converse from et-dev-backend-task), and no extraction call of the run logged NO_TOOL_USE; evidence: §B F38-f “F38-f evidence — AC-2 and AC-3”. |
F38 | open — the call half is met (F38-f WI-2.1, 2026-11-21) and the policy half (F38-b WI-2.1); F38-f-WI-3.1’s walk judges the row |
2. The cloud boot with the Bedrock provider passing AiSeamGuard |
Proved only in a local cloud-shape context with the SDK client faked (MailboxCloudShapePositiveBootIT) |
F12 spec FR-B4, FR-H5 item 2, AC-3; F12 WI-1.2 plan Decision 13 | The dev task starts with no AiSeamGuard refusal; the startup log has no cannot run in a cloud shape without an implementation of: AIProvider line |
F38 | open |
3. Latency and cost — the extraction endpoint under ~10 s at p95 over the evaluation set; the measured cost per call against the proposal’s ~0.25 US cents (§A.3.2); the et.ai.prices row in application.yml ($1.10 / $5.50 per million tokens, list + 10% for the us. profile) checked against Bedrock’s current prices |
Latency and token counts exist only on a real model | F12 spec NFR-1, FR-H5 item 3; ai-provider-strategy.md §4b; forge-et-be application.yml et.ai.prices |
Run the evaluation set against dev (row 4); read p95 from the et.ai.latency timer and the per-call cost from ai_invocation.cost_usd; compare the price row with Bedrock’s pricing page on the day and correct application.yml if it differs Update 2026-11-21 (F38-f WI-2.1): three extractions through the app on dev, on the evaluation set’s importer-full, not-an-inquiry and instruction-in-text, took 2 090, 2 204 and 1 614 ms and cost $0.004948, $0.004852 and $0.004499 each (about 0.48 US cents a call, against the proposal’s ~0.25); the p95 over the whole evaluation set and the price-row check were not part of the run; evidence: §B F38-f “F38-f evidence — AC-2 and AC-3”. |
F38 | open — three calls measured on dev (F38-f WI-2.1, 2026-11-21); the p95 over the evaluation set and the price check are left |
| 4. The FR-B9 evaluation run on the real model | InquiryExtractionEvaluationIT is the stub run only; a real run needs Bedrock in the dev account |
F12 spec FR-B9, FR-H5 item 4; F12 WI-1.1 plan Decision 21; forge-et-be docs/ai/README.md § Evaluation |
Run AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=inquiry-extract -PconfirmBedrockCalls=true from a laptop, with the lead’s OK for the Bedrock calls, and record per-field results against FR-B9’s bar (no invented value, at least 90% of stated values correct) and the run’s cost. The runner is built by forge-et-be#289 (merged 2026-10-21): a JavaExec outside test/check/build that boots the cloud shape with the real Bedrock client, reuses the runner and scorer, and prints tokens and cost from the run’s own ai_invocation rows; F11-h item 2 and F11-d item 5 use the same command. |
Dinil (F12) | open |
5. The manual paths on dev — ET_AI_FEATURES_INQUIRYEXTRACT_ENABLED=false shows the manual path (F38 FR-29); a wrong model id fails as MODEL_UNAVAILABLE to the manual path with the outcome recorded, confirming which Bedrock exception an unknown model id raises (the provider maps ValidationException, ResourceNotFoundException and AccessDeniedException to MODEL_UNAVAILABLE) |
The switch and the mapping are proved on the stub and the fake client; which exception Bedrock raises for an unknown id is read from AWS documentation, not observed | F12 spec FR-B5, FR-H5 item 5, AC-5; F12 WI-1.2 plan Decision 7; forge-et-be docs/ai/bedrock-cloud-contract.md § The request |
Set the switch off and restart: the paste dialog answers with the manual path and the row’s outcome is DISABLED; set ET_AI_FEATURES_INQUIRYEXTRACT_MODEL to a non-existent id (with a matching ET_AI_PRICES_* list) and restart: the call ends MODEL_UNAVAILABLE after one attempt, the ERROR line names the AWS error code (MODEL_UNAVAILABLE is logged at ERROR, every other kind at WARN — F38-i decision 27, forge-et-be#374); if the code is not one of the three mapped, change the mapping and its spec Update 2026-11-21 (F38-f WI-2.2): with ET_AI_FEATURES_INQUIRYEXTRACT_ENABLED=false deployed on dev (backend:179), Extract showed “AI extraction isn’t available right now. You can fill in the lead yourself.” with Continue Without AI, and the call was recorded outcome=DISABLED with no Bedrock attempt; once main was deployed again it answered SUCCESS. No wrong-model-id check was made; evidence: §B F38-f “F38-f evidence — AC-4”. |
F38 | open — the enabled: false half met 2026-11-21 (F38-f WI-2.2); the wrong-model-id half open |
6. The CloudWatch alarms on the circuit opening and on MODEL_UNAVAILABLE outcomes, from the et.ai.invocations counter (tagged feature, outcome) |
The meters ship; the alarms are FS-5’s cloud half | F12 spec FR-B6, FR-H5 item 6; forge-et-be AiInvocationRecorder |
Wire the two alarms on et.ai.invocations (outcome=CIRCUIT_OPEN, outcome=MODEL_UNAVAILABLE); force each on dev (row 5’s wrong model id; five failures in 60 s); the SNS email arrives. Update 2026-10-23 (F38-e spec decision 1): not built in dev; UAT and production decide. With no meter export (F38-e decision 6), they would read the backend’s log lines rather than the counter. The MODEL_UNAVAILABLE alarm is required in production (ai-provider-strategy.md §4c rule 3; F38-i) Update 2026-10-26 (F38-h WI-3.1): F38-h’s uat overlay is built with no alarm (the lead, 2026-10-25; F38-h Decomposition Plan § Notes); UAT’s set comes as a later F38-h work item once the .forge/owed.md row “Decide the UAT and production alarm sets” is decided. The row stays open. |
F38-h, F38-i | open |
7. The §D task-definition rows for the et.ai variables |
application.yml carries every required value, so a task definition sets none; the overrides are proved by a binding spec, not on a running task |
F12 spec FR-B3, FR-H5 item 7, AC-2; forge-et-be docs/ai/README.md § Configuration; §D “Backend — optional overrides” |
Override one value on dev (row 5) and confirm the running task reads it; keep §D in step with AiProperties |
F38 | open |
8. No AI text in the logs at DEBUG — raising the root level (or software.amazon.awssdk, org.apache.hc.client5.http.*) to DEBUG on dev logs no Converse request or response body and no SigV4 Authorization / X-Amz-Security-Token header |
The application.yml caps (org.apache.hc.client5.http.wire, .headers, software.amazon.awssdk at INFO) are asserted by a spec, but only a real SDK call over the real HTTP client shows what reaches the log |
F12 wave-1 review (forge-et-be#229, security pass); coding-standards §8.6 (“AI prompt and response bodies: never in production logs”) | On dev, raise root to DEBUG through the loggers actuator, run one extraction, grep the task’s log for the inquiry text and for Authorization:; both absent; restore the level |
Dinil (F12) | open |
9. The paste dialog’s client timeout against dev’s deadline — EXTRACTION_TIMEOUT_MS (12 s, forge-et-fe src/features/leads/inquiryExtractionApi.ts) sits above dev’s et.ai.call.deadline and the measured end-to-end time, so a call that finishes near the deadline reaches the pre-filled form, not the manual path |
The constant assumes the 9 s default deadline plus 3 s for the server’s own work, the network and a cold CSRF priming; every test runs under jsdom or against the local stub, which answers in under 1 s | F12 spec NFR-1, FR-U4; F12-WI-2.3 plan Decision 3; contract row 4 Deadline | On dev, read the task’s et.ai.call.deadline (default 9 s) and row 3’s measured p95; confirm 12 s is above both with room for the browser’s round trip; paste an inquiry whose call takes close to the deadline (or set the deadline just under 12 s for one check) and confirm the form opens; if the deadline is raised above about 11 s, raise the constant with it |
Dinil (F12) | open |
10. BUG-248: every model answers through the tool under toolChoice auto — one Converse call to Sonnet 5.5 (global.anthropic.claude-sonnet-5-5) on dev returns its answer through the prompt’s tool rather than the ValidationException a forced tool drew, and the Haiku 4.5 live evaluations of inquiry-extract, lead-triage, email-draft and account-briefing pass as they did with the forced tool, with their INVALID_OUTPUT rate and tokens read from the run (Haiku 5.5 may think before the call under auto, which changes its token use: model-choice-evaluation-plan.md P-1) |
BedrockAiProviderSpec asserts the request shape on a fake SDK client only; no Sonnet 5.5 call has been made from any environment, and the account’s Sonnet 5.5 quota is 0 (§C-5) |
bugs.md BUG-248; forge-et-be#316; ai-provider-strategy.md §4c rule 2 |
Once forge-et-be#316 is deployed to dev, and with the lead’s approval of the Bedrock calls: AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=<each of the four> -PconfirmBedrockCalls=true on Haiku 4.5; once Sonnet 5.5 answers on dev (§C-5 row “Sonnet 5.5 and Haiku 5.5 answer one call each in dev”), the same for email-draft with -Pmodel=global.anthropic.claude-sonnet-5-5; record the runs here |
Dinil (F12) | open |
11. BUG-342: a Haiku 5.5 refusal ends at once — a Haiku 5.5 reply its safety classifiers refuse arrives on Bedrock with a stop reason the backend reads as a refusal (refusal, content_filtered or guardrail_intervened), so the call ends INVALID_OUTPUT after one attempt and its Bedrock Attempt Failed line names kind=REFUSED |
BedrockAiProviderSpec and AiCallRunnerSpec use a fake SDK reply with the stop reason set as a string; which string Haiku 5.5 actually sends on Bedrock is unknown until it answers, and the account’s Haiku 5.5 quota is 0 (§C-5) |
bugs.md BUG-342; forge-et-be#361; model-choice-evaluation-plan.md P-4 |
Once Haiku 5.5 answers on dev (§C-5), and with the lead’s approval of the Bedrock calls: in the evaluation runs on Haiku 5.5, read each Bedrock Attempt Failed line; a refusal must show kind=REFUSED with one attempt. If Haiku 5.5 sends another stop reason for a refusal, add it to BedrockAiProvider’s list in a fix PR. Record the stop reason seen here |
Dinil (F12) | open |
F13 — AI account briefing
The five items F13 spec FR-H4 names (AC-27, WI-1.1’s clause), added by F13-WI-1.1. Every F13 AC is asserted against the AIProvider stub and a test-only scripted provider; these
are what only real Bedrock and a deployed environment can prove. The briefing runs on F12’s seam, runner and record
through extract, so F12’s rows 2, 6 and 7 (the cloud boot, the alarms, the overrides) cover it too.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
1. A real Converse call for account-briefing from the dev task role — on Haiku 4.5 through dev’s override now (item 4), on Sonnet 5.5 once its quota is raised — answering inside the 9 s deadline at p95 over the evaluation set, with the cost per call read from ai_invocation.cost_usd and compared with proposal §A.3.2’s 2.1 US cents |
Every test runs on the stub or a scripted provider; the account refuses Sonnet 5.5 (quota 0, §C-5); the 9 s deadline was set before a synthesis feature existed (F13 spec W-3) | F13 spec FR-H4 item 1, FR-B2, NFR-1, NFR-4, decision 5; forge-et-be account/service/AccountBriefingService.java |
On dev, prepare a brief on an account with timeline entries; the ai_invocation row shows bedrock, the configured profile id, non-zero tokens and SUCCESS; read p95 from et.ai.latency{feature=account-briefing} over item 2’s run and the cost per call from cost_usd, and record it here against 2.1 US cents; if Sonnet 5.5 cannot answer inside the deadline, reconsider streaming or a background job by a spec revision (F13 spec decision 5) Update 2026-11-21 (F38-f WI-2.1): the Haiku 4.5 half — three briefings on dev from accounts holding the evaluation set’s busy-account, entry-holding-a-price and note-with-an-instruction (entries added as notes), through dev’s us. Haiku 4.5 override, each SUCCESS in 1 606–4 698 ms at $0.003581–$0.004023 a call (about 0.39 US cents, against 2.1); no forbidden term, and the price in the notes did not leak; the Sonnet 5.5 half and the p95 over item 2’s run are left; evidence: §B F38-f “F38-f evidence — AC-2 and AC-3”. |
Dinil (F13); when dev Bedrock answers (F38) | open |
| 2. The FR-B7 evaluation run on dev — on Haiku 4.5 now and on Sonnet 5.5 once its quota is raised (§C-5) — with the per-case results, a person’s usefulness rating (at least 4 of 5 on average), and the count of re-engagement lines the dormancy rule dropped on active cases | BriefingEvaluationCaseRunner takes any AIProvider, but only the stub run (AccountBriefingEvaluationIT) runs in the suite; the scorer scores the re-engagement line after the rule (F13 spec Rev 1), so the dropped-line count is what shows the model writing one for an active account |
F13 spec FR-B7, FR-H4 item 2, Rev 1; F13 WI-1.1 plan Decision 16; forge-et-be docs/ai/README.md § Evaluation |
With the lead’s OK, run AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=account-briefing -PconfirmBedrockCalls=true -Pmodel=<profile id> on each model; record per run which cases pass every check, the dropped-line count from the report and the person’s rating, and the model kept for UAT |
Dinil, with Tharinda | open |
3. The manual path on dev — ET_AI_FEATURES_ACCOUNTBRIEFING_ENABLED=false makes the panel show the unavailable message while the rest of the account page works; a wrong model identifier ends MODEL_UNAVAILABLE and shows the same message |
Proved on the stub (AccountBriefingDisabledIT, AccountBriefingControllerIT) and in the browser suite against the stub |
F13 spec FR-H4 item 3, FR-E6, NFR-5, AC-15 | Set the switch off and restart: Prepare briefing shows the unavailable message, and the profile, contacts, notes and timeline still load; set the model to a non-existent id (with a matching price list) and restart: the row ends MODEL_UNAVAILABLE and the panel shows the same message Update 2026-11-21 (F38-f WI-2.2): with ET_AI_FEATURES_ACCOUNTBRIEFING_ENABLED=false deployed on dev (backend:179), Prepare briefing showed “The AI briefing isn’t available right now. Everything else on this page works as usual.” while the profile, contacts, notes and timeline loaded, and the call was recorded outcome=DISABLED with no Bedrock attempt; once main was deployed again it answered SUCCESS. No wrong-model-id check was made; evidence: §B F38-f “F38-f evidence — AC-4”. |
F38 | open — the enabled: false half met 2026-11-21 (F38-f WI-2.2); the wrong-model-id half open |
4. The account-briefing model override while Sonnet 5.5 is refused — the task definition of every environment applied while the account refuses Sonnet 5.5 (dev now, and the temporary UAT instance if it is applied before the quota is raised) sets ET_AI_FEATURES_ACCOUNTBRIEFING_MODEL=us.anthropic.claude-haiku-4-5-20251001-v1:0; the variable is removed once Sonnet 5.5 answers on dev. §D carries the row. 26 Oct: production included, and removed only when the lead moves the briefing off Haiku 4.5 on its evaluation’s result (ai-provider-strategy.md §5 row 10, §4d). |
application.yml defaults the feature to Sonnet 5.5 (AD #19), which the account refuses, so without the override every brief on dev fails to the manual path, and five refused presses in a minute open the shared bedrock circuit breaker (F13 spec decision 8). Dev’s entry is in forge-et-be .aws/td.dev.json on the F13 wave branch (F13-WI-1.1), with the lead’s approval recorded in the WI-1.1 plan |
F13 spec FR-B8, FR-H4 item 4, AC-28, decision 8; AiPropertiesBindingSpec (the variable binds); .forge/owed.md (FR-H3 (d): tell the DevOps team the entry is there so a direct push keeps it) |
A brief on dev succeeds and its ai_invocation.model_id is the Haiku 4.5 profile; tell Shyamin Ayesh the entry is there (owed.md); remove the variable once Sonnet 5.5 answers a brief on dev |
Dinil | open — merged in forge-et-be#296 on 2026-10-22; a brief on dev not yet checked. Update 2026-11-17 (F38-g-WI-4.4): F13.4 passed on dev — a brief succeeded on 2026-11-16 17:59 UTC (200, state BRIEFED, invocationId 378; evidence .forge/plans/F38-g/F38-g-retest-report.md group 4.4 and F38-g-WI-4.4-record.md R-2); the ai_invocation.model_id half needs a database read and the rest of the row is unchanged. |
| 5. No prompt, account data or reply text in the dev logs at the configured log level | AccountBriefingLogHygieneIT sweeps the local log at DEBUG; only a real SDK call over the real HTTP client shows what reaches the task’s log |
F13 spec FR-H4 item 5, NFR-2, AC-26; F12 row 8 | On dev, prepare a brief on an account whose name and a note hold coined words; grep the task’s log for both and for the brief’s text; then raise root to DEBUG through the loggers actuator, repeat, and restore the level |
Dinil (F13) | open |
F20-a — Mailbox connection (done)
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
authorized/microsoft-mailbox reply URL per deployed environment, registered as platform Web (not spa — a spa registration gives 24-hour non-resetting refresh tokens, which silently breaks every mailbox) |
environments-and-identity.md §2.2 registers the login callback only; registering is Empire IT’s |
F20-a-mailbox-connection-spec.md FR-2 / FR-16, W-3; environments-and-identity.md §2.2; F20-a-WI-2.1-plan.md § Notes qualification 1 |
Register https://api.dev.crm.empire.lk/authorized/microsoft-mailbox, …api.uat.crm.empire.lk… and …api.crm.empire.lk… as Web reply URLs on the one CRM registration; /oauth2/authorization/microsoft-mailbox completes with no redirect_uri mismatch. All three registered as Web on 2026-09-24 (Kavindu Anthony’s portal screenshot; environments-and-identity.md §2.2); what remains is the live check that the consent redirect completes on each environment |
Registered by Kavindu Anthony (Empire IT); the live check is F38’s | open |
| The real mailbox consent round trip — the same-user check and the scope-string normalisation | Needs a real Entra authorization-code exchange; the offline tests run against a faked token-response client (spec D-2) | spec FR-3 / FR-4 / FR-16, AC-4, AC-5 | Consent as CRM01@empire.lk; the row is written and the browser lands on ?connect=success; the id token’s real sub equals the stored user_identity.subject and the Graph /me object id is stored separately as provider_account_id; attempt as CRM02 while signed in as CRM01 and confirm the refusal; granted_scopes normalises to a space-separated list that includes User.Read Mail.ReadWrite Mail.Send — never an exact match: Microsoft returns every delegated scope the tenant has consented for the app, and Empire consents calendar and contacts for Phase 2 (F5-i spec Rev 4, L-104). The consent needs email.manage at both ends (BUG-022): give CRM01’s role the key before this check, and prove the refusal without it through the §B F5-k BUG-022 row |
— Laptop, not dev (2026-09-29, Nanndish, F5-i-WI-2.1 R-3, R-5): consent as CRM01 succeeded after a laptop-only email.manage grant to it_admin (not audited, removed by the reset); consenting as CRM02 while signed in as CRM01 was refused. granted_scopes came back wider than the three requested (the tenant’s other consented scopes); accepted by the lead 2026-09-30 (spec Rev 4). The record’s R-3 evidence block holds a 403 and none of the values the step lists, so the stored granted_scopes string, the connection row and the mailbox.connected event were not captured on the laptop (F5-i findings F-008); capture them on dev. Still owed on dev. |
open |
A live token refresh, and a forced invalid_grant; also whether Microsoft keeps the replaced refresh token exchangeable (the rolled-back-rotation safety assumes it does — read from Microsoft docs 2026-09-22, never seen live) |
Entra rotates refresh tokens and issues the error sub-codes; a stub can produce neither | spec FR-6 / FR-16, AC-8, AC-9, NFR-7; F20-a-WI-2.1-plan.md § Notes (refresh-token answer, qualification 2) |
Force a refresh: the stored ciphertext changes and last_refreshed_at advances; keep the replaced token and confirm it still exchanges (if not, give the rotation its own commit boundary); revoke consent (or reset the password) and confirm the next refresh returns invalid_grant, the connection moves to RECONNECT_REQUIRED with the mapped status_reason, and the sub-code matches the FR-6 mapping |
— Laptop, not dev (2026-09-29, Nanndish, F5-i-WI-2.1 R-4): the connect-time back-fill made a live token refresh (mailbox.token_rotated, refreshToken changed). The forced invalid_grant and the replaced-token check are still owed. Still owed on dev. | open |
Graph subscription creation against the reachable callback (et.mailbox.public-base-url); with it unset, no subscription is attempted |
Graph validates the notification URL synchronously over the public internet | spec FR-9 / FR-16, AC-14, AC-15; MailboxConnectionService.java 388–401 |
With the base URL set to the environment’s api.* host, connect a mailbox: Graph’s validation POST reaches /api/v1/mailbox/notifications, the echoed validationToken is accepted, the row stores a real subscription_id / subscription_expires_at with subscription_status = ACTIVE; the requested expiry is within Graph’s current maximum for mail |
— | open |
KMS-wrapped refresh-token data keys — the real key, the real grant, the real IAM permissions, the encryption-context binding; and the key-rotation hazard (re-pointing et.mailbox.token-key-id at a different key makes every wrapped_data_key unreadable with no modelled reconnect) |
The KMS TokenKeyProvider calls GenerateDataKey/Decrypt through the task role; every test fakes the SDK client (spec D-2); locally LocalTokenKeyProvider wraps under a fixed source-derived key that “protects nothing” |
spec FR-5 / FR-16, AC-7b, W-6; F20-a-WI-2.2-plan.md Decision 5; LocalTokenKeyProvider.java 39–64; KmsTokenKeyProvider.java |
With et.mailbox.token-key-id set: the stored wrapped_data_key is a real KMS ciphertext carrying that key id; a refresh decrypts it; the task role holds kms:GenerateDataKey + kms:Decrypt on that key and nothing wider, and removing the grant fails loudly. Copy one row’s ciphertext + wrapped key onto a second connection and confirm the decrypt is refused. Read the rotation hazard before rotating: automatic rotation keeps the key id and is safe; a new key id is a manual operator recovery today (a StatusReason addition owned by F20-a WI-1.1) Update 2026-10-21 (F38-b WI-2.1): the real key exists — alias/et-dev-mailbox-token, key id ce1dbe4f-711e-4dd3-ab6e-9ce1930e9040, customer-managed, rotation on — and the task role may call kms:GenerateDataKey and kms:Decrypt on it; the real wrap and unwrap is F38-f’s; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-6. |
— | open |
Customer-managed KMS key per environment with rotation enabled; task role granted kms:GenerateDataKey + kms:Decrypt on that key alone; ET_MAILBOX_TOKEN_KEY_ID and ET_MAILBOX_PUBLIC_BASE_URL on the task definition (MailboxCloudConfig refuses to boot naming whichever is absent) |
No account; architecture.md lists KMS for database, storage and backups but no application-level token key |
spec FR-5 / FR-16, W-6, OQ-C; MailboxCloudConfig.java 31–51 |
Terraform the key; set both values; the backend starts Update 2026-10-21 (F38-b WI-2.1): the key is Terraformed and applied (alias/et-dev-mailbox-token, ce1dbe4f-711e-4dd3-ab6e-9ce1930e9040, rotation on) with the task role’s grant on it; setting ET_MAILBOX_TOKEN_KEY_ID and ET_MAILBOX_PUBLIC_BASE_URL is F38-d’s (its FR-6); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-6 and AC-9. |
Tharinda (FS-1 cloud half) | open |
BUG-174: the KMS client’s calls are bounded on dev. Since forge-et-be#244, KmsConfig sets apiCallTimeout 5 s and apiCallAttemptTimeout 2 s on the client, and KmsConfigSpec asserts both on the built client. A mailbox connect calls GenerateDataKey while it holds the owner’s app_user row FOR SHARE, so the bound is how long a stalled call can hold an edit or deactivation of that user, and with it the administration advisory lock (access.manage.guard, renamed settings.manage.guard by the forge-et-be change that folds access.manage into settings.manage). What is not proved is that the SDK enforces it against real KMS |
KMS exists only in AWS; locally LocalTokenKeyProvider makes no call, so no local test reaches the client |
bugs.md BUG-174; BUG-100 (forge-et-be#235); forge-et-be#244; KmsConfig.java; L-149 |
On dev, with KMS made slow or unreachable from the task (a network ACL or security-group rule that drops KMS traffic for the test, then restored): a mailbox connect fails within about 5 s and lands on the connect-failed banner, and an administrator’s edit of the same user, started during the connect, completes once it fails | Tharinda | open |
The Entra certificate reused by the microsoft-mailbox registration (private_key_jwt code exchange) |
The FS-6 certificate row covers login only | spec FR-2, Constraints “Still owed externally” | When validating the FS-6 certificate row, also run the mailbox consent exchange with the same certificate | Certificates uploaded 2026-09-24 (the FS-6 row); the live exchange is F38’s Laptop, not dev (2026-09-29, Nanndish, F5-i-WI-2.1 R-3): the microsoft-mailbox code exchange worked with the non-prod certificate. Still owed on dev. | open |
The per-connection Graph rate-limit default (capacity 60, refill 240/min, interactive reserve 20 since F20-i wave 2; capacity 20, refill 60/min before) against real Graph throttling (429, Retry-After) |
Checked against Microsoft documentation only; the code runs against a stubbed Graph | spec FR-15, W-5; F20-a-WI-1.1-plan.md § Progress subtask 3; F20-i NFR-2, F20-i-WI-2.1-plan.md (the new defaults, the lead, 2026-11-03) |
On a live mailbox under load, the limiter keeps Graph from returning 429s and a real 429 is retried, not dropped; tune et.mailbox.rate-limit.* if needed |
— | open |
Recorded live evidence per phase on Empire’s test accounts CRM01–05@empire.lk (there is no separate M365 test tenant) |
D-2/D-3 moved all live Graph and Entra evidence to “once a cloud environment exists” | spec D-2, D-3; test-strategy.md § Microsoft Graph |
Record the live evidence per phase as the test strategy requires | — Laptop, not dev (2026-09-29, Nanndish, F5-i-WI-2.1 R-1 to R-5): partial evidence recorded on the laptop as CRM01 and CRM02. Still owed on dev. | open |
The mailbox stubs (POST /api/v1/mailbox/stub-connect, the error-forcing stub-state endpoint) are really absent on deployed APIs; the SPA’s dev-only stubConnect() branch is gone from the bundle |
AC-22b was proven inside a local cloud-profile test | spec FR-13 / AC-22b; MailboxStubController.java 43; forge-et-fe/src/features/email/NotConnectedPrompt.tsx 43–48 |
On a deployed API both return 404; the bundle has no stub-connect |
— | open |
CloudWatch alarm on the RECONNECT_REQUIRED gauge |
The gauge ships; the alarm is FS-5’s | spec NFR-4 / AC-27; F20-a-WI-1.1-plan.md Decision 20 |
Wire the alarm; force one connection into RECONNECT_REQUIRED; the SNS email arrives. Update 2026-10-23 (F38-e spec decisions 1 and 6): not built in dev; if UAT or production builds it, a metric filter on Mailbox Connection Status Changed … to=RECONNECT_REQUIRED replaces the gauge export Update 2026-10-26 (F38-h WI-3.1): F38-h’s uat overlay is built with no alarm (the lead, 2026-10-25; F38-h Decomposition Plan § Notes); UAT’s set comes as a later F38-h work item once the .forge/owed.md row “Decide the UAT and production alarm sets” is decided. The row stays open. |
F38-h, F38-i | open |
BUG-067: after a re-consent that widens the grant on one backend task, every other task stops serving its narrower cached token and refreshes on the next request for the new scope; also that Entra’s refresh response names the scopes in a form MailboxScopes.normalise reads (if it does not, every connection moves to RECONNECT_REQUIRED / SCOPES_CHANGED on its first refresh, with a Mailbox Token Scopes Short ... required=true WARN) |
The token cache is in memory per ECS task; locally there is one process, so the test stands in a second MailboxTokenService instance for the second task (MailboxReconsentCacheIT) |
.forge/bugs.md § BUG-067; forge-et-be#154; forge-et-be MailboxTokenService (CachedToken.serves, the same-consent skip in obtainDeclared) |
With two or more backend tasks on dev, connect a mailbox, make mail calls through each task, re-consent with an added scope through one task, then make a call needing that scope through another: it succeeds with no 403, and that task’s log shows a token refresh; across normal use, no Mailbox Token Scopes Short WARN appears. Needs a feature that asks for a scope beyond MailboxScopes.REQUIRED (F21-b or F22); before then, check only the second half. Update 2026-11-21 (F22-WI-2.1): F22’s contacts read asks for Contacts.ReadWrite, which is in MailboxScopes.REQUESTED but not REQUIRED; once F22 wave 1 is on dev, the scope half can be checked with it (§B F22). |
F38 (dev with two tasks); the scope half with F21-b | open |
F20-b — Inbound sync and in-CRM inbox (done 2026-09-25 — both waves merged; every row below is still owed on dev)
Spec FR-13 / AC-31 require the shipping PRs to add these: Wave 1 through WI-1.1, Wave 2 through WI-2.1. They are pre-registered here from the approved spec so the walk-list is complete; the WI PRs refine them and flip nothing until cloud. See also A-3.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
Real Graph change notifications arriving at /api/v1/mailbox/notifications with an accepted client_state |
“Nothing offline proves Graph will actually POST” — the stub fakes them. WI-1.1 proves the receiver against hand-built Graph-shaped bodies (MailboxNotificationIngestIT) and the stub (MailboxStubNotifyIT); the body shape it reads (value[].subscriptionId, clientState, resourceData.id, resource) is from Microsoft’s documentation, not a captured delivery |
spec FR-3, FR-13, AC-31 (WI-1.1 row); Constraints “Client dependency”; MailboxNotificationController.java, MailboxNotificationRequest.java |
Connect a live mailbox, send it an email; the notification lands, the counter et.mailbox.notifications.unattributable does not move, and one email_message row appears |
— | open |
ET_MESSAGING_MAILBOX_INGEST_QUEUE on the backend task definition, plus the ingest SQS queue and its DLQ with a redrive policy (maxReceiveCount 3, the et.messaging.* default) |
WI-1.1 plan D16/D18: MailboxIngestJobListener is @ConditionalOnProperty on the key and MailboxIngestEnqueuer sends nothing without it (one WARN line per dropped job), so a task definition without it boots healthy and ingests nothing — no notification and no back-fill ever lands. The queue pair is QueueProvisioner’s against LocalStack only |
WI-1.1 plan D16, D18; EtProperties.Messaging#mailboxIngestQueue; MailboxIngestJobListener.java; QueueProvisioner.java |
Terraform the queue + DLQ + redrive; set the key; connect a mailbox and see the back-fill land; stop the worker, send a notification, and see one message on the queue Update 2026-10-21 (F38-b WI-2.1): the queue et-dev-mailbox-ingest and its dead-letter queue exist (redrive at 3, 30-second visibility, SQS-managed encryption, 14-day dead-letter retention) and the task role may consume from and send to it; setting the key is F38-d’s (its FR-6); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4 and AC-9. Update 2026-10-23 (F38-b WI-3.1): the running task definition backend:60 sets ET_MESSAGING_MAILBOX_INGEST_QUEUE=et-dev-mailbox-ingest, and the running backend consumed a malformed message sent to the queue and failed on it three times, so it reached the dead-letter queue; the back-fill and the stop-the-worker check are still open (F38-f); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. |
FS-1 cloud half | open |
POST /api/v1/mailbox/stub-notify is really absent on deployed APIs |
WI-1.1 proves it as mapping absence inside a local no-profile context (MailboxCloudShapeIT); only a deployed API shows it |
spec FR-12, AC-27 (wave-1 clause); MailboxStubController.java |
On a deployed API, an authenticated POST answers 404 (an unauthenticated one answers 401 — the filter chain runs first) |
— | open |
| The notification endpoint answers fast enough that Graph does not retry (a retry storm) (uncertain — not named in FR-13) | Only real Graph shows its retry behaviour | spec NFR-3 | Under live delivery the endpoint returns 202 quickly and Graph delivers each notification once | — | open |
The delta query against a real mailbox (real delta pages and @odata.deltaLink) |
The stub’s fake delta link advances; AC-15 / AC-2d run on the stub (MailboxDeltaPollIT), AC-27b on faked transport. WI-2.1 stores one link per folder in delta_token (<folder>\t<link> lines), drops a link Graph refuses with a non-429 4xx so the folder restarts, and clears the links on a reconnect out of RECONNECT_REQUIRED |
spec FR-8, FR-13; AC-15, AC-2d, AC-27b; MailboxDeltaPollJobHandler.java |
Drop a notification on a live mailbox, run the poll; the message lands exactly once and last_synced_at moves. Let a stored link age past Graph’s delta-token lifetime and confirm the refusal is dropped and the folder restarts. Laptop, not dev (2026-10-14, Dinil Gamage with Claude, F20-c-WI-2.1 R-6): a hand-fired poll’s first round over inbox and sentitems read CRM01’s real mailbox (active=1, reached=1, succeeded=1) and ingested the run’s four Sent Items copies, each matched once. Still owed on dev. |
— | open |
The delta poll’s reach and its first-round cost (added by WI-2.1; revised by BUG-069): it polls only Graph’s inbox and sentitems (message delta is per folder; mail filed into a user’s own folders or Archive is recovered by notifications only). Since BUG-069, a first round on a folder with no stored link asks Graph for mail received from the back-fill window’s start only ($filter=receivedDateTime ge …, which Graph keeps in the round’s links) and every delta request asks for 50 messages a page (Prefer: odata.maxpagesize=50); the poll still pages at most 10 pages per folder per run and still skips mail older than et.mailbox.backfill-days without a claim. Graph’s documentation says a filtered delta round returns at most 5,000 messages. Only the first round after a connect is backed by the back-fill, which pages the window once; a round the poll restarts later (after a refused or dropped link) is bounded by the cap alone, so in a folder with more than 5,000 messages in the window it may miss mail whose notification was also missed. Whether to narrow a restarted round’s filter (for example to last_synced_at less a margin) or start a gap back-fill is open for the lead (forge-et-be#192 review). A refusal of a fresh round now fails that connection’s poll rather than being logged as a discarded link. Each delta request sends Prefer as two header lines, IdType="ImmutableId" first |
Only a real, large mailbox shows that Graph honours the filter, the page size and the immutable-id preference together on message delta, the page count of a first round, and how the 5,000 cap behaves | .forge/bugs.md § BUG-069; GraphMailboxProvider.java and MailboxDeltaPollJobHandler.java class comments; spec FR-8, NFR-4 |
On a busy live mailbox, time the first poll and count its pages and messages: no message older than the window comes back, pages hold up to 50, and no poll logs Mailbox Delta Poll Failed with a 400. Confirm the delta response’s Preference-Applied header lists both preferences, or that a message seen by both the poll and a notification lands as one email_message row. On a mailbox with more than 5,000 messages in the window, record which messages a capped round returns (newest or oldest), confirm the round still ends with a deltaLink, and that later rounds deliver new mail. Decide whether other folders need polling |
— | open |
| Renewal against Graph’s real expiry clock (7-day maximum minus a 15-minute margin) | Proven against the stub only (MailboxRenewalIT, MailboxRenewalServiceSpec). WI-2.1 treats a 404/410 on renewal as SUBSCRIPTION_GONE, other 4xx as RENEWAL_FAILED, and 429/5xx/network/limiter as transient (no lapse); since BUG-025 (forge-et-be#60) a 401 and any non-HTTP failure are transient too, and a 401 clears the cached token — the status Graph actually returns for a subscription it has forgotten is from its documentation |
spec FR-6, FR-7, FR-13, AC-31 (WI-2.1 row); MailboxRenewalService.java |
Leave a live subscription past one renewal threshold; subscription_expires_at moves forward and Graph agrees. Delete a subscription at Graph by hand, run renewal, and confirm the row goes FAILED with SUBSCRIPTION_GONE |
— | open |
| 30-day back-fill within Graph’s throttling budget on a busy mailbox (uncertain — not named in FR-13) | The budget is from documentation | spec D-2, NFR-4 | Connect a busy live mailbox; the back-fill finishes without starving live ingest or sustained 429s; tune et.mailbox.backfill-days if needed |
— | open |
| EventBridge schedule + target SQS queue + IAM for the renewal job (and its message body per FR-S1) | The FS-9 cloud half does not exist; without it renewal never fires and sync dies silently after 7 days (R-12). Update 2026-10-21: the mailbox-subscription-renewal schedule, its role and the queue were applied on dev with forge-et-iaac#3 on 20 Oct; F38-b WI-2.1 checks them against this row. |
spec FR-6, FR-13, AC-13b; forge-et-be/docs/messaging/scheduler-cloud-contract.md § “Mail subscription renewal” (the FR-D1 register: jobKind mailbox-subscription-renewal, rate(6 hours) = et.mailbox.renewal.cadence, the scheduler queue, idempotencyKey from <aws.scheduler.scheduled-time>) |
Terraform the schedule per the register’s expression, queue, payload and IAM shape (except aws:SourceArn, which is the schedule group’s ARN, as AWS requires — F38-b spec Rev 2; the contract fix is in .forge/owed.md); a renewal message reaches the worker (Mailbox Subscription Renewal Completed line). If the cadence property is ever changed, the expression changes with it Update 2026-10-21 (F38-b WI-2.1): mailbox-subscription-renewal (rate(6 hours), UTC) targets the scheduler queue with the contract’s Input, first fired at 2026-10-20 18:35 UTC with no target error, and its messages reached the queue; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. Update 2026-10-26 (F38-f WI-1.1): four Mailbox Subscription Renewal Completed lines at INFO in the 26 hours read (00:35, 06:35, 12:35 and 18:35 UTC, the latest 2026-10-26 00:35 with due=0, reached=0); evidence: §B F38-f “F38-f evidence — AC-8 (d)”. |
FS-1 cloud half | validated — producer half (F38-b WI-2.1, 2026-10-21); the worker’s log line (F38-f WI-1.1, 2026-10-26) |
| EventBridge schedule + target SQS queue + IAM for the delta-poll job | Same reason. Update 2026-10-21: the mailbox-delta-poll schedule was applied on dev with forge-et-iaac#3 on 20 Oct; F38-b WI-2.1 checks it against this row. |
spec FR-8, FR-13, AC-13b; the same register § “Mail delta reconciliation poll” (mailbox-delta-poll, rate(15 minutes), the same queue) |
Same steps for the poll job (Mailbox Delta Poll Completed line) Update 2026-10-21 (F38-b WI-2.1): mailbox-delta-poll (rate(15 minutes), UTC) targets the scheduler queue with the contract’s Input, first fired at 2026-10-20 18:35 UTC and every 15 minutes since with no target error, and its messages reached the queue; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. Update 2026-10-26 (F38-f WI-1.1): 104 Mailbox Delta Poll Completed lines at INFO in the 26 hours read, one per tick, each active=2, reached=2, succeeded=2; evidence: §B F38-f “F38-f evidence — AC-8 (d)”. |
FS-1 cloud half | validated — producer half (F38-b WI-2.1, 2026-10-21); the worker’s log line (F38-f WI-1.1, 2026-10-26) |
ET_MESSAGING_SCHEDULER_QUEUE on the backend task definition |
Required in every cloud shape since WI-2.1: the renewal and poll handlers exist everywhere, so SchedulerConfigGuard refuses to start a task definition without it (the message names the variable). AC-13e/f/g prove it inside local cloud-shape contexts only (SchedulerConfigGuardIT). Those contexts also set spring.cloud.aws.sqs.listener.auto-startup=false, which must never reach a task definition — the listener would exist and never poll |
spec FR-6b, FR-13, AC-13e, AC-13f, AC-13g; SchedulerConfigGuard.java |
Set the key; the backend starts, the listener consumes, a scheduled-demo-job message is rejected to the DLQ (AC-13g); a task definition without the key refuses to start Update 2026-10-23 (F38-b WI-3.1): the running task definition backend:60 sets ET_MESSAGING_SCHEDULER_QUEUE=et-dev-scheduler, the service runs one task, and the listener consumed a scheduled-demo-job message and rejected it three times to et-dev-scheduler-dlq (AC-13g on dev); that a task definition without the key refuses to start is not recorded, so the row stays open; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. |
FS-1 cloud half | open |
The lapsed-subscription alarm on subscription_status = FAILED |
WI-2.1 ships the signal: gauge et.mailbox.subscriptions.failed (live connections with a FAILED subscription — the alarm reads this) and counter et.mailbox.subscriptions.lapses, tag reason_code (RENEWAL_FAILED, EXPIRED, SUBSCRIPTION_GONE); the alarm is FS-5’s (R-12’s mitigation) |
spec FR-7, FR-13, Out of Scope “The CloudWatch alarm itself”; MailboxMetrics.java |
Wire the alarm on the gauge (> 0); force a renewal failure; the alarm reaches the team. Update 2026-10-23 (F38-e spec decisions 1 and 6): not built in dev; if UAT or production builds it, a metric filter on Mailbox Subscription Status Changed … to=FAILED replaces the gauge export Update 2026-10-26 (F38-h WI-3.1): F38-h’s uat overlay is built with no alarm (the lead, 2026-10-25; F38-h Decomposition Plan § Notes); UAT’s set comes as a later F38-h work item once the .forge/owed.md row “Decide the UAT and production alarm sets” is decided. The row stays open. |
F38-h, F38-i | open |
POST /api/v1/mailbox/stub-renewal-failure is really absent on deployed APIs (added by WI-2.1) |
Proven as mapping absence inside a local no-profile context (MailboxCloudShapeIT) |
spec FR-12, AC-27 (wave-2 clause); MailboxStubController.java |
On a deployed API, an authenticated POST answers 404 (an unauthenticated one 401) |
— | open |
| Reconnect from a lapsed subscription on a deployed environment (added at the wave-2 merge, from WI-2.2’s pre-PR review) | Re-consent against real Entra/Graph should create a fresh ACTIVE subscription that clears a FAILED one. Only the stub path is proved: beginMailboxConnect’s DEV branch calls the stub connect, and on a laptop the repair lands in Waiting (NONE), never ACTIVE, because et.mailbox.public-base-url is unset |
WI-2.1 plan D14, D15; WI-2.2 plan (the Reconnect action); spec FR-7, AC-26; SyncIndicator.tsx, SyncLapsedNotice.tsx, MailboxConnectionService.attemptSubscription |
On dev, force a lapse (or let one expire), press Reconnect in the inbox notice, complete the Microsoft consent, and confirm subscription.status returns ACTIVE with a null reasonCode and the notice clears on /email and /email/mailbox. Expected on a deployed environment, not defects (fe#55 review): the repair lands straight in Healthy — “Receiving mail · Last synced <the pre-lapse time>”, because last_synced_at never moves backwards — not in Waiting as it does on a laptop; and a Reconnect pressed on /email returns to /email/mailbox?connect=success (the Wave-1 redirect contract), not to the inbox |
— | open |
Immutable message ids (Prefer: IdType="ImmutableId") keep one email_message row when a message moves folder |
Added by the wave-1 PR review (be#51). Without the header Graph re-issues a message’s id on a folder move, so an Outlook rule, manual filing or Archive produced a second row. The fix sends the header on subscription create and every read; the adapter spec asserts the header on every call path, but only a real mailbox shows that Graph’s notification ids and fetched ids then agree | wave-1 review (be#51), spec FR-3; GraphMailboxProvider.java |
On a live mailbox, send a message, let a rule move it to another in-scope folder; still exactly one email_message row, same graph_message_id |
— | open |
Deferred review findings — ingest under real load (be#51 /forge-review-pr, deferred by Dinil 2026-09-24 to WI-2.1): (1) the Graph fetch runs inside the claim’s DB transaction, so a burst can hold the Hikari pool during 15–20 s HTTP waits; (2) MailboxIngestEnqueuer.accept() holds a read-only transaction across synchronous SQS sends on the unauthenticated request thread; (3) a rate-limiter refusal is thrown as a transient failure and spends maxReceiveCount, so a burst beyond about 80 messages per minute on one connection dead-letters the excess; (4) a back-fill run is bounded by pages, not time, so it can outlive the 30 s visibility timeout and run twice, and the unconditional backfill_cursor write lets the slower run move it backwards |
None of these can occur in local — one user, a stub mailbox, no burst. They matter only with real mailboxes, real Graph latency and concurrent SQS consumers. All four fixed in code by WI-2.1 (be c6fe728), none accepted, each with a local test that fails on the old code: (1) the fetch and the folder resolution run with no transaction open, only the claim and the row share one, and a replay is refused by a claim-store read before any fetch (MailboxIngestTransactionBoundaryIT); (2) accept() is no longer transactional, so no connection is held across the SQS sends (same spec); (3) a limiter refusal is caught by MailboxIngestJobListener and the job re-sent with a delay of the visibility timeout plus up to as much again, the delivery acknowledged — and when no copy can be sent the refusal is rethrown instead, so the original is redelivered rather than lost (be 7035ed6) (MailboxIngestJobListenerSpec; end to end, five armed refusals then one row and an empty DLQ, MailboxIngestFailurePathsIT); (4) a back-fill run also stops after half the visibility timeout, and the cursor write and completion stamp are compare-and-set on the cursor the run read (MailboxBackfillIT, two rows). Wave 1’s exactly-once specs (AC-8, AC-10d, AC-11, AC-15) stay green |
wave-1 review (be#51); MailboxIngestService.java (ingestNotified, claimAndStore), MailboxIngestEnqueuer.java (accept, redeliverLater), MailboxIngestJobListener.java, MailboxBackfillService.java, MailboxConnectionRepository.java (recordBackfillCursor, completeBackfill, isMessageClaimed) |
Still owed on dev: a 200-message burst to one mailbox lands every message once with nothing dead-lettered, the connection pool does not saturate during it, and the Mailbox Ingest Deferred lines show the refused jobs coming back spread out rather than together |
WI-2.1 → cloud walk | open |
A dashboard for the unrecognised-client_state counter (orphaned subscriptions left by BUG-018’s best-effort delete) |
Nobody will look without an FS-5 dashboard. WI-1.1 ships the counter as et.mailbox.notifications.unattributable, one reason tag with three values (CLIENT_STATE_MISMATCH, UNKNOWN_SUBSCRIPTION, MALFORMED_BODY) — the orphan case is UNKNOWN_SUBSCRIPTION |
spec D-3, FR-13, AC-31 (WI-1.1 row); MailboxMetrics.java |
Build it beside the lapsed-subscription alarm, split by reason; at that first walk, read the counter and record whether the orphan problem is real (WI-1.1 plan § Open items closed at plan time) — that decides whether a reconciliation pass is worth building. Update 2026-10-23 (F38 Rev 23; F38-e spec decisions 1 and 6): no dashboard and no meter export in dev; a candidate for UAT and production with the lapsed-subscription alarm (.forge/owed.md “Decide the UAT and production alarm sets”). With no meter export it would read the backend’s log lines rather than the counter Update 2026-10-26 (F38-h WI-3.1): F38-h’s uat overlay is built with no alarm (the lead, 2026-10-25; F38-h Decomposition Plan § Notes); UAT’s set comes as a later F38-h work item once the .forge/owed.md row “Decide the UAT and production alarm sets” is decided. The row stays open. Update 2026-11-02 (F20-i spec Rev 2): a release by a role edit or a role change now leaves its subscription to expire on purpose, so its notifications are also counted as UNKNOWN_SUBSCRIPTION until it expires (MailboxIngestEnqueuer); when judging whether the orphan problem is real, subtract the role-edit releases of the preceding days (mailbox.released with PERMISSION_REMOVED and an administrator as disconnected_by), or read the count only after their subscriptions have expired. |
F38-h, F38-i | open |
The excluded-folder cache across ECS tasks (added by BUG-050): each task keeps its own cache of a mailbox’s Drafts, Junk Email and Deleted Items ids. Every back-fill job (even one that stops at once because the back-fill is complete) clears its connection’s entries on the task that runs it, and every entry expires after one hour (MailboxIngestService.FOLDER_CACHE_LIFETIME), so live notifications on the other tasks pick up a replaced mailbox’s folders within the hour |
Locally there is one JVM: MailboxBackfillIT proves a first run, a continuation and a re-consent’s stopped job each clear or re-resolve the ids on an instance whose cache still holds the old ones, and MailboxIngestFolderCacheSpec proves the expiry with a fixed clock, but neither runs two real tasks. Only dev with two or more tasks shows a continuation landing on another task |
.forge/bugs.md BUG-050 (lead decision, 2026-09-30); spec FR-3; MailboxIngestService.java (forgetFolders, folders), MailboxBackfillService.java |
With the backend at two or more tasks, reconnect a test mailbox whose folders have changed (or one IT has re-provisioned): the back-fill stores nothing from its new Drafts, Junk Email or Deleted Items, whichever task runs each run; a notification for a new draft within the first hour may be ingested on a task that ran no back-fill, and none after the hour | — | open |
F20-c — Compose, reply and send
Spec FR-G2 / AC-40 require the shipping PR to add these; F20-c-WI-1.1 (the send API) adds them. Everything else
in F20-c’s backend runs and is proved locally against the LocalMailboxProvider stand-in with the stub login: the
email_send table (V42__email_send.sql — V42 because BUG-162 took V41 on main), the send route and its check order, the four kinds, the idempotency key,
every outcome, the transaction boundary, the body cleaning, the Sent Items match on an ingested copy, the audit
events and the 45-second bound (EmailSend*IT). Items (d) and (e) come from WI-1.2’s and WI-1.3’s plans (the
Stage 7 consistency gate, 2026-10-12). FR-G1’s laptop-mode run (WI-2.1) proves the real sends before dev exists;
the rows below are what dev still owes after it.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) The four sends repeated on dev by CRM01, with the Sent Items match — a new message, a reply, a reply-all with one Cc removed, and a forward of a message with an attachment, sent from dev’s CRM; each arrives with its recipients (Cc and Bcc included, the removed Cc not), the quoted original and the forwarded attachment, and its email_send row becomes matched (email_message_id set) when the Sent Items copy is ingested |
Dev has no Entra sign-in or mailbox connection yet (§A-5); locally every Graph call is the stand-in’s, and the match is proved by seeding a copy with the draft’s Internet message id. Whether Microsoft’s Sent Items copy carries that id is first seen in FR-G1’s laptop run | spec FR-G1, FR-G2(a), FR-R2, AC-39; EmailSendService.java, EmailSendMatcher.java |
Connect CRM01’s mailbox on dev; send each of the four; check the recipient mailboxes and SELECT status, email_message_id FROM email_send for the four rows (all SENT, each matched). Laptop, not dev (2026-10-14, Dinil Gamage with Claude, F20-c-WI-2.1 R-2 to R-6): the four kinds were sent from CRM01 on a laptop through Graph and delivered with recipients as sent; the removed Cc did not receive the reply all; the forward carried the attachment (no byte compare). Every row was matched with the same Internet message id as the draft. crm03 had no mailbox, so CRM05 stood in. Still owed on dev. |
F20-c / F38 | open |
(b) Graph’s 429 and its real error codes on a refused send and on the draft steps — MailboxSendException’s classification (REFUSED with Graph’s error.code, THROTTLED with Retry-After, OUTCOME_UNKNOWN), the 429 EMAIL_SEND_RATE_LIMITED answer with Microsoft’s wait, and a refused send’s draft delete |
Proved only against the armed stand-in (EmailSendOutcomeIT) and hand-written Graph error bodies (GraphMailboxProviderSpec); the codes Graph really sends (ErrorInvalidRecipients, ErrorSendAsDenied, throttling) are documentation reads |
spec FR-G2(b), FR-S4, FR-S5, FR-M2; GraphMailboxProvider.sending |
On dev, send to an invalid recipient and record the row’s error_code and the 502; drive a burst until Graph answers 429 and confirm the API’s 429 carries Microsoft’s Retry-After and no draft is left in Outlook’s Drafts |
F20-c / F38 | open |
| © A sent email reaching an external mailbox — deliverability under Empire’s SPF and DKIM (R-12) | Only Empire’s real tenant signs and sends; the stand-in delivers nothing | spec FR-G2©; PRD R-12 | From dev, send to an external address outside empire.lk (a team member’s own mailbox); it arrives in the inbox, not spam, and its headers show SPF and DKIM pass |
F20-c / F38 | open |
(d) The ALB idle timeout is above F20-c NFR-3’s 45 seconds, and the dialog repeats with the same key on a load balancer’s own 502/504 page |
A send can take up to 45 seconds; the ALB default idle timeout is 60, but a lower value cuts the connection and the dialog sees a 504 page with no code (F20-c Wave 1 contract, client fact 3). The dialog’s handling of such a page is proved only against a stubbed fetch in WI-1.2’s component tests |
F20-c NFR-3; F20-c Wave 1 contract, client facts 3 and 5; WI-1.2 plan | Read the dev and production ALB idle timeouts (≥ 60 s); on dev, arm a slow send if possible, or block the backend response, and confirm the dialog repeats with the same key and settles on the stored outcome rather than offering Send again | F38 / F20-c | open |
(e) A Content-Security-Policy served for the CRM frontend keeps HTML email bodies styled and visible — an about:srcdoc frame inherits the parent page’s CSP, so a CSP later served on CloudFront (F38) applies inside WI-1.3’s SafeMessageBody frames too; it must keep style-src 'unsafe-inline' and img-src data: and allow the srcdoc frames |
No CSP is served in local browser runs, so the frames render under none; only a deployed frontend serves one | WI-1.3 plan (SafeMessageBody); F20-c FR-I3, AC-35 |
Once a CSP exists on dev or UAT, open an HTML message (the HTML_UNSAFE fixture’s shape) in the inbox view: it renders with its inline styles, no remote image loads, and the browser console shows no CSP violation for the frame Update 2026-10-27: the CRM’s draft policy, which keeps style-src 'self' 'unsafe-inline', img-src 'self' data: and frame-src 'self', is in forge-et-iaac#14 (open, not merged); this check runs on dev once it is applied (.forge/owed.md, “A Content-Security-Policy on both frontends”). Update 2026-10-29: #14 was merged and applied on 28 Oct; curl -sI https://dev.crm.empire.lk/ serves that policy, and the CRM’s sign-in page loads with no CSP violation. The check above, which needs CRM01 signed in with a connected mailbox, is what is left. |
F38 / F20-c | open |
POST /api/v1/mailbox/stub-send-failure is really absent on deployed APIs |
Proven as mapping absence inside a local no-profile context (MailboxCloudShapeIT) |
F20-c Wave 1 contract (stub-send-failure); MailboxStubController.java |
On a deployed API, an authenticated POST answers 404 (an unauthenticated one 401) Validated 2026-11-16 (F38-g WI-4.2, F20-c.3, walked 2026-11-06): as CRM03 holding RETEST 4.2 email user, POST /mailbox/stub-send-failure {} → 404 “No resource exists at the requested path.”; evidence: .forge/plans/F38-g/F38-g-retest-report.md F20-c.3 and the WI-4.2 record R-7. |
— | validated |
F20-g — Open/link tracking and scheduled send
Spec FR-G2 / AC-26 require these rows; F20-g-WI-2.1 (the scheduled-email-send schedule) adds them. Everything else in
F20-g runs and is proved locally against the LocalMailboxProvider stand-in, LocalStack and the stub login: the sweep
is run by calling its handler (WI-1.4’s tests), and tracking runs on the local backend’s address (spec decision 11).
Until row (a)'s schedule exists on an environment, scheduled emails there are never sent: they stay SCHEDULED, and
the Scheduled list marks each one Waiting to Send once it is more than 5 minutes past its time (FR-V4).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) The EventBridge schedule scheduled-email-send at rate(1 minute), UTC, in the group et-<env>, targeting the scheduler queue et-<env>-scheduler, payload {"jobKind":"scheduled-email-send","idempotencyKey":"scheduled-email-send#<aws.scheduler.scheduled-time>"} — on dev from forge-et-iaac#19 (F20-g-WI-2.1); on UAT and production with each overlay’s next apply from a dev that holds it, which needs a backend with F20-g wave 1 (.forge/owed.md, “Apply the UAT and production overlays with the scheduled-email-send schedule …”). Running cost: 43,200 invocations a month (60 × 24 × 30) against EventBridge Scheduler’s free allowance of 14,000,000 invocations a month, with USD 1.00 per million above it (https://aws.amazon.com/eventbridge/pricing/, read 2026-11-09), so none is charged; and about 86,400 SQS requests a month (a SendMessage and a DeleteMessage per firing, more with redeliveries; the worker’s receives already happen through long polling) against SQS’s free allowance of 1 million requests a month — which is the account’s, counted across all regions, shared by every queue and already used in part by the backend’s long polling — with USD 0.40 per million Standard-queue requests above it in US East (https://aws.amazon.com/sqs/pricing/, read 2026-11-09; the page’s table did not render, so the rate is the first-tier Standard figure from the price data that page loads), so at most about USD 0.04 a month whichever side of the allowance the account falls |
The sweep is a scheduler jobKind (AD #12), never @Scheduled; locally it runs only when its handler is called. Without the schedule, due rows stay SCHEDULED, which the list marks Waiting to Send. Once the schedule is on an environment, the backend deployed there must contain F20-g wave 1, or every firing is refused as an unknown job kind and dead-lettered; on dev, a forge-et-be branch deployed by hand must contain it (merge main into it first — F20-g-WI-2.1 plan R-8) |
F20-g spec decision 6, FR-L3, FR-G2(a), AC-26, AC-32, NFR-6; forge-et-be docs/messaging/scheduler-cloud-contract.md (the scheduled-email-send entry F20-g-WI-1.4 adds); forge-et-iaac base/scheduler/locals.tf |
After the merge’s apply, aws scheduler get-schedule --group-name et-dev --name scheduled-email-send --profile et-dev --region us-east-1 (a read) shows ENABLED, rate(1 minute), UTC, the target et-dev-scheduler and that Input; the backend log then has the sweep’s completion line (the name WI-1.4’s contract entry gives) for a firing’s key; a scheduled email on dev is sent within 2 minutes of its time (row (b)). The same read on UAT and production after their overlay’s apply. Update 2026-11-12: created on dev by forge-et-iaac#19 (merge 4b3d883e), applied by dev.yaml run 43176389232 (“Apply complete! Resources: 1 added, 0 changed, 0 destroyed”); get-schedule reads ENABLED, rate(1 minute), UTC, target et-dev-scheduler and that Input. A firing reaching the worker is seen only in part: the scheduler queue is empty and et-dev-scheduler-dlq holds 0, but /ecs/dev/backend has no sweep completion line, because the sweep logs one at INFO only when something was due (DEBUG otherwise) and nothing was scheduled — the line is still to be seen, in wave 3’s run. Update 2026-11-16: fired on dev, seen in F20-g-WI-3.1 R-7 (one email, the sweep line 10 s after its time) and R-9 (attempt 2: ten emails in one sweep of about 10 s, sent=10, dead-letter queue 0; attempt 1 on 2026-11-12 sent 7 of 10, BUG-418). UAT and production are still owed. |
Dinil (with Shyamin Ayesh); UAT and production Tharinda (F38-h, F38-i) | open (dev schedule created and seen firing; UAT and production still owed) |
(b) FR-G1’s run on dev by a person with CRM01 — tracked and untracked sends to an external mailbox, scheduled and cancelled emails, the scanners’ counts, and NFR-3’s measurements; and a tracked send’s row in Sent Items and its message view on CRM01 showing Sent, then Opened, which proves Graph returns internetMessageId on the live list and message reads (locally the stub stands in for it — F20-g-WI-1.3 R-4) |
A mail program cannot reach a laptop’s tracking address, the sweep’s schedule exists only in the cloud, and only a real mailbox shows what Graph returns | F20-g spec FR-G1, NFR-3, FR-G2(b), FR-V2, AC-21, AC-25, AC-31 | F20-g-WI-3.1’s run sheet; its record .forge/plans/F20-g/F20-g-WI-3.1-record.md holds every FR-G1 item, the Sent Items and message-view check, and the 95th percentiles Dev, validated (2026-11-16, Dinil Gamage, F20-g-WI-3.1 R-1 to R-10): R-1 to R-8 passed. R-9’s burst of ten failed once on 2026-11-12 (3 of 10 got no answer from Microsoft; BUG-418, open, cause not found) and passed on 2026-11-16 (10 of 10 within the minute, dead-letter queue 0), which Dinil accepted as enough. R-10’s 95th percentile of time_total missed NFR-3’s 300 ms from Sri Lanka (about 0.97 s and 1.18 s; the backend’s own time about 0.06 s), and its follow-up is the owed.md item “Decide NFR-3’s measure after F20-g’s wave-3 run”. The external mailbox was Gmail, not Outlook on the web (Decision 14), so FR-G1’s Outlook wording was not met; owed.md has the retest. Update 2026-11-16 (Dinil Gamage): BUG-418 was closed wontfix; NFR-3’s measure became the backend’s own time (spec Rev 3), so R-10 meets it; and Dinil dropped the Outlook-on-the-web retest — opens and clicks are shown as estimates, and an Outlook link scanner that clicked every link would show at once in the reps’ Clicked labels. |
Dinil (F20-g-WI-3.1, person-run) | validated |
© The tracking endpoints GET /api/v1/email/track/{token}/o and …/c/{n} reached from the internet through the load balancer on dev and production, and production’s ET_MAILBOX_PUBLIC_BASE_URL set |
Locally the links point at the local backend; only a deployed load balancer is reached by a recipient’s mail program, and production’s task definition does not exist yet (F38-i) | F20-g spec FR-T3, FR-T4, decision 11, NFR-1, FR-G2©; §D’s ET_MAILBOX_PUBLIC_BASE_URL row |
From outside AWS, curl -s -D - -o /dev/null https://api.dev.crm.empire.lk/api/v1/email/track/not-a-token/o answers 200, an image type, Cache-Control: no-store and no Set-Cookie; …/track/not-a-token/c/1 answers 404. The same against api.crm.empire.lk once production runs, after reading ET_MAILBOX_PUBLIC_BASE_URL on production’s task definition (a read). The ALB has no access logs (aws elbv2 describe-load-balancer-attributes shows access_logs.s3.enabled false), because spec decision 3 and NFR-2 keep no IP address or user agent of a tracking request. Dev, checked 2026-11-16 (Dinil Gamage, F20-g-WI-3.1 R-11): …/track/not-a-token/o answered 200, image/gif, Cache-Control: no-store, no Set-Cookie; …/c/1 answered 404; the ALB’s access logs are off. Production is still owed. |
Dinil; production with Tharinda (F38-i) | open |
| (d) The effect of Microsoft Defender Safe Links and other scanners on the open and click counts | Only real mail through real filters shows what is still counted despite the 60-second rule (decision 10) | F20-g spec decision 10, FR-T5, FR-G2(d); PRD R-12 | After the first weeks of real use on production, compare the counted opens and clicks against what the senders know of their recipients, record whether scanners still count, and raise a change if the estimate needs a different rule. Update 2026-11-16: F20-g-WI-3.1 R-2 saw nothing counted (SENT, both counts 0) 2 minutes 24 seconds after the send, before anyone opened the email; this is an early observation, and the check after weeks of real use is still owed. |
Dinil | open |
F20-h — Email attachments
Spec FR-G2 / AC-28 require the shipping PR to add these; F20-h-WI-1.1 (the uploads and the send path with files) adds
them, items (e) to (g) from its plan’s subtask 11. Everything else in F20-h’s backend runs and is proved locally against
the LocalMailboxProvider stand-in, LocalStack’s versioned S3 bucket and the stub login: the upload API and its check
order, the Office recognition with its bounds, the email_upload owner and its visibility, deletion with a delete
marker, V44__email_attachments.sql, the message-attachment read, the send with files (direct and upload session),
a forward’s removals, the stored deadline and the token count, deletion after the send, the hourly clean-up run by
hand, and the audit events (EmailUpload*IT, EmailSendFiles*IT, EmailUploadCleanupIT). FR-G1’s laptop-mode run
(WI-2.1) proves real sends with files before dev exists; the rows below are what dev still owes after it.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) The attachments bucket on dev, versioned, with no lifecycle rule that expires non-current versions or delete markers — and FR-G1’s sends repeated on dev by CRM01 |
Dev has no attachments bucket yet (environments-and-identity.md §4; ET_FILES_STORAGE_BUCKET a placeholder). The spec’s decision 3 — a deleted file stays recoverable — rests on versioning and on no rule expiring the kept version; LocalStack’s bucket is versioned by the test base, which proves the code, not the bucket. Update 2026-10-21: et-dev-attachments exists since forge-et-iaac#3 was applied on 20 Oct — versioning Enabled, and get-bucket-lifecycle-configuration answers NoSuchLifecycleConfiguration; the sends on dev wait on ET_FILES_STORAGE_BUCKET (F38-d FR-6). |
spec decision 3, FR-G1, FR-G2(a), FR-R2; proposal §A.5, §A.6.3 | Read the dev bucket’s versioning (Enabled) and lifecycle configuration (no NoncurrentVersionExpiration, no ExpiredObjectDeleteMarker). From dev’s CRM as CRM01, send a new email with a PDF under 3 MB and an .xlsx over 3 MB, a reply with a .docx, and a forward of a message with two attachments with one removed; each arrives with its files intact, the removed original absent; after the send each file’s attachment.deleted_at is set and aws s3api list-object-versions on its key shows a delete marker as the latest version with the uploaded version kept. Laptop, not dev (2026-10-16, Dinil Gamage, F20-h-WI-2.1 R-3 to R-9): FR-G1’s three real sends with files from CRM01, and two more (a forward with new files and a removal; a file of 3,141,628 bytes attached directly), delivered intact (checksums equal), the forwards’ removed original absent, every send matched in Sent Items, and the sent files deleted with a delete marker over a recoverable earlier version on the laptop’s versioned LocalStack bucket. Still owed on dev. Update 2026-10-21 (F38-b WI-2.1): et-dev-attachments has versioning Enabled and no lifecycle configuration (NoSuchLifecycleConfiguration), and the task role holds s3:DeleteObject and s3:DeleteObjectVersion on its objects; the send on dev is still owed; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-5. |
F20-h / F38 | open |
| (b) The load balancer’s idle timeout is above 130 seconds on every environment | A send with files may take its deadline plus 5 seconds — up to 125 (F20-h FR-S4). Below that the browser gets the load balancer’s own 504 page and the send finishes behind it, answered later through F20-c’s same-key repeat |
spec FR-S4, FR-G2(b); F20-h Wave 1 contract, ### Changes to F20-c's send rules; F20-c contract client fact 3 |
Read the dev, UAT and production ALB idle timeouts (≥ 130 s, or the Terraform value); on dev send ten megabytes of files and confirm the dialog gets the 201 itself, not a 504 page |
F38 / forge-et-iaac |
open |
© The scan-on path — PENDING, INFECTED, an infected file deleted before its verdict (AC-35), and GuardDuty’s verdict on Office packages |
GuardDuty is off on every environment (F38-j), and locally the stub scanner’s verdicts are delivered by hand (EmailUploadScanStatusIT, EmailUploadInfectedAfterDeleteIT); whether GuardDuty opens an .xlsx, .docx and .pptx and what it answers is unseen |
spec FR-G2©, AC-5, AC-35; FS-11 FR-S3 | When scanning is switched on for an environment: upload a PDF and an .xlsx from the dialog — each reads PENDING, then CLEAN, and sends; upload a PDF carrying the EICAR string, delete it before the verdict, and confirm the row becomes INFECTED and the key lists no version and no delete marker |
F20-h / F38 | open |
(d) The email-upload-cleanup EventBridge schedule, firing hourly on dev |
The FS-9 cloud half does not exist; locally the clean-up runs only when its handler is called (EmailUploadCleanupIT). Without it, files of cancelled dialogs and of sends the after-send deletion missed stay live. Update 2026-10-21: the email-upload-cleanup schedule exists on dev at rate(1 hour) since forge-et-iaac#3 was applied on 20 Oct; its firing is still to be seen. |
spec FR-R3, FR-G2(d), AC-16; forge-et-be/docs/messaging/scheduler-cloud-contract.md § “Email file clean-up” |
Terraform the schedule per the contract (rate(1 hour), the Input with email-upload-cleanup#<aws.scheduler.scheduled-time>, the scheduler queue, its IAM shape — with aws:SourceArn the schedule group’s ARN, F38-b spec Rev 2); on dev, after a firing the backend log shows one Email Upload Cleanup Completed line, and a file uploaded and abandoned more than 24 hours earlier has deleted_at set with an attachment.deleted event by SYSTEM email-upload-cleanup Update 2026-10-21 (F38-b WI-2.1): email-upload-cleanup (rate(1 hour), UTC) targets the scheduler queue with the contract’s Input, first fired at 2026-10-20 18:35 UTC and hourly since with no target error, and its messages reached the queue; the deleted file is F38-f’s; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. Update 2026-10-26 (F38-f WI-1.1): 26 Email Upload Cleanup Completed lines at INFO in the 26 hours read (the latest 2026-10-26 04:35 with candidates=0, deleted=0, failed=0) and no Email Upload Cleanup Failed line; the abandoned file deleted with its attachment.deleted event is read through the F38-f tunnel by F38-f WI-4.1; evidence: §B F38-f “F38-f evidence — AC-8 (d)”. Update 2026-10-26 (F38-f spec Rev 4): no tunnel is built; the deletion is read from the deleted= count of a Completed line and the attachment.deleted event from the audit service’s DEBUG line, in F38-f WI-2.6’s log window, if a run with a candidate falls inside it — otherwise this half stays open with that reason. |
F38 / forge-et-iaac |
validated — producer half (F38-b WI-2.1, 2026-10-21) and the worker’s log line (F38-f WI-1.1, 2026-10-26); the database half open — read from the logs in F38-f WI-2.6’s window (spec Rev 4) |
(e) Microsoft’s throttling of attachment calls and parts — the per-mailbox 150 MB of uploads in 5 minutes, and a 429 on an attach call or an upload part |
Proved only against the armed stand-in; Microsoft’s upload-volume limit is not modelled by the per-connection limiter, which counts calls, not bytes (WI-1.1 plan ## Notes, Q-2) |
spec FR-S5, Q-2; WI-1.1 plan Decision 16 and ## Notes (Q-2); Microsoft Graph throttling limits § Outlook service limits |
On dev, send about six maximum-size emails from one mailbox inside five minutes; a throttled send ends FAILED with 429 EMAIL_SEND_RATE_LIMITED and Microsoft’s Retry-After, nothing is sent, and no draft is left in Outlook’s Drafts |
F20-h / F38 | open |
(f) The cross-site upload — an upload from the dev CRM origin to api.dev.crm.empire.lk through the dialog’s progress-reporting request carries the session cookie and X-XSRF-TOKEN, reports progress, and a 413 shows as the row’s “could not be uploaded” |
Locally the frontend and the API share an origin through the Vite proxy; only a deployed pair is cross-site (AD #33) | WI-1.2 plan Decision 1; F20-h Wave 1 contract, client-side fact 1; AD #33 | On dev, attach a file in the dialog: the request to api.dev.crm.empire.lk answers 201 with the cookie and token sent, the progress bar advances, and a file over the container’s limit shows as refused on its row Validated 2026-11-16 (F38-g WI-4.2, F20-h.5, run in the built-in browser as CRM01 with the IT Admin role): each upload from the dialog, POST https://api.dev.crm.empire.lk/api/v1/email/attachments → 201 with the session cookie and X-XSRF-TOKEN, its progress bar advancing; a 26 MB file and a file taking the total over 25 MB were refused on their rows in the dialog with the reason and no request sent, so the server’s own 413 was not seen; evidence: .forge/plans/F38-g/F38-g-retest-report.md F20-h.5 and the WI-4.2 record R-8. |
F20-h / F38 | validated |
(g) The task role’s new S3 actions — s3:GetObject (the send path’s read of a file) and s3:DeleteObject with no version id (the delete marker) on attachments/email_upload/* |
FS-11’s task-role row covers put, version listing and version deletion; LocalStack enforces no IAM, so a missing grant is seen only on a deployed environment | spec Constraints (“Adds to the file substrate”); WI-1.1 plan Decision 8; ObjectStore.read, ObjectStore.deleteKeepingVersions |
On dev, Remove on an uploaded file answers 204 and the key’s version list shows a delete marker as latest; a send with a file answers 201 and the file reaches the recipient Update 2026-10-21 (F38-b WI-2.1): the task role’s s3 policy holds s3:GetObject and s3:DeleteObject (with s3:PutObject and s3:DeleteObjectVersion) on et-dev-attachments/*; the delete and the send on dev are still owed; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-5. |
F38 / forge-et-iaac |
open |
POST /api/v1/mailbox/stub-attachments-failure and GET /api/v1/mailbox/stub-last-send are really absent on deployed APIs |
Proven as mapping absence inside a local no-profile context (MailboxCloudShapeIT) |
F20-h Wave 1 contract (the two stub routes); MailboxStubController.java |
On a deployed API, an authenticated request to either answers 404 Validated 2026-11-16 (F38-g WI-4.2, F20-h.2, walked 2026-11-06): as CRM03 holding RETEST 4.2 email user, POST /mailbox/stub-attachments-failure → 404 and GET /mailbox/stub-last-send → 404; evidence: .forge/plans/F38-g/F38-g-retest-report.md F20-h.2 and the WI-4.2 record R-8. |
— | validated |
F20-i — Live inbox (all six waves merged by 2026-11-20, laptop walk F20-i-WI-5.1; feature done 2026-11-20)
Spec FR-25 and AC-43d require these rows; F20-i-WI-5.1 writes them. The laptop-mode walk of 2026-11-19 (record .forge/plans/F20-i/F20-i-WI-5.1-record.md) proved the automatic connect and its refusal, the granted scopes and the live reads against CRM01’s real mailbox, and settled the Graph behaviours the waves left to it (record R-20): what still needs Graph itself without a deployed environment is in .forge/owed.md, not here. A laptop result below is partial evidence for its row, never a close.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
| 1. The two-second first page | The target is measured on dev against a mailbox of a few thousand messages; a laptop talks to Graph from another network and with a small mailbox. Laptop, not dev (2026-11-19, Tharinda Rajapaksha with Claude, F20-i-WI-5.1 R-7): the Inbox’s first page took 638 ms on a reload, CRM01 holding 46 Inbox messages | spec NFR-1, FR-25 | On dev, with a test mailbox of a few thousand messages connected, reload /email and read the …/messages request’s time in the Network panel; it is under 2 s |
Tharinda | open |
| 2. New-mail notices from Graph change notifications — and the folder-tree reads F20-b’s notification path makes under real mail flow (at most one tree read per connection per five minutes per task, never a re-read for an unknown folder id), and whether Graph sends a notification for a message in a hidden folder | Graph notifications cannot reach localhost; locally the delta poll fired by hand is the only path in. Laptop, not dev (2026-11-19, F20-i-WI-5.1 R-15): a message sent from CRM02 raised the New mail notice 37 s after the delta poll was fired by hand, and Show put it on top |
spec FR-11, FR-25; WI-4.1 Decision 10, escalation E-2 (a) | On dev, send a message to a connected test mailbox: the New mail notice shows within a minute with no poll fired; watch ingest’s Graph calls while mail arrives for the tree-read limit; move a message into a hidden subfolder and note whether a notification arrives | Tharinda | open |
| 3. The per-connection limiter and the interactive priority under real Graph load, including the 20-parallel-request check | Graph’s own throttling and the limiter’s behaviour against it are seen only with real traffic from a deployed backend | spec NFR-2, FR-25; .forge/test-strategy.md § Microsoft Graph |
On dev, run the 20-parallel-request check against a connected mailbox while a background read runs; interactive reads answer first and no request fails with Graph’s 429 passed through |
Tharinda | open |
| 4. The automatic connect on dev’s own reply URL | Laptop mode uses the two localhost:8080 reply URLs; dev’s reply URL and its cross-site cookies are only exercised there (AD #33). Laptop, not dev (2026-11-19, F20-i-WI-5.1 R-3, R-5): the prompt=none connect returned with no Microsoft page; a refused one showed Try again once per session, and Try again connected |
spec FR-17, FR-25 | On dev, grant a test user email.manage, sign out and in: the mailbox connects with no Microsoft page; repeat the refusal recipe of F20-i-WI-5.1 step 6 |
Tharinda | open |
| 5. Reply and forward of a live message delivered to an outside address | A laptop run sends only within Empire’s tenant. Laptop, not dev (2026-11-19, F20-i-WI-5.1 R-14): a reply and a forward by Graph id were delivered within the tenant, the forwarded attachment byte-identical | spec FR-14, FR-25 | On dev, reply to and forward a live message to an outside test address; both arrive, the forward with its attachments | Tharinda | open |
6. The live attachment download fetched cross-site from api.* with credentials |
Locally the stub and the laptop walk run same-site; only a deployed pair is cross-site (AD #33). Laptop, not dev (2026-11-19, F20-i-WI-5.1 R-13): the download saved a byte-identical file, same-site | spec FR-15, FR-25; AD #33; the lead’s decision of 2026-11-04 | On dev, open a message with a file attachment at crm.dev… and press Download, then check: the file saves under its name, and the CRM tab does not navigate; the request to api.dev.crm.empire.lk/api/v1/mailbox/live-messages/{id}/attachments/{attachmentId} carries the SESSION cookie and passes CORS — Access-Control-Allow-Origin names the CRM origin and Access-Control-Allow-Credentials is true, on the 200 and on an error response alike, or the browser hides a refusal’s problem body; a refused download (for example after signing out in another tab) shows the inline error with Try again. Repeat in Safari (its third-party cookie rules) |
Tharinda | open |
| 7. A conversation of more than 101 messages read on dev | A test mailbox holds no conversation that long. Laptop, not dev (2026-11-19, F20-i-WI-5.1 R-16): with the cap set to two, the read followed Graph’s @odata.nextLink for /me/messages to a second page (pages=2, no Conversation Read Stopped line), so the link keeps the conversation’s $filter |
forge-et-be#370 review; spec FR-10, FR-25 | On dev, open a conversation of more than 101 messages; the backend log’s Live Inbox Conversation Read line shows more than one page and truncated=true |
Tharinda | open |
| 8. Two mailbox shapes no test mailbox has: a message in a hidden subfolder, and a mailbox without Archive | The walk’s direct Graph checks (record R-17) needed an administrator to approve Graph Explorer, and the lead decided on 2026-11-20 not to ask (.forge/engagement-log.md). These two are the ones a rep could meet: a message in a hidden subfolder of a shown folder may be listed in All mail and then answer “not found” when opened (forge-et-be#369 review), and a mailbox without Archive depends on Graph answering the well-known archive lookup with 404 (WI-2.1 E-5) |
forge-et-be#369 review; WI-2.1 E-5 | Watch for them on dev and in UAT with real mailboxes: an All-mail message that opens to “not found”, or a rep whose folder list fails or lacks Archive. Either is filed as a bug when seen | Tharinda | open |
F22 — Contact sync
Spec FR-17 / AC-35 require these rows; F22-WI-2.1 adds them, replacing F22’s row in the backlog table below. Everything else in F22 runs and is proved locally: the local/test stub address book and its stub routes (spec FR-16) drive every read, the sweep runs through its stub route or its handler, and GraphMailboxProvider’s contact calls are proved against a mock Graph server. Laptop mode (spec FR-18, F22 wave 3’s walk) can prove the real Graph read first; that is partial evidence for rows (a), (e)–(h) and (j)–(l), never a close. Until row (b)'s schedule exists on an environment, nothing is read there each hour, so the From Outlook list fills only on Check Outlook now. BUG-067’s scope half (its row in §B F20-a) becomes checkable once F22 wave 1 is on dev, because F22’s read asks for Contacts.ReadWrite, which is in MailboxScopes.REQUESTED but not REQUIRED; the check stays with that row’s owner (F38).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
| (a) The real Graph contacts delta read against a real mailbox — the default Contacts folder and every folder under it, a first full read, then a delta round | Locally the stub address book answers; the Graph calls are proved only against a mock server | spec FR-1, FR-17, FR-18 | On dev once F22 wave 1 is there, with a test mailbox (CRM0x) connected and granted Contacts.ReadWrite, holding Outlook contacts the CRM lacks: after a run, each with an address is listed on /contacts/from-outlook; Add creates the contact; changing an empty-in-the-CRM field of a matched contact in Outlook fills it after the next run; deleting a suggested contact in Outlook removes it from the list after the next run. Record the mailbox, the counts and the run times here, never an address |
Tharinda (F22) | open |
(b) The EventBridge schedule contact-sync-sweep at rate(1 hour), UTC, in the group et-<env>, targeting the scheduler queue et-<env>-scheduler, payload {"jobKind":"contact-sync-sweep","idempotencyKey":"contact-sync-sweep#<aws.scheduler.scheduled-time>"}, and its first firing on dev — on dev from forge-et-iaac#30 (F22-WI-2.1); on UAT and production with each overlay’s next apply from a dev that holds it, which needs a backend with F22 wave 1 (.forge/owed.md, “Apply the UAT and production overlays with the contact-sync-sweep schedule …”). Running cost: 720 invocations a month (24 × 30) and about 1,440 SQS requests (a SendMessage and a DeleteMessage per firing), against EventBridge Scheduler’s free allowance of 14,000,000 invocations a month and SQS’s free allowance of 1 million requests a month (the account’s, shared by every queue), with USD 1.00 and USD 0.40 per million above them, as §B F37-a row (a) records them (read 2026-11-15) — at most about USD 0.0013 a month even with no free allowance left, under a fifth of a cent (spec NFR-1) |
The sync is a scheduler jobKind (AD #12), never @Scheduled; locally it runs only through its stub route or its handler. Once the schedule is on an environment, the backend there must contain F22 wave 1, or every firing is refused as an unknown job kind and dead-lettered; on dev, a forge-et-be branch deployed by hand must contain it (.forge/team-guide.md § Environments) |
spec FR-2, FR-17, NFR-1, NFR-2, AC-31; forge-et-be docs/messaging/scheduler-cloud-contract.md (the contact-sync-sweep section F22-WI-1.2 adds); forge-et-iaac base/scheduler/locals.tf |
After the merge’s apply, aws scheduler get-schedule --group-name et-dev --name contact-sync-sweep --profile et-dev --region us-east-1 (a read) shows ENABLED, rate(1 hour), UTC, the target et-dev-scheduler and that payload as its Input; within 65 minutes the backend log has the run’s completion line and no UnknownScheduledJobKindException for the kind, and et-dev-scheduler-dlq gains nothing. The same reads on UAT and production after their overlay’s apply |
Tharinda (with Shyamin Ayesh); UAT and production Tharinda (F38-h, F38-i) | open |
© The queue Check Outlook now uses — none added. Check now is a third message kind, CONTACT_CHECK, on the existing mailbox ingest queue (F22-WI-1.1’s plan, Decision 1), available wherever ET_MESSAGING_MAILBOX_INGEST_QUEUE is set: dev and UAT now (et-dev-mailbox-ingest, et-uat-mailbox-ingest in forge-et-be .aws/td.dev.json and td.uat.json), production once F38-i sets it in production’s task definition (§D, ET_MESSAGING_MAILBOX_INGEST_QUEUE). Where it is unset, the button is hidden and the refresh route answers 503 |
The queue and the press that reaches it exist only on a deployed environment; locally LocalStack’s queue carries it | spec FR-12, FR-17, NFR-2, AC-24 | On dev with row (a)'s mailbox: the status route reports Check now available; pressing Check Outlook now shows the check running, the list refreshes within a minute, the backend log shows the run under the job name contact-sync-sweep, and et-dev-mailbox-ingest-dlq gains nothing. In production, after F38-i’s task definition, the status route reports it available |
Tharinda (F22); production Tharinda (F38-i) | open |
| (d) The run time of a first full read of a large address book on dev | Only a real mailbox of thousands of contacts, Graph’s latency and the limiter show how many runs a first read takes | spec FR-2, FR-17, NFR-4 | On dev, a test mailbox holding at least 2,000 fictional contacts (imported into Outlook from a file of invented names on the team’s own test domain, never a real person’s data) is connected; from the runs’ completion lines, record per run the pages read, the time taken and whether the budget stopped it, and the number of runs until every folder holds a delta link; no run passes half of et.messaging.visibility-timeout plus one page, no Graph 429 is passed through, and no contact is suggested twice |
Tharinda (F22) | open |
(e) The default Contacts folder’s id, read from the top-level folders’ parentFolderId, with the /me/contacts?$top=1&$select=parentFolderId fallback |
F22-WI-1.1 proves it only against a mock Graph server; only real Graph shows the folder ids it reports | spec FR-1, FR-17, FR-18; F22-WI-1.1’s plan, ## Notes item 1 (a) |
F22 wave 3’s laptop walk, on a CRM0x mailbox with no subfolder and then with one: the default folder’s contacts appear on the From Outlook screen each time. Then on dev, the first contact-sync-sweep run’s log (folder count, default resolved) |
Tharinda (F22) | open |
| (f) The system-folder names F22-WI-1.1’s Decision 4 leaves out | F22-WI-1.1 proves it only against a mock Graph server; only a real mailbox shows the folder names Graph returns | spec FR-1, FR-17, FR-18; F22-WI-1.1’s plan, ## Notes item 1 (b) |
F22 wave 3’s laptop walk records the folder names the real list returns (in its record, not in a log) and confirms no recipient-cache address became a suggestion; a name the list does not cover is a spec revision under FR-18 | Tharinda (F22) | open |
(g) A 400, 404 or 410 on a stored delta link starts that folder afresh on the next run |
F22-WI-1.1 proves it only against a mock Graph server; only real Graph shows how it answers a stale or invalid delta link | spec FR-1, FR-17, FR-18; F22-WI-1.1’s plan, ## Notes item 1 © |
On dev, with the lead’s approval of the database write, set one stored contact_sync_folder.link’s $deltatoken to an invalid value; the next run logs a dropped link and a first read of that folder, and no suggestion repeats |
Tharinda (F22) | open |
| (h) Immutable ids on the contacts delta: a contact moved between folders keeps its id | F22-WI-1.1 proves it only against a mock Graph server; only real Graph shows how it reports a move | spec FR-1, FR-17, FR-18; F22-WI-1.1’s plan, ## Notes item 1 (d) |
F22 wave 3’s laptop walk, twice: move a suggested contact from Contacts into a subfolder, and after the next run it is still one suggestion with its state kept; dismiss a suggestion, move its contact into a subfolder, and after the next run it is still dismissed and not shown |
Tharinda (F22) | open |
(i) The page size Graph honours for Prefer: odata.maxpagesize=100 on a contacts delta |
F22-WI-1.2 proves the run only against the stub address book or a mock Graph server; only real Graph shows the page size it honours | spec FR-1, FR-2, FR-7, FR-17; F22-WI-1.2’s plan, ## Notes item 2 |
On dev, from the first runs’ DEBUG page counts for row (a)'s or row (d)'s mailbox, every page holds at most 100 contacts (fewer is fine); record the largest page seen | Tharinda (F22) | open |
| (j) A contact in a newly created Outlook folder appears after the next hourly run, because the folders are listed on every run | F22-WI-1.2 proves the run only against the stub address book or a mock Graph server; only real Graph shows how it reports a new folder | spec FR-1, FR-2, FR-7, FR-17; F22-WI-1.2’s plan, ## Notes item 2 |
On dev, create a folder in row (a)'s mailbox and add a contact the CRM lacks to it; after the next hourly run (not Check Outlook now) it is listed on /contacts/from-outlook |
Tharinda (F22) | open |
| (k) A contact deleted in Outlook leaves the From Outlook list after one run | F22-WI-1.2 proves the run only against the stub address book or a mock Graph server; only real Graph shows how it reports a deletion | spec FR-1, FR-2, FR-7, FR-17; F22-WI-1.2’s plan, ## Notes item 2 |
On dev, delete a suggested contact in Outlook; after the next run it is no longer listed | Tharinda (F22) | open |
| (l) A dismissed contact moved between Outlook folders stays dismissed | F22-WI-1.2 proves the run only against the stub address book or a mock Graph server; only real Graph shows how it reports a move, and whether the removal and the add arrive in the same run | spec FR-1, FR-2, FR-7, FR-17; F22-WI-1.2’s plan, ## Notes item 2 |
On dev, dismiss a suggestion, move its contact to another folder, and after the next run it is still dismissed and not shown, also when the move’s removal and add are read in different runs (the laptop walk’s row (h) checks it first) |
Tharinda (F22) | open |
F26-a — Standards and price substrate
Everything else in F26-a runs and is proved on the local Postgres (Testcontainers) with no AWS, Entra, Bedrock or SES call: the three tables, their constraints and triggers, the BS- allocator, the as-of and history reads, the opt-in build check, the NFR-1 query-plan proof, both price endpoints, the PriceModel bean, the no-suggestion AuctionPriceProvider and its guard, and the local seed (spec FR-D3). The one item below moved here from “Nothing cloud-gated” with spec Rev 3.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
NFR-2 response times, measured on dev — FR-A1 GET /api/v1/buying-standards/{id}/pricing under 100 ms and FR-A2 …/prices (one page) under 150 ms, 19th of 20 calls, at the NFR-1 four-year volume |
The binding measurement is on the environment the application runs in (spec Rev 3). On the developer’s laptop the per-request cost dominates: endpoints F26-a does not touch took a median of 43–50 ms there, and two local readings each breached one FR-A1 shape by about 10 ms (110.5 ms, 110.9 ms) while the database part of the read was about 0.2 ms | spec NFR-2, AC-24, Rev 3; F26-a-WI-2.1-plan.md D18 (the method) and ## Progress subtask 6 (the local readings and the diagnostic) |
On dev, load src/test/resources/db/testfixture/pricing-volume.sql into a throwaway copy of the database (never the shared dev data — the fixture adds 35,000 standards and 300,000 prices), or into a dev-shaped database behind the same ALB and task size. Sign in as a user holding buying_standard.view, pick a VOLUME-% standard that has prices by SQL, and run D18’s method from a host in the same region: warm every shape with 5 calls, then 20 measured calls per shape for FR-A1 with and without asOf and FR-A2 default and sort=validFrom,asc. Pass when every call answers 200 and the 19th value is under the limit. Record the figures in F26-a’s findings; a breach fails NFR-2 and is filed as a BUG against F26-a. Remove the fixture afterwards |
F38 (the §C-7 retest) | open |
F26-b — Standards book on the account
Everything else in F26-b runs and is proved on the local Postgres (Testcontainers) and the local frontend, with no AWS, Entra, Bedrock, SES or scheduler call: the buying_standard_certification join and its indexes, the standard writes and their audit events, cost entry, the account-context read, the book list, the client lookup, the cost list, the NFR-1 query plans, the local seed, the Pricing page, the account panel and the picker (spec FR-D3). The one item below is spec FR-D3’s.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
NFR-2 response times, measured on dev — FR-R1 GET /api/v1/accounts/{accountId}/buying-standards and FR-R4 GET /api/v1/buying-standards (one page of 25) under 150 ms each; a picker search (FR-R1 with q) under 200 ms; FR-W1 POST /api/v1/buying-standards, FR-W2 PATCH …/{id} and FR-W4 POST …/{id}/costs under 300 ms each — the 19th of 20 calls after a warm-up per request shape, at the NFR-1 volume |
The binding measurement is on the environment the application runs in: on a developer’s laptop the per-request cost dominates these reads (F26-a Rev 3, .forge/test-strategy.md § Performance Baselines, .forge/lessons.md L-084), so a local reading over a limit does not by itself fail NFR-2 |
spec NFR-2, FR-D3, AC-35, AC-37; F26-b-WI-1.4-plan.md (the local readings, recorded in its ## Progress) |
On dev, load src/test/resources/db/testfixture/pricing-volume.sql (F26-b’s revision: standards spread across accounts, with held certifications) into a throwaway copy of the database, never the shared dev data. Time FR-R1 on Pricing Volume Account (500 standards) as a user holding account.view and account.all, and the book list, a picker search and the three writes as a buying_standard.manage holder: 5 warm-up calls, then 20 measured calls per shape, from a host in the same region. Pass when every call answers its success status and the 19th value is under its limit. Record the figures in F26-b’s findings; a breach fails NFR-2 and is filed as a BUG against F26-b. Remove the fixture afterwards |
F38 (the §C-7 retest) | open |
F26-f — Tea Room review, edit and publish
Everything else in F26-f runs and is proved on the local Postgres (Testcontainers) with the stub login, with no AWS, Entra, Bedrock, SES or scheduler call: the V39__price_withdrawal.sql migration (the withdrawn state, its columns, withdrawn_is_complete, withdrawal_only_when_withdrawn and the two extended triggers), the round summary and rows reads, adding, replacing and removing a draft, publishing a round, withdrawing a price and a round, the round lock, the audit events, the local seed (sale 2030 / 1, standards 90201 – 90206, the round_manage_test and round_publish_test principals) and the volume round fixture (spec FR-25). The NFR-1 publish and round-withdraw timings (spec AC-27) are taken on the local stack by the spec’s own rule, not on dev. The one item below is spec FR-25’s.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
NFR-2 response times, measured on dev — FR-3 GET /api/v1/auction-sales/{saleId}/round under 150 ms; FR-4 GET …/{saleId}/prices (one page of 25) under 300 ms, with and without q and sorted by changePct; FR-6 POST …/{saleId}/draft-prices and FR-7 DELETE /api/v1/draft-prices/{priceId} under 300 ms each — the 19th of 20 calls after a warm-up per request shape, at the NFR-1 volume (a round of 1,450 drafts on the four-year price history) |
The binding measurement is on the environment the application runs in: on a developer’s laptop the per-request cost dominates these reads (F26-a Rev 3, .forge/test-strategy.md § Performance Baselines, .forge/lessons.md L-084), so a local reading over a limit does not by itself fail NFR-2 |
spec NFR-2, FR-25, AC-28; F26-f-WI-1.1-plan.md ## Notes (the commands, and the local readings the orchestrator records in its ## Progress) |
On dev, load src/test/resources/db/testfixture/pricing-volume.sql and then src/test/resources/db/testfixture/pricing-round-volume.sql into a throwaway copy of the database, never the shared dev data. As a user holding buying_standard.manage, time the summary and the rows page of sale 2104 / 1 (default, q=VOLUME-0007, sort=changePct,desc), and a draft add and its remove on a sale created for the purpose, as the plan’s ## Notes describe — 5 warm-up calls, then 20 measured calls per shape, from a host in the same region. Pass when every call answers its success status and the 19th value is under its limit. Record the figures in F26-f’s findings; a breach fails NFR-2 and is filed as a BUG against F26-f. Remove the fixtures afterwards |
F38 (the §C-7 retest) | open |
F26-g — Price expiry roll and manual override
Everything else in F26-g runs and is proved on the local Postgres (Testcontainers) and LocalStack with the stub login, with no AWS, Entra, Bedrock or SES call: the daily price-expiry-sweep handler, proved through the LocalStack scheduler queue (PriceExpirySweepIT) and run by hand on local (the backend README’s “Running the price expiry sweep by hand”), the sale-list members and the date edit, the pricing warnings, not in this round and carry forward, the correction, the import create’s saleStartsOn re-check, the audit events, the local seed (standards 90301 and 90302) and the two volume fixtures; the NFR-1 sweep and carry-forward timings (spec AC-27) and the local NFR-2 readings (AC-28) are taken on a local stack. The two items below are spec FR-24’s.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The EventBridge Scheduler schedule for price-expiry-sweep — cron(5 0 * * ? *) with the schedule time zone Asia/Colombo, Input {"jobKind": "price-expiry-sweep", "idempotencyKey": "price-expiry-sweep#<aws.scheduler.scheduled-time>"}, target the scheduler queue, the IAM shape of the scheduler contract (with aws:SourceArn the schedule group’s ARN, F38-b spec Rev 2) |
The FS-9 cloud half does not exist; locally nothing schedules the sweep, so it runs only when a message is put on the queue by hand. Without the schedule, prices stay live past their auction and the Pricing page’s sweep_overdue warning is the only signal. Update 2026-10-21: the schedule exists on dev since forge-et-iaac#3 was applied on 20 Oct — cron(5 0 * * ? *), Asia/Colombo, the Input above (aws scheduler get-schedule); its first run is still to be seen. |
spec FR-1, FR-24, AC-32; forge-et-be/docs/messaging/scheduler-cloud-contract.md § “Price expiry sweep” |
Terraform the schedule per the contract’s expression, time zone, payload and IAM shape. On dev, after a firing, the backend log shows one Price Expiry Sweep Completed line; with prices on dev that no publish replaced, their buying_standard.price_validity_closed events and one auction_sale.prices_lapsed are recorded as SYSTEM price-expiry-sweep, and the sweep_overdue warning is gone Update 2026-10-21 (F38-b WI-2.1): price-expiry-sweep (cron(5 0 * * ? *), Asia/Colombo) targets the scheduler queue with the contract’s Input, fired at 2026-10-20 18:35 UTC and again at 2026-10-21 18:35 UTC with no target error, and its messages reached the queue; the sweep’s own result on dev is F38-f’s; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4. Update 2026-10-26 (F38-f WI-1.1): one Price Expiry Sweep Completed line at INFO in the 26 hours read, 2026-10-25 18:35 UTC (closeDate=2026-10-26, closed=0, held=0); the sweep’s price_validity_closed events, prices_lapsed and the sweep_overdue warning are read through the F38-f tunnel by F38-f WI-4.1; evidence: §B F38-f “F38-f evidence — AC-8 (d)”. Update 2026-10-26 (F38-f spec Rev 4): no tunnel is built; the sweep’s audit events are read from the audit service’s DEBUG line if a run with a price to close falls inside F38-f WI-2.6’s log window — otherwise this half stays open with that reason. |
F38 / forge-et-iaac |
validated — producer half (F38-b WI-2.1, 2026-10-21) and the worker’s log line (F38-f WI-1.1, 2026-10-26); the database half open — read from the logs in F38-f WI-2.6’s window (spec Rev 4) |
NFR-2 response times, measured on dev — FR-14 GET /api/v1/pricing/warnings under 150 ms; the round summary with notInRound (GET /api/v1/auction-sales/{saleId}/round) under 150 ms; FR-5 GET …/{saleId}/not-in-round (one page of 25) under 300 ms, with and without q; FR-10 POST /api/v1/buying-standard-prices/{priceId}/correct and FR-12 PATCH /api/v1/auction-sales/{saleId} under 300 ms each — the 19th of 20 calls after a warm-up per request shape, at the NFR-1 volume |
The binding measurement is on the environment the application runs in: on a developer’s laptop the per-request cost dominates these reads (F26-a Rev 3, .forge/test-strategy.md § Performance Baselines, .forge/lessons.md L-084), so a local reading over a limit does not by itself fail NFR-2 |
spec NFR-2, FR-24, AC-28, AC-32; F26-g-WI-1.1-plan.md ## Notes (the commands, and the local readings the orchestrator records in its ## Progress) |
On dev, load src/test/resources/db/testfixture/pricing-volume.sql, pricing-round-volume.sql and pricing-carry-volume.sql into a throwaway copy of the database, never the shared dev data. As a user holding buying_standard.manage and price.publish, time the warnings, the summary and the not-in-round page of sale 2106 / 1 (default and q=VOLUME-0007), 25 corrections of open VOLUME-% prices and 25 date edits of a sale created for the purpose, as the plan’s ## Notes describe — 5 warm-up calls, then 20 measured calls per shape, from a host in the same region. Pass when every call answers its success status and the 19th value is under its limit. Record the figures in F26-g’s findings; a breach fails NFR-2 and is filed as a BUG against F26-g. Remove the fixtures afterwards |
F38 (the §C-7 retest) | open |
F33-a — File attachments on accounts, contacts and leads
Spec FR-H1 / AC-22 require the shipping PR to add these; F33-a-WI-1.3 adds them, and moves F33-a’s half of the F33 backlog row here. Everything else in F33-a runs and is proved on the local stack — LocalStack’s versioned S3 bucket and Postgres, with scanning off and with the stub scanner on (spec NFR-4), and the stub login: the contact owner type and its visibility, the upload policies of the three record owner types, the list with uploadedBy and its query count at 200 files, the 404 and 403 refusals, the delete with its delete marker, the download URL that saves the file under its own name (fetched from LocalStack), and the Attachments card on the account, contact and lead pages in the browser (F33-a-WI-1.1, WI-1.4 and WI-1.5). Scanning is off in every environment, so nothing about a real scan is added here; §B F20-h row © covers the scan-on path for the shared store. The task role’s S3 grants on et-dev-attachments/* are §B F20-h row (g)'s. The rows below are what dev still owes.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) Upload, list, download and delete on dev, on each record type — a PDF and an .xlsx on an account, a contact of it and a lead the test user reaches, through the Attachments card, and the 404 for a record the user does not reach |
Only a deployed backend writes to et-dev-attachments through the task role and is called cross-site from the dev CRM origin with the session cookie and X-XSRF-TOKEN (AD #33); locally LocalStack enforces no IAM and the frontend and API share an origin through the Vite proxy |
F33-a spec FR-H1 (a), AC-21, FR-O3; F33-a Wave 1 contract ### Endpoints; AD #33; §B F20-h rows (f) and (g) |
On https://dev.crm.empire.lk, signed in as CRM01 after checking (or granting, through Settings) that its role holds account.manage and lead.manage and neither account.all nor lead.all — CRM01 is the seeded IT Admin, whose keys are configuration (AD #34): on an account, a contact of that account and a lead CRM01 reaches, upload a PDF and an .xlsx through the card — each answers 201 with scanStatus NOT_SCANNED and is listed with CRM01’s name, today’s date and its size; Download saves each; Delete on one removes it, and it stays gone after a reload. Then, signed in as a test user whose reporting line does not reach that account, the list GET /api/v1/attachments/account/<that id> answers 404. Record the record ids, the users and the answers here |
F33-a / F38 | open |
| (b) The download is saved under the file’s own name, from the real presigned URL | Real S3 must apply the response-content-disposition and response-content-type overrides on a URL signed with the task role’s temporary credentials; AC-7 proves them only against LocalStack |
F33-a spec FR-H1 (b), FR-D2, AC-7, NFR-1 | On dev, upload a file named with a space and a non-ASCII character (for example Spec sheet – Ceylon.pdf) and press Download: the browser saves it under that name, not the opaque key. Read the url of the download answer from the browser’s network log and, within 5 minutes, curl -s -D - -o /dev/null "<url>": the response carries Content-Disposition: attachment; filename="…"; filename*=UTF-8''… and the sniffed Content-Type. Record the headers here, never the URL, which is a short-lived credential |
F33-a / F38 | open |
© A deleted file’s version is kept behind a delete marker in et-dev-attachments |
LocalStack’s bucket is versioned by the test base, which proves the code, not the dev bucket; decision 3 (a deleted file stays recoverable, proposal §A.5) rests on the real bucket’s versioning and on no lifecycle rule expiring the kept version | F33-a spec FR-H1 ©, decision 3, FR-X1, AC-8; §B F20-h row (a) (versioning Enabled, no lifecycle configuration) |
After (a)'s delete: aws s3api list-object-versions --bucket et-dev-attachments --prefix attachments/<ownerType>/<ownerId>/ --profile et-dev (a read) lists, for the deleted file’s key, the uploaded version with IsLatest: false and a delete marker with IsLatest: true; the record’s other files keep a current version. The key is attachments/<ownerType>/<ownerId>/<uuid> (AttachmentService), so the prefix finds it without a database read |
F33-a / F38 | open |
(d) A-1’s boot check with the contact owner type |
RoutingRecordVisibilityCheck routes contact to ContactRecordVisibility from F33-a on; offline, SeamGuardIT proves only the wiring a cloud task would boot with |
F33-a spec FR-H1 (d), FR-O1, FR-O2; A-1; §B F5-f | After the first dev deploy that carries F33-a, the backend starts with no SeamGuard failure in its log, and (a)'s contact upload answers 201 while an upload by (a)'s out-of-reach test user to that contact answers 404 |
F33-a / F38 | open |
| (e) The list answers in under 300 ms for a record with 200 files, on dev | NFR-3’s binding measurement is on dev; a laptop reading against LocalStack is evidence only (L-084), and AC-23 proves locally only that the query count does not grow with the number of files | F33-a spec FR-H1 (e), NFR-3, AC-23; .forge/test-strategy.md § Performance Baselines |
On dev, as CRM01 with the role (a) checks, upload 200 copies of the smallest valid PDF to one account through POST /api/v1/attachments/account/<id>. By § Performance Baselines’ method: warm every shape first; call an endpoint this feature does not touch (GET /api/v1/accounts/<id>) as the baseline, interleaved round-robin with GET /api/v1/attachments/account/<id> (first page, 25); 20 measured calls of each; compare the 19th readings — the list’s is under 300 ms. Then delete the 200 files through the route; their kept versions stay (no lifecycle rule) and cost a few hundred kilobytes. Record the readings here and a dated ## Log line |
F33-a / F38 | open |
F36-a — Capture and upload (card photo, badge QR)
Spec FR-CG1 / AC-37 require the shipping PR to add these; F36-a-WI-1.4 adds them, and replaces F36-a’s row in the backlog table below. Everything else in F36-a runs and is proved on the local stack: the stub sign-in (no local Microsoft round trip, AD #28), Postgres, LocalStack’s S3 for the card photos and the browser’s fake camera. Wave 1’s sign-in is proved by F36-a-WI-1.3 in the capture app’s seam Playwright project against the local backend, through the capture app’s Vite /api proxy, where the CSRF cookie is same-origin and no CORS list is read. The rows below are what dev, UAT and production still owe.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) The capture redirect address in each environment’s task definition — ET_SECURITY_CAPTURE_REDIRECT_URI = the environment’s capture origin, a bare origin with no path and no trailing slash: dev https://capture.dev.crm.empire.lk; UAT https://capture.uat.crm.empire.lk and production https://capture.crm.empire.lk once their capture hosts exist |
A task-definition value. Locally application-local.yml.example sets http://localhost:5173 and AC-2’s integration tests set their own values. Unset, a sign-in started from the capture app returns to the CRM (FR-A2), so a missing value fails quietly |
F36-a spec FR-CG1 (a), FR-A2, AC-2; F36-a Wave-1 contract § Configuration (backend); AD #40; §D ET_SECURITY_CAPTURE_REDIRECT_URI; environments-and-identity.md §1 |
Dev: forge-et-be .aws/td.dev.json carries the value as a plain environment entry, never a secrets reference (added in F36-a-WI-1.1’s PR). After the first dev deploy that carries it, aws ecs describe-task-definition for dev’s backend (a read) shows it, and the backend started, since a malformed origin stops startup naming the key. UAT and production: their task definitions carry the value once F38-h and F38-i build the capture hosts. Record each environment’s check here. 2026-11-19: UAT’s td.uat.json carries https://capture.uat.crm.empire.lk since forge-et-be#474 (F38-h FR-13); checked by F38-h-WI-6.1 at the first -rc deploy that carries it (F38-h AC-18) |
F36-a (dev) · F38-h (UAT) · F38-i (production) | open |
| (b) The sign-in round trip from the capture app, on dev — started on the capture host, through Microsoft, back to the capture host, with the cross-site cookies | Needs the Entra registration, the reply URL on api.dev, the capture host on CloudFront and the cross-site cookies of AD #33; locally only the stub sign-in runs, and the app and the API share an origin through the Vite proxy |
F36-a spec FR-CG1 (b), FR-A2, FR-A3, FR-A6, AC-2, AC-3, AC-6; F36-a Wave-1 contract E-1 to E-4; AD #28, AD #33; §B F5-a, §B F5-j | Once F36-a’s wave 1 and row (a) are on dev, with a test account whose role holds capture.use (granted through Settings › Roles): on https://capture.dev.crm.empire.lk, signed out, press Sign In With Microsoft. The browser goes to https://api.dev.crm.empire.lk/oauth2/authorization/microsoft?app=capture, through Microsoft, and lands on https://capture.dev.crm.empire.lk with the tabs; the SESSION cookie is host-only on api.dev.crm.empire.lk; the XSRF-TOKEN-DEV cookie, whose Domain is dev.crm.empire.lk, is readable from the capture host and echoed as X-XSRF-TOKEN on the sign-out, which answers 204; the network log holds no call to /api/v1/mailbox/connection. A tenant account with no CRM user lands on https://capture.dev.crm.empire.lk/sign-in?login=failed, and a sign-in started on the CRM still returns to https://dev.crm.empire.lk. Record the accounts and the results here |
F36-a / F38 | open |
© The capture origins in the UAT and production CORS lists — ET_SECURITY_CORS_ALLOWED_ORIGINS names https://capture.uat.crm.empire.lk beside UAT’s CRM origin, and https://capture.crm.empire.lk beside production’s |
The capture app calls api.<env> cross-site with credentials (AD #33); locally the Vite proxy makes it same-origin and the list is empty. Dev’s list already names https://capture.dev.crm.empire.lk (forge-et-be .aws/td.dev.json) |
F36-a spec FR-CG1 ©; AD #33; §D ET_SECURITY_CORS_ALLOWED_ORIGINS; F38-h, F38-i |
When UAT and production are built: each task definition’s list holds the environment’s CRM and capture origins, and from the capture host a signed-in GET /api/v1/me answers 200 with Access-Control-Allow-Origin equal to the capture origin and Access-Control-Allow-Credentials: true. Record here. 2026-11-19: UAT’s list names the capture origin since forge-et-be#474 (F38-h FR-13); the request is made by F38-h-WI-6.1 at the first -rc deploy that carries it (F38-h AC-18) |
F38-h (UAT) · F38-i (production) | open |
(d) The card photo’s put and read on dev’s S3 — a card capture’s photo written under capture/ in et-dev-attachments and read back only through GET /api/v1/capture/leads/{id}/photo |
Only a deployed backend writes to the attachments bucket through the task role; locally LocalStack enforces no IAM | F36-a spec FR-CG1 (d), FR-C7, AC-10, AC-43, NFR-5; §B FS-11, the task-role S3 permissions row; environments-and-identity.md §4 (the task role’s grants on et-dev-attachments); §B F33-a row (a) |
Once F36-a’s wave 2 is on dev: submit a card capture on https://capture.dev.crm.empire.lk as a user whose role holds capture.use. aws s3api list-objects-v2 --bucket et-dev-attachments --prefix capture/ --profile et-dev (a read) lists its key; the lead’s page shows the photo (once row (f) is done); an unsigned request for the object answers 403; the backend’s log holds no photo bytes. Record the key and the answers here, never a signed URL |
F36-a / F38 | open |
| (e) The camera and the badge reader on real phones — on a real iPhone (Safari) and a real Android phone (Chrome), over HTTPS: the live rear camera with the shutter and Retake, Choose a Photo when the camera is refused, and a vCard, a MECARD and a web-link badge QR read into their fields | A browser test drives the browser’s fake camera and cannot hold a phone to a card or a badge; a phone’s camera needs a secure context, which only a deployed host gives it | F36-a spec FR-CG1 (e), FR-C1 to FR-C3, NFR-1; .forge/test-strategy.md § System Verification step 6 |
In the regression round that step 6 names (Phase 2’s in its text today; .forge/owed.md, “Carry the capture app’s move to Phase 1 into every document”), on the round’s capture host (dev before then): take and submit a card and each badge format on each phone; each becomes a captured lead with its fields. Two cases only a real phone shows (F36-a-WI-2.2’s pre-PR review, N-4): lock the phone, or switch to another app, while Capture shows the live camera, then come back — the live picture resumes or the screen says the camera is off, and the shutter never takes a black or frozen frame (useCamera does not watch for the track ending after it went live); and on the iPhone with Low Power Mode on, the live picture starts (the video relies on autoPlay with no play() call). A failure is a defect for .forge/bugs.md. Record the phones, their OS and browser versions and the results here |
F36-a (Nanndish) | open |
(f) blob: images in the capture app’s Content-Security-Policy — img-src gains blob: in dev, UAT and production |
The app shows a taken photo, and reads a card capture’s photo from …/photo, through blob: URLs; dev’s policy is img-src 'self' (forge-et-iaac#14), so both are blank there. pnpm dev and pnpm preview send no policy |
F36-a spec FR-CG1 (f), FR-C1, FR-L6, AC-38; environments-and-identity.md §4 (forge-et-iaac#14, the capture app’s policy); .forge/owed.md, “Allow blob: images in the capture app’s CSP” |
A forge-et-iaac PR adds blob: to the capture distribution’s img-src, reviewed with /forge-review-pr forge-et-iaac#<N>. Then on dev, with the browser console open: take a card photo (its preview shows) and open a card capture’s lead page (its photo shows), with no CSP violation in the console; curl -sI https://capture.dev.crm.empire.lk/ shows the new img-src. UAT and production carry the same policy when built. Record here |
Nanndish, with the DevOps team · F38-h, F38-i | open |
| (g) The install prompt, standalone mode and safe-area insets on real phones — on a real iPhone (Safari) and a real Android phone (Chrome), over HTTPS: the install card offered and installing the app, Not Now hiding it, the iOS Share › Add to Home Screen text, no install card once the app runs installed (standalone), and the header and tab bar clear of the notch, the status bar and the home indicator on every edge | The browser tests fire a synthetic beforeinstallprompt, force standalone mode with a matchMedia override and prove the safe-area padding from the shell’s CSS source: no browser installs the app in a test and no desktop browser has a notch. A phone offers installation only over HTTPS on a deployed host. Raised by F36-a-WI-1.2’s plan (## Risks), appended at Stage 12 |
F36-a spec FR-CG1, FR-SC1, AC-7; F36-a-WI-1.2 plan Decision 13 and ## Risks; .forge/plans/F36-a/F36-a-decomposition-plan.md § Notes, “Stage 12 hand-over, wave 1”; .forge/test-strategy.md § System Verification step 6 |
With row (e), on the round’s capture host (dev before then): on each phone, signed in, open Settings — Android Chrome offers Install the App, Not Now hides it for that browser, and Install adds the app to the home screen; iPhone Safari shows the Share › Add to Home Screen text. Open the installed app: it runs without browser chrome and Settings shows no install card; on a phone with a notch or a home indicator, in portrait and landscape, the header, the tab bar and the content stay clear of them. Since the shell change of 2026-11-15 (design-system Rev 125: <main> is the only scroll region, forge-et-pwa#132), also check on each phone (forge-et-pwa#132’s delta review): the on-screen keyboard over the booth note near the foot of Capture scrolls the field into view, and the header is back at the top with no leftover offset once it closes; a drag inside <main> scrolls only <main>, and a drag on the header or tab bar does not bounce the page (iOS); whether Android Chrome’s pull-to-refresh from the header is wanted (it no longer starts inside <main>); iOS’s tap on the status bar no longer scrolls <main> to the top — accept or not; momentum scrolling in <main>, with the browser toolbars no longer collapsing on scroll in the browser (not installed); and <main> usable in landscape. Record the phones, their OS and browser versions and the results here |
F36-a (Nanndish) | open |
| (h) Choose a Photo with a real phone’s camera — on a real iPhone (Safari, which may hand over an HEIC photo) and a real Android phone (Chrome, through its camera app), over HTTPS, with the camera refused for the site: the photo decoded, upright as taken, and sent as a JPEG no longer than 2,048 px on its longer side | The browser tests give the file input a PNG made in the page; no test browser here decodes HEIC, and createImageBitmap’s orientation handling differs between engines and phones. Raised by F36-a-WI-2.2’s plan (## Risks, Photo decoding on real phones) |
F36-a spec FR-CG1, FR-C2, FR-C4, AC-11, AC-13; F36-a-WI-2.2 plan Decision 6, subtask 11 and ## Risks |
With row (e), on the round’s capture host (dev before then), with the site’s camera permission refused on each phone: on Capture, Choose a Photo, take a photo with the phone’s camera (and, on the iPhone, choose an HEIC photo from the library), check the preview is upright and readable, and Submit for Research. The captured lead’s photo (through E-2, once wave 4 shows it, or aws s3api head-object on its key, a read) is image/jpeg with its longer side at most 2,048 px. A photo the browser cannot decode shows That photo couldn’t be read. Choose another. and is not sent. Record the phones, their OS and browser versions and the results here |
F36-a (Nanndish) | open |
| (i) A capture kept on a real phone survives a restart and is sent on reconnect — on a real iPhone (Safari, and the app added to the home screen) and a real Android phone (Chrome), over HTTPS: a capture submitted with no connection is listed under Not Sent, is still listed after the browser or the installed app is closed and reopened a day later, and is sent on its own when the connection returns | The browser tests keep captures in the test browser’s IndexedDB and reload the page; no test closes a phone’s browser, leaves it a day, or exercises Safari’s clearing of a site’s storage after days without use, nor shows whether navigator.storage.persist() is granted on each phone. Raised by F36-a-WI-2.3’s plan (## Risks, Browser storage can be cleared; Decision 8) |
F36-a spec FR-CG1, FR-R1, FR-R2, AC-16; F36-a-WI-2.3 plan Decision 8, subtask 13 and ## Risks |
With row (e), on the round’s capture host (dev before then), signed in as a user whose role holds capture.use: turn on airplane mode, submit a card and a badge (both listed under Not Sent, Leads shows the count), close the browser (and, separately, the installed app), reopen it the next day still offline (both still listed), then turn airplane mode off with the app open: both are sent, oldest first, 2 captures sent. shows, and each becomes a captured lead. Record the phones, their OS and browser versions, whether the app was installed, and the results here; a capture lost is a defect for .forge/bugs.md |
F36-a (Nanndish) | open |
(j) Choosing an exhibitor list file through a real phone’s file picker — on a real iPhone (Safari) and a real Android phone (Chrome), over HTTPS: Upload Exhibitor List, in the Add Exhibitors sheet and on the nothing-exists panel, opens the phone’s own file picker, which offers a .csv and an .xlsx file (from the phone’s files, iCloud Drive or Google Drive, and a mail attachment saved to the phone), and the chosen file is read into Check the Exhibitor List |
The browser tests answer Playwright’s filechooser event, or set the hidden input’s files in the page; no test opens a phone’s own picker, and iOS Safari and Android open it only from a press, filtering by the input’s accept (extensions and media types). Raised by F36-a-WI-3.5’s plan (## Risks, The file picker on phones; Decision 14; subtask 10) |
F36-a spec FR-CG1, FR-P5, AC-29; F36-a-WI-3.5 plan Decision 14, subtask 10 and ## Risks |
With row (e), on the round’s capture host (dev before then), as a user whose role holds capture.use, on a campaign with no exhibitors and on one with some: press Upload Exhibitor List, check the picker opens and lets a .csv and an .xlsx be chosen (an old .xls greyed out or refused by the backend with its message), and that each opens Check the Exhibitor List with its rows. Record the phones, their OS and browser versions and the results here; a picker that does not open, or a file it will not offer, is a defect for .forge/bugs.md |
F36-a (Nanndish) | open |
| (k) The toasts under the deployed Content-Security-Policy, on dev — after row 11 of the capture app’s shadcn move (altrium-io/forge-et-pwa#213): a toast (for example Campaign archived. on Plan) appears above the tab bar, slides in and out, and its Dismiss works, with Sonner’s rules served from the app’s own bundled stylesheet | Sonner 2.0.8 inserts a <style> that the capture app’s style-src 'self' blocks, so #213 bundles sonner/dist/styles.css instead; no local run sends the policy, so only a deployed environment shows whether the bundled rules place the toast |
pwa-shadcn.md D-5, row 11; #213’s review finding 1 (Nanndish’s decision, 2026-11-22) | On capture.dev.crm.empire.lk after #213 deploys: raise a toast at phone and desktop widths, check its place, motion and Dismiss, and that the console shows only the expected blocked inline <style> from Sonner (and BUG-543’s scroll lock) and no other policy violation. A misplaced toast is a defect for .forge/bugs.md |
F36-a (Nanndish) | open |
F36-c — Capture research, scoring and in-app notification
Spec FR-CG1 / AC-44 require the shipping PR to add these; F36-c-WI-1.6 adds them in F36-c’s wave 1, and replaces F36-c’s row in the backlog table below. The captured-lead half shipped with F36-c’s wave 2 (forge-et-be#475, forge-et-pwa#171, merged 2026-11-18), so every row below has code on main to check; the rows’ Wave 2 marks were removed at F36-c’s Reflect (2026-11-20). Everything else in F36-c runs and is proved on the local stack: the stub search provider’s phrases and the stub’s lead-research and capture-ranking answers drive every research and ranking state (spec FR-TS1), LocalStack’s SQS carries both queues, and BusinessClock drives the retries and the sweep. Dev’s queues, schedule and settings are in place since 2026-11-20 (forge-et-iaac#25, forge-et-be#485; items (a)–©). The rows below are what dev, UAT and production still owe.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) The two research queues and their keys — the plan-research queue and its dead-letter queue in forge-et-iaac’s dev overlay (redrive maxReceiveCount 3), the backend task role’s send, receive, delete and get-attributes rights on them, and ET_MESSAGING_PLANRESEARCHQUEUE in forge-et-be .aws/td.dev.json as a plain value (AD #40); the same for the capture-research queue and ET_MESSAGING_CAPTURERESEARCHQUEUE; UAT and production the same. The *DLQ keys are not set in a task definition: EtProperties reads them only for the local QueueProvisioner, as for every other queue (corrected 2026-11-19) |
Locally QueueProvisioner creates both queues in LocalStack. A queue name with no queue fails the deploy, and without the key no subject on that environment is researched or ranked, so each reads researching |
F36-c spec FR-CG1 (a), FR-JB1, AC-41; §D ET_MESSAGING_PLANRESEARCHQUEUE, ET_MESSAGING_CAPTURERESEARCHQUEUE |
Dev: a forge-et-iaac PR to dev adds the queues (named as dev’s other queues, such as et-dev-lead-research), reviewed with /forge-review-pr forge-et-iaac#<N>; then aws sqs get-queue-attributes for each queue (a read) shows its redrive policy, and after the deploy that carries the keys, aws ecs describe-task-definition (a read) shows them. The plan queue is in place before wave 1 is checked on dev, the capture queue before wave 2. UAT and production: their overlays and task definitions carry the same through F38-h and F38-i. Record the queue names and the PRs here. 2026-11-19: forge-et-iaac#25 (open, awaiting Shyamin Ayesh’s approval) adds et-dev-plan-research (visibility 300 s) and et-dev-capture-research (330 s), each with a -dlq, and the task role’s rights in overlays/dev and overlays/uat; forge-et-be#485 (draft, merges after #25’s apply) sets the two queue keys in td.dev.json. Production’s overlay (forge-et-iaac#17) is asked on that PR to list both queues. UAT’s td.uat.json keys come with F38-h. Update 2026-11-20 (Dinil, with Claude; read-only AWS calls): forge-et-iaac#25 merged into dev on 2026-11-20 (6fca402, approved on GitHub by Tharinda) and applied by dev.yaml run 44673993352 (“Apply complete! Resources: 5 added, 3 changed, 0 destroyed”); et-dev-plan-research, et-dev-capture-research and their -dlq queues exist on dev. forge-et-be#485 merged on 2026-11-20 (d240a830) and its dev deploy (build run 44719709516) rolled out backend:172, whose environment describe-task-definition shows with ET_MESSAGING_PLANRESEARCHQUEUE=et-dev-plan-research and ET_MESSAGING_CAPTURERESEARCHQUEUE=et-dev-capture-research; the service is steady on it. Dev’s half is in place; UAT and production open |
F36-c (Dinil), with the DevOps team (Shyamin Ayesh) · F38-h (UAT) · F38-i (production) | dev done 2026-11-20; UAT and production open |
(b) The EventBridge schedule capture-research-sweep at rate(1 hour) targeting the scheduler queue, payload {"jobKind": "capture-research-sweep", "idempotencyKey": "capture-research-sweep#<aws.scheduler.scheduled-time>"} |
The sweep is a scheduler jobKind (AD #12), never @Scheduled; without the schedule a subject whose research failed, was stopped by a limit or lost its message is never sent again. It sweeps exhibitors from wave 1 and captured leads from wave 2 |
F36-c spec FR-CG1 (b), FR-JB6; forge-et-be docs/messaging/scheduler-cloud-contract.md |
In the same forge-et-iaac PR as (a): aws scheduler get-schedule --group-name et-dev --name capture-research-sweep (a read) shows it ENABLED with the expression, payload and target; within the next hour the backend’s log shows the sweep’s run line. UAT and production through F38-h and F38-i. Record here. 2026-11-19: forge-et-iaac#25 adds the job to base/scheduler’s shared list, so dev gets it on the merge’s apply and UAT and production on their next apply; a UAT backend without F36-c dead-letters each firing until its release carries F36-c. Update 2026-11-20 (Dinil, with Claude; read-only AWS calls): dev’s apply of forge-et-iaac#25 (run 44673993352) created et-dev/capture-research-sweep, and get-schedule reads it ENABLED. Still to check on dev: the sweep’s run line in the backend’s log after a firing with the queue keys set (item (a)). UAT and production open |
F36-c (Dinil), with the DevOps team · F38-h · F38-i | dev: schedule in place 2026-11-20 (the sweep’s run line not yet checked); UAT and production open |
© The research models’ overrides in every environment — ET_AI_FEATURES_PLANRESEARCH_MODEL = Haiku 4.5’s us. profile (us.anthropic.claude-haiku-4-5-20251001-v1:0) in dev’s, UAT’s and production’s task definitions; ET_AI_FEATURES_CAPTURERESEARCH_MODEL = Haiku 4.5’s Global profile (global.anthropic.claude-haiku-4-5-20251001-v1:0) in all three, and ET_AI_FEATURES_CAPTURERANKING_MODEL = the Global profile — dev’s in forge-et-be .aws/td.dev.json, UAT’s and production’s through F38-h and F38-i. plan-ranking stays on its us. default everywhere |
Task-definition values; locally the stub answers whatever model is named. Without a research override the feature calls its design model, Sonnet 5.5, which the account refuses: the research ends MODEL_UNAVAILABLE and the subject is ranked with basis no_research |
F36-c spec FR-CG1 ©, FR-RS2, AC-18; §D rows; ai-provider-strategy.md §4d |
After each deploy that carries them, aws ecs describe-task-definition (a read) shows each value; an exhibitor added on dev gets a plan-research ai_invocation row with the Haiku 4.5 id and outcome SUCCESS (and a captured lead’s capture-research and capture-ranking rows the same). An account with no Global Haiku 4.5 quota keeps the capture features on the us. profile (F43 FR-R3). Each override is removed once item (g)'s evaluation passes for that feature and the lead moves it (ai-provider-strategy.md §4d). Record here. 2026-11-19: forge-et-be#485 (draft, merges after forge-et-iaac#25’s apply) sets dev’s three overrides as listed. Update 2026-11-20 (Dinil, with Claude; read-only AWS calls): forge-et-be#485 merged (d240a830); dev’s backend:172 carries ET_AI_FEATURES_PLANRESEARCH_MODEL = the us. profile and ET_AI_FEATURES_CAPTURERESEARCH_MODEL and ET_AI_FEATURES_CAPTURERANKING_MODEL = the Global profile. Still to check on dev: the ai_invocation rows of a research run. UAT and production open |
F36-c (Dinil) · F38-h · F38-i | dev: values set 2026-11-20 (ai_invocation rows not yet checked); UAT and production open |
(d) The startup visibility check in the cloud shape for PLAN_RESEARCH and PLAN_RANKING, and for CAPTURE_RESEARCH and CAPTURE_RANKING |
AiPropertiesCloudShapeIT boots the cloud shape locally; only a deployed task shows each feature’s visibility registered with the environment’s real queue keys |
F36-c spec FR-CG1 (d), AC-10; F43 FR-D3; §B F43 | After the first dev deploy that carries each wave: the task starts (a background feature with no registered visibility stops startup, naming it), aws ecs describe-services (a read) shows the service steady, and the log holds no visibility refusal. Record the task-definition revision here |
F36-c (Dinil) | open |
(e) A live capture run on dev, approved by the lead: a badge capture for a real, well-known public company reaches ready with working source links; one for an invented company is ranked with the basis nothing_found; the time from the upload’s 201 to ready is recorded against NFR-3 (2 minutes at the 95th percentile) |
Exa, Bedrock, the queues and the pacing exist only on dev; locally the stubs answer within 5 s | F36-c spec FR-CG1 (e), NFR-3, FR-RK2 | Once wave 2 and items (a) to (d) are on dev, with the lead’s approval of the calls (each search and model call costs): submit the two badges on https://capture.dev.crm.empire.lk as a user whose role holds capture.use; open each lead’s page; follow each source link; read the two ai_invocation rows of each run. Record the public company’s name (never a person’s details), the invented name, the times and the outcomes here |
F36-c (Dinil); the lead approves the calls | open |
(f) A 100-exhibitor list imported on dev (about USD 1.50), approved by the lead: the time to the last ready, NFR-3’s 500-exhibitor figure (about 2 hours, about 25 minutes per 100) checked against it, and the run’s cost |
The pacing on the us. profile’s background share and Bedrock’s latency are only measurable on dev |
F36-c spec FR-CG1 (f), NFR-3, NFR-4, FR-RS3 | Once wave 1 and items (a) to (d) are on dev, with the lead’s approval: import a list of 100 public companies (no person’s data) into a test campaign; record the time from the import to the last exhibitor at ready; read the run’s cost from ai_invocation (feature_key plan-research and plan-ranking) and its search count from search_call for the run’s window. Record the times, the counts and the cost here |
F36-c (Dinil); the lead approves the calls | open |
(g) The live evaluation runs — FR-RK6’s capture-ranking set on dev on Haiku 4.5, approved by the lead, and again on Haiku 5.5 and Sonnet 5.5 once the account serves them; with F11-c FR-A8’s set for the research features on each model they may move to |
The stub run proves the runner and scorer only; the set decides which model capture-ranking and plan-ranking may use, and with F11-c FR-A8’s set, whether a research feature moves (FR-RS2) |
F36-c spec FR-CG1 (g), FR-RK6, FR-RS2; §B F11-c item 7; ai-provider-strategy.md §4d |
AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=plan-ranking -PconfirmBedrockCalls=true runs the capture-ranking set through plan-ranking, and AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=plan-research -PconfirmBedrockCalls=true runs F11-c FR-A8’s set through plan-research (F11-g’s runner; add -Pmodel=<profile id> for each model; each run costs money — ask the lead first). The same with -Pfeature=capture-ranking and -Pfeature=capture-research. Record per model: cases matching the expected priority, cases two levels off, scores inside their band before correction, forbidden words, and pass or fail against FR-RK6’s bar; for the research runs, F11-c FR-A8’s figures and its bar, as §B F11-c item 5 records them. (Rev 6) From F36-c’s wave 3 on dev, the plan-ranking and capture-ranking commands run the capture-ranking-v2 set, so its Haiku 4.5 run is recorded once, under item (k), and that record covers this row’s Haiku 4.5 run; this row keeps the Haiku 5.5 and Sonnet 5.5 runs and the research runs. Before wave 3 is on dev, this row stands as written |
F36-c (Dinil); the lead approves each run | open |
| (h) Notifications on two real phones over HTTPS: rep A’s capture brings rep A’s Ready to review notice and rep B’s bell item; rep B’s conversion brings rep A’s bell item | A browser test runs two pages on one machine against the stubs; two phones polling over a real network reach only a deployed host | F36-c spec FR-CG1 (h), FR-NT1–FR-NT8, NFR-3 (shown within 20 s) | On dev, with items (a) to (e) done: two test accounts whose role holds capture.use, each on a real phone, both capturing into the same campaign; rep A captures, rep B converts. Record the phones, their OS and browser versions, and the seconds from each change to its notice or bell item here |
F36-c (Dinil) | open |
(i) A conversion’s copy on dev — converting a researched capture gives the CRM lead the capture’s research as its lead_research row, and writes no search_call row and no lead-research ai_invocation row for the new lead; (Rev 6) and the new lead’s ranking is the carried one, with no lead-ranking ai_invocation row for it |
Only the real chain on dev shows that F11-c’s job finds the copy and does nothing, and (Rev 6) that F11-g’s job finds the carried ranking and does nothing; locally the integration tests prove it on the stubs | F36-c spec FR-CG1 (i), FR-CV1, FR-CV2; (Rev 6) F11-g FR-S4, FR-J9; AC-54 | After item (e): convert the researched capture; read the new lead’s lead_research row (its ai_invocation_id is the capture’s research call) and count the search_call rows with lead_id = the new lead (0) and the lead-research ai_invocation rows for it (0). (Rev 6, from F36-c’s wave 3 on dev) read the new lead’s lead_ranking row — origin capture, its ai_invocation_id the capture’s ranking call, and focus and focus_campaign_name set when the capture’s ranking read a focus — and count the lead-ranking ai_invocation rows for the new lead (0); before wave 3 is on dev, the lead’s F11-g ranking row appears instead, as Rev 5 has it. Record the ids and the counts here |
F36-c (Dinil) | open |
(j) (Rev 6) A focus edit on dev, approved by the lead, on a campaign of about 20 exhibitors and a few captured leads to review: each is ranked again with one capture-ranking or plan-ranking call and no search_call row, converted leads are untouched, and the time to the last new ranking and its cost are read from ai_invocation |
The pacing on both Haiku 4.5 profiles, Bedrock’s latency and the real cost are only measurable on dev; locally the stub ranks again within seconds (AC-51, AC-52 on the stubs) | F36-c spec FR-CG1 (j), FR-TR7, FR-FC5, FR-FC8, NFR-3; AC-51 | Once wave 3 and items (a) to (d) are on dev, with the lead’s approval of the calls (each ranking call costs): on https://capture.dev.crm.empire.lk, as a user whose role holds capture.use, take a test campaign that is neither archived nor ended (no days, or its last day today or later), with about 20 exhibitors (public companies, no person’s data), a few captured leads at To Review and one converted lead, all researched and ranked; set or change its Focus and save (the form shows the being ranked again line); wait until no subject reads Being ranked again for the campaign’s new focus. For the window from the save, read the ai_invocation rows with feature_key plan-ranking (one for each exhibitor not archived) and capture-ranking (one for each lead at To Review that is not a card at reading and not ranked not_ranked), and confirm no plan-research or capture-research row and no search_call row; confirm the converted lead’s CRM lead_ranking row is unchanged. Record the campaign’s size, the counts, the time from the save to the last new ranking (against FR-FC8’s figures, about 8 exhibitor rankings a minute on the us. profile) and the run’s cost from cost_usd (against FR-FC8’s about USD 0.0025 a subject) here. 2026-11-20, partly done (Dinil watching, Claude in the browser pane; the lead’s approval as Dinil passed it on in the session: “lead approves”): campaign Live check F36-c 2026-11-20 (no dates, running) with 5 exhibitors added by hand — Twinings, Tata Consumer Products, Starbucks, Tesco, Nestlé; all five researched with working source links and ranked in about 2 minutes. Before a focus: Tesco warm 52, Tata 18, Nestlé 15, Twinings 15, Starbucks 8 (all cold but Tesco). The focus Supermarket chains that buy private-label tea for their own brands saved with the being ranked again line; within about 30 s every exhibitor was ranked again: Tesco hot 78 with the line Ranked for the focus of … and a new reason and next action, Tata 22, Nestlé 15, Twinings 15, Starbucks 5. Not yet done: about 20 exhibitors, captured leads at To Review and a converted lead in the campaign, and the ai_invocation / search_call counts and cost (no database read was made) |
F36-c (Dinil); the lead approves the calls | open |
(k) (Rev 6) The v2 evaluation sets on dev, a precondition of the first UAT or production release that carries F36-c’s wave 3 — FR-RK6’s capture-ranking-v2.json set and F11-g FR-A8’s lead-ranking-v2.json set, each run on Haiku 4.5 on dev, approved by the lead, meeting its v2 bar |
No setting chooses a prompt version (AD #19’s per-feature configuration is {model, enabled}), so wave 3’s code calls capture-ranking/v2 and lead-ranking/v2 in every environment it is deployed to; the stub run proves the set loads and runs; the scorer test proves the bar’s arithmetic (AC-59; FR-RK6 Rev 7). Dev may call v2 before this run; UAT and production may not receive wave 3 until it passes |
F36-c spec FR-CG1 (k), FR-RK6, FR-FC3, D-31; F11-g spec FR-A2, FR-A8, FR-H4; AC-59; §B F11-g item 5; ai-provider-strategy.md §4d |
Once wave 3 is on dev, with the lead’s approval of each run (each costs money): AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=plan-ranking -PconfirmBedrockCalls=true runs the capture-ranking-v2 set through plan-ranking (one run covers the set: capture-ranking calls the same prompt on the same model), and AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=lead-ranking -PconfirmBedrockCalls=true runs the lead-ranking-v2 set. Record per run: the model; on the cases with no focus, v1’s bar (cases matching the expected priority, cases two levels off, the research pairs, for the capture set the scores inside their band before correction, forbidden words); on the focus cases, the v2 bar (pairs moved opposite to the stated direction, pairs matching both expected priorities, the unrelated focus moving no priority, the instruction case keeping its priority with no forbidden word); and pass or fail. This row must read validated, with a pass for both sets, before the first UAT or production release that carries F36-c’s wave 3 is tagged; a fail holds that release, and the lead decides what follows. 2026-11-20 — failed, both sets (run by Claude for Dinil on forge-et-be main b33e11b8, model us.anthropic.claude-haiku-4-5-20251001-v1:0; the lead’s approval as Dinil passed it on in the session: “lead approves”). plan-ranking (capture-ranking-v2), US$0.154, 43 requests, p95 8.2 s: no-focus cases 13 of 20 as expected (needs 75%), 0 two levels off, 0 pairs lower without research, fit score in band 18 of 18, 0 forbidden words, 2 failed calls (INVALID_OUTPUT, NO_TOOL_USE — BUG-518); focus pairs 0 moved against the direction, 4 of 8 matching both (needs 75%), 1 unrelated focus moved a priority (focus-unrelated-vague P2 → P3, must be 0), instruction case held, 1 failed call (TIMEOUT). lead-ranking (lead-ranking-v2), US$0.135, 39 requests, p95 8.2 s: 22 of 23 as expected, 0 two levels off, 1 ranked lower without research (group3: unavailable P3 below done P2, must be 0), 0 forbidden words; focus pairs 0 moved against the direction, 2 of 7 matching both (needs 75%), 0 unrelated moved, instruction case held. In both sets the model ranked many without focus cases one level lower than expected (P3 for P2), so the focus pairs miss their expected without priority. Reports: build/reports/ai-evaluation/live/plan-ranking-20261120-101437.md and lead-ranking-20261120-102050.md (local). Wave 3 may not go to UAT or production until a run passes; the lead’s decision is an owed.md row |
F36-c (Dinil); the lead approves each run | open — failed 2026-11-20 |
F36-b — AI card reading
Spec FR-CG1 / AC-29 require the shipping PR to add these; F36-b-WI-1.4 adds them, and replaces F36-b’s row in the backlog table below. Everything else in F36-b runs and is proved on the local stack: the AIProvider stub’s card answers and fixture photos (each failure keyed by the image, FR-H2), the Bedrock request built with a fake client and no network (AC-10), LocalStack’s SQS for the card-read queue, Postgres and the stub sign-in. Until the queue and its key exist on an environment, the backend starts there with no card-read listener: a card capture is not queued (one WARN line) and its lead’s page stays at Reading the card… (from F36-b’s wave 3, after 10 minutes, This is taking longer than usual. with Read Again, which also queues nothing).
Items (b) to (e) can run once F36-b’s wave 1 (the image operation, the photo preparation and answer cleaning, the evaluation set and its runner) is on main: (b) to (d) are one live run of the evaluation set, which the lead approves first because it calls Bedrock (.claude/CLAUDE.md § Boundaries › Ask first), recorded before F36-b-WI-2.1 is dispatched (.forge/owed.md, “Run F36-b’s early checks on dev”; WI-2.1’s plan was approved before the run and stops its implementer while (d) is open); (e) needs only reads. Items (a), (g) and (h) need F36-b’s wave 2, the card-read job, on dev; (f) needs its wave 3, the capture-app screens, as well; and (b) closes only when (f)'s phone photograph of a printed fictional card has gone through the same call.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) The card-read queue and its dead-letter queue, and ET_MESSAGING_CARDREADQUEUE — in forge-et-iaac: the queue and its DLQ (redrive maxReceiveCount 3, the et.messaging.* default; the queue’s own visibility timeout 150 s, equal to et.card-read.visibility-timeout, which the listener asks for on every receive), the backend task role’s consume and send rights on the queue and none on the DLQ, as on lead-ranking, and ET_MESSAGING_CARDREADQUEUE on the task definition as a plain value (AD #40); dev first, then UAT and production |
The listener exists only with the key set and the enqueuer sends nothing without it, so a task definition without it starts healthy and reads no card; the queue pair is QueueProvisioner’s against LocalStack only. A dead-letter key, if F36-b-WI-2.1 adds one, is read only by the local provisioner and stays unset in cloud, where the DLQ is Terraform’s |
F36-b spec FR-CG1 (a), FR-R1, FR-R3, FR-R6, NFR-4, AC-28; F36-b Decomposition Plan § Notes, “The dev queue is a DevOps request”; §D ET_MESSAGING_CARDREADQUEUE; §B F11-g item 1 (the same shape) |
Dev, by the F36-b team’s own PRs when wave 2 is about to reach dev (Dinil’s decision, 2026-11-13): a forge-et-iaac PR to dev adds the queue, the DLQ and the task role’s rights, approved by Shyamin Ayesh; then a forge-et-be PR sets the key in .aws/td.dev.json, only once the queue exists (a queue name with no queue fails the deploy, §D ET_MESSAGING_LEADRESEARCHQUEUE); after the deploy, aws ecs describe-task-definition for dev’s backend (a read) shows the key, and a card capture on dev is queued and read (the job’s log line, not its WARN). UAT and production: the same lines in overlays/uat (F38-h) and overlays/prod (F38-i), and F38-i’s oldest-message alarm list names the card-read queue (F38-i decision 26; as §B F11-g row 7). Record each environment here Update 2026-11-16 (Dinil; no AWS call): dev’s forge-et-iaac PR is open, raised while wave 2 is built so dev can read cards once it merges — forge-et-iaac#22: base/sqs/card_read.tf (queue et-dev-card-read, DLQ et-dev-card-read-dlq, redrive 3, visibility 150 s), and in overlays/dev and overlays/uat the task role’s consume and send rights on the queue and the card_read_queue_name output (UAT added at Dinil’s request after review, 94ecf78, because base/sqs creates the queue pair in every overlay). /forge-review-pr posted approve on 94ecf78; waiting on Shyamin Ayesh’s approval, and the merge’s dev.yaml apply creates the queue on dev. Then the forge-et-be PR sets ET_MESSAGING_CARDREADQUEUE=et-dev-card-read in .aws/td.dev.json. UAT gets et-uat-card-read on its next hand-run uat.yaml apply, after which .aws/td.uat.json can take the key. Production’s first apply, once forge-et-iaac#17 merges, creates the queue pair with no task-role rights until overlays/prod lists it; asked for on forge-et-iaac#17. Update 2026-11-17 (Dinil): dev done. forge-et-iaac#22 merged on Senuthmi Thimansa’s approval (Shyamin Ayesh unavailable; Dinil accepted it), and its dev.yaml run 44021940709 applied it: et-dev-card-read and et-dev-card-read-dlq created and the task role’s two SQS policies changed (2 added, 2 changed, 0 destroyed). forge-et-be#458 set ET_MESSAGING_CARDREADQUEUE=et-dev-card-read in .aws/td.dev.json; after its deploy (build run 44025082231, success), aws ecs describe-task-definition for dev’s backend (backend:150, a read) shows the key. Still to check on dev: a card capture queued and read (the job’s log line). UAT and production stay open. Update 2026-11-17 (Tharinda, with Claude; F38-h WI-5.2 record R-2): UAT’s task definition had no key. forge-et-be#464 sets ET_MESSAGING_CARDREADQUEUE=et-uat-card-read in .aws/td.uat.json, the name forge-et-iaac base/sqs/card_read.tf gives with environment = uat. The UAT queue pair and the task role’s rights are on iaac dev (#22) but exist in AWS only after the next uat.yaml apply, which runs before the next -rc tag’s deploy (.forge/owed.md, “Bring UAT up to forge-et-iaac dev with the next -rc tag”). UAT stays open until that tag’s backend shows the key (describe-task-definition, a read) and reads a card. |
Dinil, with Shyamin Ayesh (dev; .forge/owed.md) · F38-h (UAT) · F38-i (production) |
dev done 2026-11-17 (the read itself not yet seen on dev); UAT and production open |
(b) A live Haiku 4.5 vision call on dev through the us. profile — Bedrock accepts the Converse image block, the model answers through the card-read tool, and the call’s cost is at or under 0.5 US cents |
The stub records zero tokens and cost, and BedrockAiProviderSpec asserts the built request with a fake client and no network (AC-10); only Bedrock shows that it accepts the JPEG image block and answers through the tool, and what a card costs |
F36-b spec FR-CG1 (b), FR-A1, FR-A7, NFR-4, AC-10, decision D-17; .forge/client/sow-cost-annexe.md §6; proposal §A.3.2 |
Runnable once wave 1 is on main, after the lead approves it: item ©'s live run is these calls. Its cases are the set’s fictional cards (FR-H1), drawn with Java2D — the rotated card and the one drawn at an angle among them — not photographs; no real person’s card is sent to dev. From the run’s printout and report: the calls end SUCCESS (a refused image block fails every case), and the cost it prints, divided by its calls, is at or under 0.5 US cents (NFR-4 estimates about 0.42). Record the calls, their outcomes, the tokens and the cost a call here, against the annexe. The “real card photo” part: when (f) runs, the phone’s photograph of a printed fictional card goes through the same call; record that call’s AI Call Completed line for card-read from /ecs/dev/backend (a read) here — outcome SUCCESS, cost at or under 0.5 US cents. (b) closes only then Update 2026-11-16 (Dinil, the lead’s approval of the same day; the run of item ©): 20 calls, 20 SUCCESS, none refused — Bedrock accepts the JPEG image block and the model answers through the card-read tool; US$0.089047 for the 20 (58,980 input and 4,395 output tokens), 0.45 US cents a call, under 0.5. The drawn-card half is met; (b) stays open for (f)'s phone photograph. |
Dinil (the lead approves the run) | open — drawn cards met; real photo with (f) |
| © The evaluation set’s run on dev, meeting FR-H1’s bar: no field expected null comes back filled in any case, and at least 90% of the expected non-null values match after trimming and case-folding | The stub’s fixed card proves the runner and the scorer only (F36-b-WI-1.3’s stub run) | F36-b spec FR-CG1 ©, FR-H1, AC-25, decision D-15; forge-et-be docs/ai/README.md § Evaluation (“The live run”) |
Runnable once wave 1 is on main: AWS_PROFILE=et-dev ./gradlew liveEvaluation -Pfeature=card-read -PconfirmBedrockCalls=true from a laptop (it calls Bedrock and costs money, so the lead approves it first; the command F36-b-WI-1.3’s plan locks). The task fails when the run misses the bar. Record the report’s invented-value count and match rate here and in F36-b-WI-1.3’s plan ## Progress; a miss is raised with the lead before F36-b-WI-2.1 is dispatched *Update 2026-11-16 (Dinil; run from the wave-1 branch at 5c5fd48f, because main then held two V72 migrations; report card-read-20261116-062706.md on us.anthropic.claude-haiku-4-5-20251001-v1:0): 20 cases, 0 failed calls; 122 of 125 expected values matched (97.6%), over the 90% bar; but 2 invented values, both country: plain-no-address came back US and plain-name-and-mobile GB, cards that print no address, most likely read from the phone’s country code. So the run misses FR-H1’s bar (no invented value). Also: city-only-ambiguous’s phone came back with its label (Tel 555 0119), and rotated-quarter-turn lost its email and misread one phone digit (confidence 65). Raised with the lead before WI-2.1 is dispatched (.forge/owed.md, “Decide on card reading’s two invented countries”). Update 2026-11-16, second run (Dinil; the lead approved it; prompt card-read/v1 tightened before release in forge-et-be b68dd452: the country only from a printed address, null without one, never from a phone code, a domain, the language or the company name; the phone without its label): 123 of 125 matched (98.4%), the phone label fixed, but 3 invented countries: the two no-address cards still US and GB, and city-only-ambiguous GB. Haiku 4.5 infers the country whatever the wording, so the bar is still missed; US$0.090647 for 20 calls, p95 3,231 ms. Report card-read-20261116-064643.md. Update 2026-11-16, third run (Dinil; the lead approved it; forge-et-be baeb9777 adds six worked examples of the country rule to the prompt, none taken from the set): 123 of 125 matched (98.4%); 2 invented countries: plain-no-address US (no address, a +1 phone) and city-only-ambiguous GB (“200 Mill Street, Richmond”); the +44-mobile card now null. US$0.096108, p95 3,260 ms. Report card-read-20261116-071507.md. Still missed. Update 2026-11-16, decision (Dinil, F36-b’s owner): spec Rev 4 counts a country filled where the card states none as a suggested country the rep checks, not an invented value; the other six fields keep no invented value. forge-et-be a03cb29c changes the scorer to match. Under Rev 4 the third run has 0 invented values and 2 suggested countries, with 98.4% matched, so it meets the bar; no new run was needed. |
Dinil (the lead approves the run) | met under spec Rev 4 on the third run (0 invented, 2 suggested countries, 98.4% matched) |
(d) The call’s time on dev — the 95th percentile of ©'s run inside the 9-second call deadline (et.ai.call.deadline) |
Locally the stub answers at once; a vision call’s time is measurable only on Bedrock, and FR-R3’s visibility sum (10 s read + 5 s preparation + 60 s wait + 9 s deadline + 60 s = 144 s ≤ 150 s) rests on the 9-second deadline | F36-b spec FR-CG1 (d), FR-R3, NFR-3, W-7; F36-b Decomposition Plan § Notes, “The early checks on dev” | The p95 latency ©'s live run prints. Record it here; F36-b-WI-2.1’s plan records it too, or states that (d) has not run and which visibility it assumes. If it is near or over 9 seconds, ask the lead for a deadline of card reading’s own (spec W-7); the spec is revised and FR-R3’s sum recomputed before F36-b-WI-2.1 is dispatched Update 2026-11-16 (Dinil; the run of item ©): p95 3,097 ms for the 20 successful calls, well inside the 9-second deadline and under WI-2.1’s 7-second stop line; no deadline of card reading’s own is needed (spec W-7), and FR-R3’s 150 s visibility stands. | Dinil (the lead decides a deadline of the feature’s own) | done — p95 3.1 s |
(e) The startup check of CARD_READ’s visibility in the cloud shape — dev’s backend starts with CardReadReceiveVisibility registered and AiPacingCloudConfig accepting it |
F43’s check runs only in a cloud shape; CARD_READ is a background feature, so without a registered visibility, or with et.card-read.visibility-timeout below the wait plus the deadline plus 60 s, a cloud shape refuses to start, naming the feature or the keys (fail closed). Locally only F36-b-WI-1.1’s cloud-shape test context proves it |
F36-b spec FR-CG1 (e), FR-R3, AC-8; F43 spec FR-D3, FR-D5; F36-b-WI-1.1 plan | On the dev deploy of wave 1’s merge, with reads only: aws ecs describe-services --cluster et-dev shows the new task definition revision running and the service steady, and aws logs filter-log-events on /ecs/dev/backend around the deploy finds no startup refusal naming card-read or et.card-read.visibility-timeout. Record the revision and the time here |
Dinil | open |
(f) The end-to-end case of AC-23 on dev from a real phone — a card photographed on a phone on https://capture.dev.crm.empire.lk, read, its details on the lead’s page, converted, and the photo attached to the CRM lead |
Locally the capture app’s browser tests submit a fixture photo through Choose a Photo to the stub; only a deployed host gives a phone’s camera a secure context, and only Bedrock reads a photograph | F36-b spec FR-CG1 (f), AC-23; F36-a spec FR-CG1 (e) (the camera on real phones); .forge/owed.md, “Prove a card capture’s convert end to end once F36-b reads cards” |
Once F36-b’s wave 3 and item (a) are on dev, and F36-a’s register items for the sign-in, the photo on dev’s S3 and blob: images are met: on a real iPhone (Safari) and a real Android phone (Chrome), as a user whose role holds capture.use, photograph a printed card from FR-H1’s fictional set (never a real person’s card); the lead’s page shows Reading the card…, then the read details; Convert creates the CRM lead with the photo attached. The call’s AI Call Completed line for card-read in /ecs/dev/backend (a read) gives its cost. Record the phones, their OS and browser versions, the times and the results here. Update 2026-11-18 (F36-b Reflect): F36-b’s wave 3 is on main (forge-et-be#468, forge-et-pwa#159), so dev has it after each repo’s next dev deploy; the local half of this case — a card capture’s convert end to end against the real backend — passed in F36-b-WI-3.5’s seam specs, and that owed.md row is closed. What remains is item (a) on dev and the real phones. |
Dinil | open |
(g) The time from upload to read on dev — within 30 seconds of the upload’s 201 at the 95th percentile, with free pacing slots; and the photo preparation within FR-R3’s 5-second term |
Locally the stub reads within 3 seconds (NFR-3); the queue, the pacing and Bedrock’s latency are measurable only on dev. The dev task’s 0.5 vCPU and the backend’s container image are the first place the photo preparation runs as deployed (F36-b-WI-1.2 Risks), and FR-R3’s visibility sum (144 s ≤ 150 s) rests on its 5-second term | F36-b spec FR-CG1 (g), FR-R3, NFR-3; F36-b-WI-1.2 plan Decision 12 | After (a) and (f): upload ten card captures on dev, one at a time and at least a minute apart, and for each read the requested and read times in its cardRead payload (GET /api/v1/capture/leads/{id}). Record the ten gaps here; the largest of ten is their 95th percentile (nearest rank). For the same ten reads, read each preparation’s elapsed milliseconds from the card-read handler’s log line in /ecs/dev/backend (a read; F36-b-WI-2.1’s plan names the line — .forge/owed.md, “Settle the card-photo preparation’s memory and time on dev in F36-b WI-2.1’s plan”), record the largest against the 5-second term, and raise one near 5 s with the lead (FR-R3: bound the preparation or raise the visibility timeout) |
Dinil | open |
(h) A burst on dev — 20 card captures uploaded together; the time from the first 201 to the last read, against NFR-3’s 3-minute figure, and the profile the background features named in AI Call Completed lines during the run (Global or us.) |
The burst’s time depends on Haiku 4.5’s request quota, the background share of the us. profile (8 of its 10 requests a minute on one task, F42) and whether triage, research and ranking share that profile; none of these exists locally |
F36-b spec FR-CG1 (h), NFR-3, decision D-3; §B F43 item 1; §D, the background route | After (a): upload 20 card captures within a minute, as a user whose role holds capture.use (about US$0.10 of Bedrock calls at NFR-4’s figure). Record the first 201, the last read and the gap, and, from /ecs/dev/backend (a read), the model of every AI Call Completed line for lead-triage, lead-research and lead-ranking in those minutes. Where they named the Global profile, the gap is held to about 3 minutes; where they shared us., NFR-3’s longer figures apply. Record here |
Dinil | open |
(i) The backend container’s memory on dev while card reads run — no OOM kill and no ExitOnOutOfMemoryError restart while two large card photos are prepared at once, and the peak of the task’s memory against its 1,024 MB limit |
Two progressive JPEGs near 40 megapixels decoded at once held the local container at its limit and took 5.4–6.7 s each, so spec Rev 6 refuses a progressive JPEG above 12 megapixels; F36-b-WI-2.1’s local re-run with two progressive 4,000 × 3,000 JPEGs at once read both with no OOM kill, anonymous memory at most 919 MiB and preparations of 2.1 and 2.8 s (its plan’s ## Progress subtask 6). F36-b-WI-2.1 measured it locally in a container at the dev task’s limits (1,024 MB, 0.5 CPU, -XX:MaxRAMPercentage=65, the whole heap committed), but the deployed image, the task’s JVM and Fargate’s accounting are the dev task’s own |
F36-b spec FR-A3, FR-R3, NFR-5; F36-b-WI-2.1 plan Decision 10 and ## Progress subtask 6 (the local figure); .forge/owed.md, “Settle the card-photo preparation’s memory and time on dev in F36-b WI-2.1’s plan” |
With (g) and (h), reads only: CloudWatch’s MemoryUtilization (Maximum, one-minute periods) for dev’s backend service over the minutes of (h)'s burst and of two large card photos uploaded together, aws ecs describe-tasks for any stopped task’s stopCode and reason, and /ecs/dev/backend for OutOfMemoryError. Record the peak percentage and the megabytes it stands for, against WI-2.1’s local peak; a peak above about 90%, or any OOM, goes to Dinil for WI-2.1 Decision 10’s choices. The preparation’s time per read is the prepareMillis of the handler’s Card Read Completed line, which (g) reads |
Dinil | open |
F37-a — Weekly price broadcast to every customer
Spec FR-C1 / AC-25 require these rows; F37-a-WI-2.1 (the price-broadcast-send schedule) adds them. Everything else in
F37-a runs and is proved locally: the sweep is driven by the scheduler stub, which calls its handler (F37-a-WI-1.3’s
tests), and the sends go to the LocalMailboxProvider stand-in. Until row (a)'s schedule exists on an environment, a
confirmed batch’s items there stay WAITING, and the results dialog shows them as Waiting (FR-U4).
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) The EventBridge schedule price-broadcast-send at rate(1 minute), UTC, in the group et-<env>, targeting the scheduler queue et-<env>-scheduler, payload {"jobKind":"price-broadcast-send","idempotencyKey":"price-broadcast-send#<aws.scheduler.scheduled-time>"} — on dev from forge-et-iaac#20 (F37-a-WI-2.1); on UAT and production with each overlay’s next apply from a dev that holds it, which needs a backend with F37-a wave 1 (.forge/owed.md, “Apply the UAT and production overlays with the price-broadcast-send schedule …”). Running cost: 43,200 invocations a month (60 × 24 × 30) against EventBridge Scheduler’s free allowance of 14,000,000 invocations a month, with USD 1.00 per million above it (https://aws.amazon.com/eventbridge/pricing/, read 2026-11-15), so none is charged; and about 86,400 SQS requests a month (a SendMessage and a DeleteMessage per firing, more with redeliveries; the worker’s receives already happen through long polling) against SQS’s free allowance of 1 million requests a month — which is the account’s, counted across all regions, shared by every queue and already used in part by the backend’s long polling — with USD 0.40 per million Standard-queue requests above it in US East (https://aws.amazon.com/sqs/pricing/, read 2026-11-15; the page’s table did not render, so the rate is the first-tier Standard figure from the price data that page loads), so at most about USD 0.04 a month whichever side of the allowance the account falls. Together with F20-g’s minute schedule (row (a) of its block): about 86,400 invocations and 172,800 requests a month, at most about USD 0.07 |
The sweep is a scheduler jobKind (AD #12), never @Scheduled; locally it runs only when its handler is called. Without the schedule, a confirmed batch’s items stay WAITING, which the results dialog shows as Waiting. Once the schedule is on an environment, the backend deployed there must contain F37-a wave 1, or every firing is refused as an unknown job kind and dead-lettered; on dev, a forge-et-be branch deployed by hand must contain it (merge main into it first — .forge/team-guide.md § Environments) |
F37-a spec FR-S1, FR-C1 (a), NFR-7, AC-25; forge-et-be docs/messaging/scheduler-cloud-contract.md (the price-broadcast-send entry F37-a-WI-1.3 adds); forge-et-iaac base/scheduler/locals.tf |
After the merge’s apply, aws scheduler get-schedule --group-name et-dev --name price-broadcast-send --profile et-dev --region us-east-1 (a read) shows ENABLED, rate(1 minute), UTC, the target et-dev-scheduler and that Input; the backend log has no UnknownScheduledJobKindException for the kind and et-dev-scheduler-dlq gains nothing; a confirmed batch’s first items leave Waiting within 2 minutes (with row (b)). The same reads on UAT and production after their overlay’s apply. Update 2026-11-17: created on dev by forge-et-iaac#20 (merge fa20b5e8), applied by dev.yaml run 44085299528 (“Apply complete! Resources: 1 added, 0 changed, 0 destroyed”), after dev’s backend (backend:151, image build-b901e4bb) was confirmed to hold F37-a wave 1; get-schedule reads ENABLED, rate(1 minute), UTC, target et-dev-scheduler and that Input. Five minutes after the apply, /ecs/dev/backend has no UnknownScheduledJobKindException for the kind and et-dev-scheduler-dlq holds 0, as it did before the merge. The sweep logs its completion line at DEBUG when nothing is waiting, so no line is expected yet; a confirmed batch leaving Waiting on dev is still to be seen, in row (b)'s run. UAT and production are still owed. |
Tharinda (with Shyamin Ayesh); UAT and production Tharinda (F38-h, F38-i) | open (dev schedule created, no firing refused; a batch leaving Waiting on dev, and UAT and production, still owed) |
(b) A batch sent from a real Microsoft 365 mailbox on dev at 20 a minute without Microsoft’s 429 |
Locally the sends go to LocalMailboxProvider, which has no Exchange limit; a real send from a Microsoft 365 mailbox is not yet proved on dev (spec W-1; F38-f and F38-g) |
F37-a spec decision 4, FR-S1, FR-S5, NFR-3, FR-C1 (b), W-1 | After F38-f and F38-g prove the mailbox and row (a) is on dev: signed in as a user with email.manage whose mailbox is CRM01, confirm a batch of at least 25 accounts whose primary contacts are test mailboxes the team controls (never a real customer’s address); the results dialog shows 20 sent in the first minute and the rest in the next; no item is deferred or FAILED with a rate-limit reason, and the backend log holds no 429 from Graph for the batch. Record the batch id, the counts per minute and the result here. Update 2026-11-15: a pace below 20 a minute is expected and is not a fault when the CRM’s own rate limiter causes it. Each send takes 4 tokens from F20-c’s per-connection bucket (forge-et-be MailboxProperties.RateLimit: 60 tokens, refilled at 240 a minute by default), and the user’s own mail reads draw from the same bucket, so a minute can send fewer than 20 of a batch, never more; a send the bucket refuses is deferred under a new key (spec FR-S5), and the batch takes longer than decision 4’s estimate. Such a deferral shows as a price_broadcast_item.deferred event with no Microsoft 429 behind it. What this check confirms is that no 429 from Microsoft arrives for the batch and that no batch sends more than 20 emails in any 50 seconds; “20 sent in the first minute” and “no item is deferred” above hold only when the bucket refuses nothing, and a deferral with a Microsoft 429 behind it still fails the check. Source: F37-a-WI-1.3’s plan review (pass 2, N3), its risk “The production pace is lower than 20 a minute”. |
Tharinda (F37-a) | open |
| © A batch’s emails received by a test mailbox with the right To, Cc, standards and tracking | Only a real mailbox shows what Exchange delivered | F37-a spec decisions 1 and 7, FR-S2, FR-C1 ©, AC-7 | In row (b)'s run, or a smaller batch: in a receiving test mailbox, one email per account, To the account’s primary contact, Cc its other active contacts with copies on and empty with copies off, the body listing only that account’s priced standards for the sale; with tracking on, opening the email marks it opened in the CRM (F20-g’s tracking address on dev, §B F20-g row ©). Record the accounts and what each mailbox received here, never an address or a price | Tharinda (F37-a) | open |
F37-b — Bulk email to a saved contact list
Spec FR-C1 / AC-25 require these rows; F37-b-WI-2.1 (the bulk-email-send schedule) adds them. Everything else in
F37-b runs and is proved locally: the sweep is driven by the scheduler stub, which calls its handler (F37-b-WI-1.3’s
tests), the sends go to the LocalMailboxProvider stand-in, a bounce comes from the local stub route that brings a
non-delivery report into the local mailbox (F37-b-WI-1.5), and the unsubscribe pages and the footer run on the local
stack. Until row (a)'s schedule exists on an environment, a confirmed send’s items there stay WAITING, and the
results show them as Waiting.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
(a) The EventBridge schedule bulk-email-send at rate(1 minute), UTC, in the group et-<env>, targeting the scheduler queue et-<env>-scheduler, payload {"jobKind":"bulk-email-send","idempotencyKey":"bulk-email-send#<aws.scheduler.scheduled-time>"} — on dev from forge-et-iaac#29 (F37-b-WI-2.1); on UAT and production with each overlay’s next apply from a dev that holds it, which needs a backend with F37-b wave 1 (.forge/owed.md, “Apply the UAT and production overlays with the bulk-email-send schedule …”). Running cost: 43,200 invocations a month (60 × 24 × 30) against EventBridge Scheduler’s free allowance of 14,000,000 invocations a month, with USD 1.00 per million above it (https://aws.amazon.com/eventbridge/pricing/, read 2026-11-19; the page states the allowance, and the rate is the US East figure from the price data that page loads), so none is charged; and about 86,400 SQS requests a month (a SendMessage and a DeleteMessage per firing, more with redeliveries; the worker’s receives already happen through long polling) against SQS’s free allowance of 1 million requests a month — which is the account’s, counted across all regions, shared by every queue and already used in part by the backend’s long polling — with USD 0.40 per million Standard-queue requests above it in US East (https://aws.amazon.com/sqs/pricing/, read 2026-11-19; the rate is the first-tier Standard figure from the price data that page loads), so at most about USD 0.04 a month whichever side of the allowance the account falls. Together with F20-g’s and F37-a’s minute schedules (row (a) of each block): about 129,600 invocations and 259,200 requests a month, at most about USD 0.10 |
The sweep is a scheduler jobKind (AD #12), never @Scheduled; locally it runs only when its handler is called. Without the schedule, a confirmed send’s items stay WAITING, which the results show as Waiting. Once the schedule is on an environment, the backend deployed there must contain F37-b wave 1, or every firing is refused as an unknown job kind and dead-lettered; on dev, a forge-et-be branch deployed by hand must contain it (merge main into it first — .forge/team-guide.md § Environments) |
F37-b spec FR-S1, FR-C1 (a), NFR-7, AC-25; forge-et-be docs/messaging/scheduler-cloud-contract.md (the bulk-email-send entry F37-b-WI-1.3 adds); forge-et-iaac base/scheduler/locals.tf |
After the merge’s apply, aws scheduler get-schedule --group-name et-dev --name bulk-email-send --profile et-dev --region us-east-1 (a read) shows ENABLED, rate(1 minute), UTC, the target et-dev-scheduler and that Input; the backend log has no UnknownScheduledJobKindException for the kind and et-dev-scheduler-dlq gains nothing; a confirmed send’s first items leave Waiting within 2 minutes (with row (b)). The same reads on UAT and production after their overlay’s apply |
Tharinda (with Shyamin Ayesh); UAT and production Tharinda (F38-h, F38-i) | open |
(b) A send from a real Microsoft 365 mailbox on dev at no more than 20 a minute without Microsoft’s 429 |
Locally the sends go to LocalMailboxProvider, which has no Exchange limit; a real send from a Microsoft 365 mailbox is not yet proved on dev (spec W-1; F38-f and F38-g) |
F37-b spec FR-S1, NFR-3, FR-C1 (b) | After F38-f and F38-g prove the mailbox and row (a) is on dev: signed in as a user whose role holds email.manage and email.bulk_send and whose mailbox is CRM01, send to a saved contact view of at least 25 test mailboxes the team controls (never a real customer’s address); no send is more than 20 in any 50 seconds, no item fails with a rate-limit reason caused by a Microsoft 429, and the backend log holds no 429 from Graph for the send. A slower pace caused by the CRM’s own per-connection rate limiter is expected, as §B F37-a row (b)'s Update 2026-11-15 explains. Record the send id, the counts per minute and the result here |
Tharinda (F37-b) | open |
| © A real Exchange non-delivery report matched to its item | Locally a bounce comes from the stub route; the Graph fields that name the message class and the original’s Internet message id are taken from Microsoft’s documentation (spec Q-1, F37-b-WI-1.5) and are not yet seen on a real report | F37-b spec decision 3, FR-D1, FR-C1 ©, Q-1 | In row (b)'s send, or a smaller one, include one contact whose address is at a domain the team controls but whose mailbox does not exist; when Exchange’s report arrives in CRM01, the item shows Bounced, the address is suppressed as BOUNCED, and the report stays in the mailbox unmoved. Record which match was used (Internet message id, or recipient and subject) and the result here |
Tharinda (F37-b) | open |
| (d) The unsubscribe page reached from a received email, and Microsoft Defender’s Safe Links opening the link without unsubscribing | Only a received email in a Microsoft 365 mailbox has its links rewritten by Safe Links, which opens them with a GET; locally the page is reached directly |
F37-b spec FR-E4, FR-C1 (d) | In a test mailbox that received row (b)'s email, read the message’s link and confirm the address is unchanged by any GET (the contact shows no suppression line after Safe Links has scanned it), then click Unsubscribe from these emails, press the page’s button, and see the contact’s suppression line in the CRM. Record the mailbox and the result here, never an address |
Tharinda (F37-b) | open |
| (e) The unsubscribe link in a tracked email received unrewritten | F20-g’s tracking rewrites links on dev through its public tracking address, which exists only in the cloud | F37-b spec FR-E2, FR-C1 (e); §B F20-g row © | In row (b)'s send with tracking on, the received email’s other links go through the tracking address and the unsubscribe link goes straight to <public address>/api/v1/email/unsubscribe/{token} |
Tharinda (F37-b) | open |
(f) crm.bulk-email.postal-address set to spec decision 12’s address in the dev, uat and production task definitions, and seen in a received email’s footer on dev |
The value is a plain task-definition value (AD #40), which only a deployed environment reads; an empty value refuses every send (422 POSTAL_ADDRESS_MISSING) |
F37-b spec decision 12, FR-E1, FR-C1 (f); the F37-b Decomposition Plan’s ## Notes item 9 |
git -C "$WS/forge-et-be" grep -n -e crm.bulk-email.postal-address -e CRM_BULK_EMAIL_POSTAL_ADDRESS origin/main -- .aws shows the value in td.dev.json and td.uat.json, and in td.prod.json once F38-i adds it; dev’s running task definition carries it (aws ecs describe-task-definition, a read); row (b)'s received email’s footer shows the address |
Tharinda (F37-b) — the dev and uat values by F37-b-WI-1.1 if the lead agrees to the edit of the DevOps team’s deploy files, otherwise Shyamin Ayesh; production Tharinda (F38-i; .forge/owed.md, “Set crm.bulk-email.postal-address in production’s task definition”) |
open |
F38-d — The backend container image (built ahead of the spec: forge-et-be PR #136)
Proved locally on Apple silicon (arm64): the image builds on the Temurin 21 bases, named by tag, with Ubuntu’s updates applied at build time (forge-et-be PR #144), runs as uid 10001 with no application-local* file in it, starts against the compose Postgres and LocalStack with the local profile, answers liveness and readiness UP, shuts down gracefully on SIGTERM, and refuses to start with no configuration. trivy image reports 0 HIGH/CRITICAL in the Ubuntu layer as of 2026-10-02. The lead’s decisions on the rest are in the F38-d and F38-b specs’ Open Questions (2026-10-02). Update 2026-10-19 (F38-d WI-1.1): this block is F38-d’s (spec FR-11); its evidence is in environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The image runs on Fargate ARM64, pulled from ECR, with a cloud-shape task definition |
There is no ECR repository, task definition or Fargate service to run it on yet | F38 FR-20, FR-23, NFR-4; F38-b spec (2026-10-02 decisions) | deploy-dev builds --platform linux/arm64, pushes to ECR, and the service reaches steady state with no Spring profile; the ALB target group is healthy on /actuator/health/liveness. Update 2026-10-23 (F38-d WI-4.2): validated on dev — since forge-et-be#275 the build workflow’s deploy job builds --platform linux/arm64 and pushes crm/backend:build-<full sha> to ECR; read on 22 Oct, the service’s deployment of backend:58 (ARM64) is COMPLETED, the running task reports arm64 and its image’s config linux/arm64, its start log reads No active profile set, falling back to 1 default profile: "default", and the target group et-dev-backend reports the task healthy; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-2 and AC-4. |
F38-d / F38-b | validated |
Task-definition values for the container — stopTimeout 60 s, deregistration delay 30 s, MaxRAMPercentage against the chosen memory, a writable /tmp |
Only a real task and target group carry them | F38-b spec (2026-10-02 decisions) | A rolling deploy under light load drains without 5xx; the task’s JAVA_TOOL_OPTIONS shows in the start log (Picked up JAVA_TOOL_OPTIONS); a multipart upload succeeds. Update 2026-10-19 (F38-d WI-1.1): none of the values is set at forge-et-be 5c457091 or forge-et-iaac#3’s head — the image’s own JAVA_TOOL_OPTIONS (75 %, exit on out-of-memory) applies; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, the review, DR-2 and DR-3. Update 2026-10-26 (F38-d WI-6.1): validated on dev — backend:71 has stopTimeout 60, its task logs Picked up JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=65 -XX:+ExitOnOutOfMemoryError, the target group’s deregistration_delay.timeout_seconds is 30, the first multipart upload (POST /api/v1/email/attachments) answered 201 with the object in et-dev-attachments, and four rolling deploys under light load left no HTTPCode_Target_5XX_Count or HTTPCode_ELB_5XX_Count datapoint (four of 1,048 liveness calls from the laptop got no answer within 5 s, one of them while no deployment was running; the load balancer recorded no 5xx — .forge/owed.md “Look at the four liveness calls that got no answer within 5 s on 26 Oct”); a writable /tmp needs nothing while readonlyRootFilesystem is off (F38-d spec FR-6); evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-2 and AC-7. |
F38-d (WI-4.1 values, WI-3.1 deregistration delay; three values from F38-b’s outputs) | validated |
trivy image in deploy-dev — the OS packages and the JRE are outside AD #27’s trivy fs lockfile scan |
The step lives in a workflow that only runs once the AWS deploy role exists | F38-d spec (2026-10-02 decisions) | The deploy-dev run log shows the image scan’s table, and a finding never fails the run (AD #38) |
F38-d | withdrawn |
| The jar is built on the runner and copied into the runtime image | Only a GitHub-hosted deploy-dev run shows the cache and the minutes saved |
F38-d spec (2026-10-02 decisions); Actions-minutes budget | A second deploy-dev run with no Gradle input change restores setup-gradle’s cache and skips compilation. Update 2026-10-23 (F38-d WI-4.2): since forge-et-be#275 the jar is built in the build job and loaded as the jar build context ([context jar] load from client, no [build stage in Docker); the merge run 39228765669 restored the Fast lane’s cache and took :compileJava from the build cache, and run 39235519408, with no Gradle input change, restored the merge run’s cache but compiled, because its Java sources changed — each shown by its Cache hit for restore-key line; WI-6.1 ticks it with AC-2; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-2. Update 2026-10-26 (F38-d WI-6.1): validated on dev — with run 39235519408’s restore of an earlier run’s cache and run 39228765669’s :compileJava FROM-CACHE, the manual run 40331078887 of a commit already built restored setup-gradle’s cache by its exact key and took :compileJava from the build cache (5 actionable tasks: 4 executed, 1 from cache); evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-2. |
F38-d | validated |
One image per commit: ECR tags are immutable, and deploy-dev does not rebuild a commit ECR already holds |
No ECR repository or deploy-dev workflow exists yet. Since PR #144 two builds of one commit differ (the apt-get upgrade step), so a rebuild would replace the tested image |
F38-d spec (2026-10-02, PR #144 decision); F38 NFR-4, FR-35, AC-41 | Pushing a second image under an existing SHA tag is refused by ECR; a manual deploy-dev re-run for a commit that has an image redeploys that image’s digest, and the digest UAT promotes equals the one dev ran. Update 2026-10-23 (F38-d WI-4.2): since forge-et-be#275 every image is tagged build-<full commit SHA> (ECR holds one such tag for each run since, read on 22 Oct) and deployed by its digest; the refused re-push (ECR IMMUTABLE, F38-d WI-3.1) and a manual run that redeploys a held image are F38-d WI-6.1’s; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-3 and AC-4. Update 2026-10-26 (F38-b WI-6.1): a manual build run on main for a commit ECR already held (40274424861, 30fd601c) skipped the build and push and redeployed the same digest as backend:66, healthy as backend:67 (§4, under “Still WI-6.1’s (AC-2)”). Still open: the refused re-push, and the digest UAT promotes equalling dev’s. Update 2026-10-26 (F38-d WI-6.1): the skip shown again (runs 40331078887 and 40331100359 deployed build-e88fef50…'s digest with docker::build skipped) and IMMUTABLE re-read; the refused re-push is not shown, because aws ecr put-image as svc-harness was refused by IAM (AccessDeniedException, no ecr:PutImage) before ECR’s tag check — .forge/owed.md “F38-d AC-11: show ECR refusing a second push under an existing tag”; the digest UAT promotes is F38-h’s; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-2 and AC-11. Update 2026-10-26 (F38-d WI-7.1): re-read at the closing sweep; the lead chose to leave the refused re-push open rather than accept IMMUTABLE alone or grant a push, so F38-d’s AC-11 carries a Not met note and this row stays open for it and for the digest UAT promotes (F38-h); evidence: §4, AC-10. |
F38-d / forge-et-iaac |
open |
The backend verifies the RDS certificate (sslmode=verify-full, the RDS CA bundle in the image) |
No RDS instance exists to present a certificate; the local Postgres has none | F38-b spec (2026-10-02 decisions) | On dev, the backend connects with verify-full; pointing it at a host name the certificate does not cover is refused Update 2026-10-26 (F38-d WI-6.1): validated on dev — the running revision sets ET_DATASOURCE_SSL_MODE=verify-full and its task’s pool started and Flyway validated 52 migrations; a manual run of a throwaway branch pointing ET_DATASOURCE_HOST at the instance’s IP address was refused by pgjdbc (“The hostname 10.0.1.55 could not be verified by hostnameverifier PgjdbcHostnameVerifier.”, “Application run failed”) and rolled back, and the branch was deleted; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-8. |
F38-d (backend change) / F38-b | validated |
A deploy that ECS rolls back leaves the previous task serving and is recorded failed — revised 2026-10-20 (F38-d spec Rev 3): deploy-dev no longer waits for the rollout (forge-et-be#286), so the run stays green. deploy-dev waits for the service to become stable (wait-for-service-stability), and |
Only a real ECS service shows a rollback; the dev workflow’s deploy step ends in seconds, before ECS starts the task | F38-d spec FR-2, NFR-1, AC-2 (Rev 3); environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, DR-9 (added 2026-10-19 by F38-d WI-1.1; withdrawn 2026-10-20) | A task definition that fails its health check, deployed from a branch by hand and then deleted, leaves the previous task serving (liveness polled throughout), and ECS records the deployment as failed: rolloutState FAILED in describe-services while the rollback runs, the service events, or list-service-deployments ROLLBACK_SUCCESSFUL Update 2026-10-26 (F38-d WI-6.1): validated on dev — a task definition with a health check that always fails, deployed by hand from a throwaway branch, was rolled back by the circuit breaker after three failed tasks: rolloutState FAILED while the rollback ran, the events “deployment failed: tasks failed to start.” and “rolling back to deployment …”, and ROLLBACK_SUCCESSFUL; the previous task (the same ARN) served throughout, liveness answered, and the branch was deleted; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-2. |
F38-d (WI-6.1 proves) | validated |
The CRM deploy’s new build job on dev — the credential-free build job resolving the VITE_* values under environment: dev, the guard failing on a whitespace-only value, and the web-dist download under permissions: {} |
Only a GitHub-hosted run with the dev Environment shows the variables, the guard and the artifact download; the wave-2 merge changes only the workflow, so it starts no run (FR-4’s paths-ignore) |
F38-d spec FR-4, AC-5; .forge/plans/F38-d/F38-d-WI-2.2-plan.md Risks (added 2026-10-19 at its plan review) |
The first non-workflow forge-et-fe merge’s run, or a manual run, deploys with the identity printed and the bundle synced; a VITE_* variable set to a whitespace-only value in a quiet window fails the guard before the build, then is restored. Update 2026-10-23 (F38-d WI-4.2): the deploy half is shown — the pre-merge manual run 39226907639 (the merged file, dispatched on #298’s head) and the first non-workflow merge’s run 39287294321 (8c0bc93a, forge-et-fe#301) each passed check::config, built with the job-level VITE_* values, downloaded web-dist under permissions: {}, printed svc-github and synced and invalidated; the guard failing on a whitespace-only value is F38-d WI-6.1’s; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-1 and AC-12. Update 2026-10-26 (F38-d WI-6.1): validated on dev — with VITE_CSRF_COOKIE_NAME set to a single space in a quiet window, the manual run 40340419206 failed at check::config (“VITE_CSRF_COOKIE_NAME is empty or blank in the dev GitHub Environment’s variables”) with the install, the build and the deploy job skipped, and the variable was restored 37 s later; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-5. |
F38-d (WI-2.2 builds, WI-4.2 or WI-6.1 proves) | validated |
The capture app’s deploy on dev — the credential-free build job resolving the VITE_* values under environment: dev, the guard failing on a whitespace-only value, the web-dist download under permissions: {}, the capture bucket’s Cache-Control and Content-Type on sw.js, the manifest and the install icons (max-age=0, must-revalidate, never immutable), and the invalidation of the capture distribution |
The capture app’s hosting (capture.dev.crm.empire.lk, its bucket and distribution) and forge-et-pwa’s dev Environment do not exist yet (read 2026-10-19), and the wave-5 merge changes only the workflow, so it starts no run (FR-4’s paths-ignore) |
F38-d spec FR-4, AC-5; .forge/plans/F38-d/F38-d-WI-5.1-plan.md (approved 2026-10-19) |
The capture app’s first deploy — a manual run the lead approves, or the next non-workflow forge-et-pwa merge — prints the identity, syncs the bundle and invalidates the capture distribution; curl -sI on sw.js, manifest.webmanifest and an install icon through CloudFront shows max-age=0, must-revalidate (and application/manifest+json for the manifest); a whitespace-only VITE_* value in a quiet window fails the guard, then is restored. Update 2026-10-26 (F38-d WI-7.1): validated on dev — the manual run 40224118047 and the merge runs 40228839882 (forge-et-pwa#93) and 40279386333 each ran check::config before the build with the VITE_* values resolved under environment: dev, downloaded web-dist under permissions: {}, printed svc-github, and invalidated EZ06T8FTKF7UZ only; sw.js, the manifest (application/manifest+json), the three install icons, the touch icon and the favicons are served max-age=0, must-revalidate; a single-space VITE_CSRF_COOKIE_NAME, set with the lead’s approval after the team was told, failed run 40351896602 at check::config with nothing deployed, and was restored 27 s later; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-1, AC-3 and AC-5. |
F38-d (WI-5.1 builds, WI-7.1 proves) | validated |
F38-e — Observability and mail on dev (spec drafted 2026-10-23)
F38-e’s other items are the §B FS-5 rows (log groups, JSON format, field names, the alert topic, SES, the error reporter, the exporter). The row below is the one item F38-e adds. Its evidence is below the table.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
An email when a backend deploy on dev fails. An EventBridge rule on ECS Deployment State Change / SERVICE_DEPLOYMENT_FAILED for et-dev/backend, sent to the alert topic, because deploy-dev does not wait for the rollout and its run stays green when the circuit breaker rolls back |
Only a real ECS deployment emits the event, and an event with an aws. source cannot be sent by hand |
F38-e spec FR-5, AC-4; F38 Rev 22, Rev 23; F38-d spec Rev 3; .forge/owed.md “Email the team when a backend deployment fails on dev” |
Take the failure from the first of: one of F38-d WI-6.1’s deliberate failed deploys (if the rule is applied before it); one deliberate failed deploy through the manual dev deploy of a throwaway branch, at a time agreed with the team; the first real failure. The four developers each get one email naming the service, the deployment and ECS’s reason, and the previous task kept serving (F38-e AC-4) Update 2026-10-25 (F38-e WI-2.1): the rule is ENABLED with the expected pattern and one target, the topic, through the input transformer; the email is F38-e WI-3.1’s (AC-4); evidence: §B F38-e “F38-e evidence — AC-3”. Update 2026-10-26 (F38-d WI-6.1): the first of the listed failures happened — F38-d WI-6.1’s deliberate failed deploy, deployment ecs-svc/4268833645685348749, failed at 14:05:56 UTC and rolled back with the previous task serving — and one subscriber of et-dev-alerts received the email (“A deployment of the backend service on dev failed. ECS’s deployment circuit breaker stopped it.”, with the service, the deployment, the reason “ECS deployment circuit breaker: tasks failed to start.” and the time) and showed it in that session; the second deliberate failure, ecs-svc/1719102153073437308 at 14:17:24 UTC, should have sent another. Whether all four developers got each email is F38-e WI-3.1’s to read; evidence: environments-and-identity.md §4 “F38-d — the dev deploy pipelines, checked”, AC-2 and AC-8. Update 2026-11-21 (F38-e WI-3.1): one deliberate failed deploy (source B, ecs-svc/7652176713461752993, rolled back at 20:42 UTC with the previous task serving throughout) published once to et-dev-alerts, and SNS delivered it to all four subscriptions (4 delivered, 0 failed); each recipient’s inbox report is owed (owed.md); evidence: §B F38-e “F38-e evidence — AC-4”. |
F38-e | validated — 2026-11-21, F38-e WI-3.1 (SNS delivery; the four inbox reports owed) |
Evidence. The subsections below are F38-e’s evidence for its acceptance criteria (spec .forge/specs/F38-e-observability-mail-spec.md). The rules: every AWS command is a read — describe, list or get, plus sts get-caller-identity — made with the CLI profile et-dev (the IAM user svc-harness) in the development account 570417736903, in us-east-1 unless the command names another Region, except where an entry names a call the lead approved in that session; times are UTC; account IDs, ARNs, resource names and message ids are recorded, but no secret value, no access-key ID and no personal email address (spec NFR-4, NFR-5). Every entry has the same shape: a bold line **<YYYY-MM-DD HH:MM> UTC — <who> (<wi-id>), profile et-dev (svc-harness)** (the profile part is left out for a read that is not an AWS call); a text block holding $ <the command>, the output lines that show the result (trimmed with … where lines were cut, never edited) and EXIT=<n>; then Result: — met, not met — <why>, or reported to the lead — .forge/owed.md. Where one command was run in a loop, the block shows it once and a table gives each item’s output. A register row updated from this evidence gets *Update <YYYY-MM-DD> (F38-e WI-<x.y>): <one-sentence result>; evidence: §B F38-e "F38-e evidence — <AC-n or NFR-1>".* appended to its Validate / finish when cloud is up cell, after the row’s existing updates. Each criterion has one #### subsection, in AC order with NFR-1 last, added by the work item that records it and never as an empty heading; a later work item that records more of a criterion already here appends entries to its subsection. A manual work item’s full record stays in its record file (.forge/plans/F38-e/F38-e-WI-3.<n>-record.md); its subsection quotes the lines that show the result and points at the file.
F38-e evidence — AC-1: log retention and the backend’s log format
AC-1’s first half: every log group at 30 days, and the backend’s log driver and format in the committed and the running task definition. The second half — F38-e’s forge-et-iaac PR adds no log source — is recorded by F38-e-WI-2.1 after that PR merges.
2026-10-23 05:29–05:30 UTC — Claude (F38-e-WI-1.2), profile et-dev (svc-harness)
$ aws ec2 describe-regions --profile et-dev --region us-east-1 --query 'Regions[].RegionName' --output text
ap-south-1 eu-north-1 eu-west-3 eu-west-2 eu-west-1 ap-northeast-3 ap-northeast-2 ap-northeast-1 ca-central-1 sa-east-1 ap-southeast-1 ap-southeast-2 eu-central-1 us-east-1 us-east-2 us-west-1 us-west-2
EXIT=0
$ for r in "${regions[@]}"; do
aws logs describe-log-groups --profile et-dev --region "$r" --query 'logGroups[].[logGroupName,retentionInDays]' --output text
done
EXIT=0 for each of the 17 commands
| Region | Log groups (name, retentionInDays) |
|---|---|
us-east-1 |
/ecs/dev/backend 30 |
| The other 16 Regions | none (empty output) |
Result: met — the account has one log group, /ecs/dev/backend, and it keeps 30 days; no group has no retention (None).
2026-10-23 05:30 UTC — Claude (F38-e-WI-1.2)
$ git -C forge-et-be rev-parse --short origin/main
2efd29cd
$ git -C forge-et-be log -1 --format='%h %cI %s' origin/main -- .aws/td.dev.json
c2e89efa 2026-10-22T12:08:38+05:30 chore(F38-d): set dev's mailbox and attachment values in the task definition
$ git -C forge-et-be show origin/main:.aws/td.dev.json > AC-1-3b-1.log
EXIT=0
$ jq -c '.containerDefinitions[] | {logConfiguration, structured: [.environment[] | select(.name == "LOGGING_STRUCTURED_FORMAT_CONSOLE")]}' AC-1-3b-1.log
{"logConfiguration":{"logDriver":"awslogs","options":{"awslogs-group":"/ecs/dev/backend","awslogs-region":"us-east-1","awslogs-stream-prefix":"backend"}},"structured":[{"name":"LOGGING_STRUCTURED_FORMAT_CONSOLE","value":"ecs"}]}
EXIT=0
Result: met — .aws/td.dev.json on forge-et-be origin/main at 2efd29cd ships the container’s stdout through the awslogs driver to /ecs/dev/backend and sets LOGGING_STRUCTURED_FORMAT_CONSOLE=ecs.
2026-10-23 05:30 UTC — Claude (F38-e-WI-1.2), profile et-dev (svc-harness)
$ aws ecs describe-services --profile et-dev --region us-east-1 --cluster et-dev --services backend --query 'services[0].taskDefinition' --output text
arn:aws:ecs:us-east-1:570417736903:task-definition/backend:60
EXIT=0
$ aws ecs describe-task-definition --profile et-dev --region us-east-1 --task-definition arn:aws:ecs:us-east-1:570417736903:task-definition/backend:60 --query "taskDefinition.containerDefinitions[0].[logConfiguration, environment[?name=='LOGGING_STRUCTURED_FORMAT_CONSOLE']]"
[
{
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/dev/backend",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "backend"
}
},
[
{
"name": "LOGGING_STRUCTURED_FORMAT_CONSOLE",
"value": "ecs"
}
]
]
EXIT=0
Result: met — the task definition the et-dev/backend service runs, backend:60, carries the same driver, group and value as the committed file.
2026-10-23 05:30 UTC — Claude (F38-e-WI-1.2)
$ git -C forge-et-iaac rev-parse --short origin/dev
5e078c0
$ git -C forge-et-iaac grep -nE 'aws_cloudwatch_log_group|retention_in_days' origin/dev
origin/dev:base/ecs/log_group.tf:1:resource "aws_cloudwatch_log_group" "backend" {
origin/dev:base/ecs/log_group.tf:3: retention_in_days = 30
origin/dev:base/ecs/outputs.tf:18: value = aws_cloudwatch_log_group.backend.name
EXIT=0
$ git -C forge-et-iaac show origin/dev:base/ecs/log_group.tf
resource "aws_cloudwatch_log_group" "backend" {
name = "/ecs/${var.environment}/backend"
retention_in_days = 30
…
EXIT=0
Result: met — forge-et-iaac origin/dev at 5e078c0 declares one log group, /ecs/${var.environment}/backend (/ecs/dev/backend on dev), with retention_in_days = 30, and the account holds no log group that it does not declare. AC-1’s first half is met; no gap was found.
2026-10-25 08:26 UTC — Claude (F38-e-WI-2.1)
AC-1’s second half: F38-e’s forge-et-iaac PR, #7, adds no log source.
$ gh pr view 7 --repo altrium-io/forge-et-iaac --json state,mergeCommit,mergedAt,files > AC-1-3a-1.log
EXIT=0
$ jq -c '{state,mergedAt,merge:.mergeCommit.oid,files:[.files[]|"\(.path) +\(.additions) -\(.deletions)"]}' AC-1-3a-1.log
{"state":"MERGED","mergedAt":"2026-10-23T10:41:18Z","merge":"1590009f6abdceef37f188636ae4b9b0bc1d105a","files":["README.md +1 -0","base/alerts/data.tf +1 -0","base/alerts/deployment_failed.tf +50 -0","base/alerts/locals.tf +10 -0","base/alerts/outputs.tf +11 -0","base/alerts/topic.tf +47 -0","base/alerts/variables.tf +21 -0","overlays/dev/env.auto.tfvars +7 -0","overlays/dev/main.tf +8 -0","overlays/dev/outputs.tf +5 -0","overlays/dev/variables.tf +6 -0"]}
EXIT=0
$ git -C forge-et-iaac rev-parse --short 1590009^1
acd1cf7
$ git -C forge-et-iaac diff 1590009^1 1590009 > AC-1-3a-2.log
EXIT=0
$ grep -nE 'aws_cloudwatch_log_group|aws_flow_log|access_logs|logging_config|aws_cloudtrail|aws_cloudwatch_metric_alarm|aws_cloudwatch_log_metric_filter|retention_in_days' AC-1-3a-2.log
EXIT=1
Result: met — #7, merged as 1590009 on 23 Oct, adds only the base/alerts module and its wiring in overlays/dev and README.md; its diff against dev before it (acd1cf7) declares no log group, flow log, access log, logging configuration, trail, alarm or metric filter, and sets no retention (the search finds nothing, EXIT=1).
2026-10-25 08:27 UTC — Claude (F38-e-WI-2.1), profile et-dev (svc-harness)
The log groups re-read after the apply, to compare with the table of 23 Oct above.
$ aws ec2 describe-regions --profile et-dev --region us-east-1 --query 'Regions[].RegionName' --output text
ap-south-1 eu-north-1 eu-west-3 eu-west-2 eu-west-1 ap-northeast-3 ap-northeast-2 ap-northeast-1 ca-central-1 sa-east-1 ap-southeast-1 ap-southeast-2 eu-central-1 us-east-1 us-east-2 us-west-1 us-west-2
EXIT=0
$ for r in "${regions[@]}"; do
aws logs describe-log-groups --profile et-dev --region "$r" --query 'logGroups[].[logGroupName,retentionInDays]' --output text
done
EXIT=0 for each of the 17 commands
| Region | Log groups (name, retentionInDays), 25 Oct |
23 Oct (above) |
|---|---|---|
us-east-1 |
/ecs/dev/backend 30 |
/ecs/dev/backend 30 |
| The other 16 Regions | none (empty output) | none |
Result: met — unchanged since 23 Oct: the account has one log group, /ecs/dev/backend, at 30 days, and the apply of #7 created none.
2026-10-25 08:27 UTC — Claude (F38-e-WI-2.1)
$ git -C forge-et-be rev-parse --short origin/main
eccf2992
$ git -C forge-et-be log -1 --format='%h %cI %s' origin/main -- .aws/td.dev.json
96d7c99d 2026-10-23T16:09:17+05:30 chore(F11-c): read the Exa API key from Secrets Manager in dev's task definition
$ git -C forge-et-be show origin/main:.aws/td.dev.json > AC-1-3c-1.log
EXIT=0
$ jq -c '.containerDefinitions[] | {logConfiguration, structured: [.environment[] | select(.name == "LOGGING_STRUCTURED_FORMAT_CONSOLE")]}' AC-1-3c-1.log
{"logConfiguration":{"logDriver":"awslogs","options":{"awslogs-group":"/ecs/dev/backend","awslogs-region":"us-east-1","awslogs-stream-prefix":"backend"}},"structured":[{"name":"LOGGING_STRUCTURED_FORMAT_CONSOLE","value":"ecs"}]}
EXIT=0
Result: met — .aws/td.dev.json on forge-et-be origin/main at eccf2992 still ships stdout through awslogs to /ecs/dev/backend and sets LOGGING_STRUCTURED_FORMAT_CONSOLE=ecs.
2026-10-25 08:28 UTC — Claude (F38-e-WI-2.1), profile et-dev (svc-harness)
$ aws ecs describe-services --profile et-dev --region us-east-1 --cluster et-dev --services backend --query 'services[0].taskDefinition' --output text
arn:aws:ecs:us-east-1:570417736903:task-definition/backend:62
EXIT=0
$ aws ecs describe-task-definition --profile et-dev --region us-east-1 --task-definition arn:aws:ecs:us-east-1:570417736903:task-definition/backend:62 --query "taskDefinition.containerDefinitions[0].[logConfiguration, environment[?name=='LOGGING_STRUCTURED_FORMAT_CONSOLE']]"
[
{
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/dev/backend",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "backend"
}
},
[
{
"name": "LOGGING_STRUCTURED_FORMAT_CONSOLE",
"value": "ecs"
}
]
]
EXIT=0
Result: met — the task definition the et-dev/backend service runs today, backend:62, carries the same driver, group and value.
2026-10-25 08:28 UTC — Claude (F38-e-WI-2.1)
$ git -C forge-et-iaac rev-parse --short origin/dev
1590009
$ git -C forge-et-iaac grep -nE 'aws_cloudwatch_log_group|retention_in_days' origin/dev > AC-1-3d-1.log
EXIT=0
$ cat AC-1-3d-1.log
origin/dev:base/ecs/log_group.tf:1:resource "aws_cloudwatch_log_group" "backend" {
origin/dev:base/ecs/log_group.tf:3: retention_in_days = 30
origin/dev:base/ecs/outputs.tf:18: value = aws_cloudwatch_log_group.backend.name
Result: met — forge-et-iaac origin/dev at 1590009 still declares one log group, /ecs/${var.environment}/backend, at 30 days, and the account holds no group it does not declare: no FR-1 gap.
Result (AC-1 as a whole): met — WI-1.2’s first half (23 Oct) and this work item’s re-reads and second half (25 Oct): every log group keeps 30 days, the committed and the running task definition send the backend’s ECS-format logs to /ecs/dev/backend, and forge-et-iaac#7 added no log source.
F38-e evidence — AC-2: one request followed by its traceId
Recorded 2026-11-21 by F38-e-WI-3.3; the full record is .forge/plans/F38-e/F38-e-WI-3.3-record.md.
2026-11-21 20:27 UTC — Claude for Tharinda (F38-e-WI-3.3), profile et-dev (svc-harness) — the request: the note F38-e WI-3.3 trace check added to the dev test account 1 as CRM01, then deleted at 20:28 UTC.
$ aws logs get-log-events --log-group-name /ecs/dev/backend --log-stream-name backend/backend/1f6af523be9c4afead8e933ff800c90e …
{"@timestamp":"2026-11-21T20:27:17.165058677Z","level":"INFO","logger":"com.empireteas.account.service.RecordNoteService","message":"Record Note Added - [target=ACCOUNT, parentId=1, noteId=55]","traceId":"6b01fea55f8d844684a413d9d1b2b42c","spanId":"33d6b44bb63672a2"}
EXIT=0
2026-11-21 20:27 UTC — Claude for Tharinda (F38-e-WI-3.3), profile et-dev (svc-harness); approved by Tharinda Rajapaksha at 20:16 UTC
$ aws logs start-query --log-group-name /ecs/dev/backend … --query-string 'fields @timestamp, log.level, log.logger, message, traceId, spanId, @message | filter traceId = "6b01fea55f8d844684a413d9d1b2b42c" | sort @timestamp asc'
queryId 147a307a-3520-45e8-888f-c5fee18092d1 — Complete; recordsMatched 2, bytesScanned 931
@timestamp=2026-11-21 20:27:17.165 | log.level=INFO | log.logger=com.empireteas.timeline.service.TimelineEntryService | message=Timeline Entry Inserted - [timelineEventId=66, accountId=1, sourceType=note, sourceId=55, kind=note] | traceId=6b01fea55f8d844684a413d9d1b2b42c | spanId=33d6b44bb63672a2
@timestamp=2026-11-21 20:27:17.165 | log.level=INFO | log.logger=com.empireteas.account.service.RecordNoteService | message=Record Note Added - [target=ACCOUNT, parentId=1, noteId=55] | traceId=6b01fea55f8d844684a413d9d1b2b42c | spanId=33d6b44bb63672a2
EXIT=0
Result: met — the query returned the request’s two lines as ECS JSON. Field names: traceId, spanId, log.level, log.logger and message each came back under its own name with no escaping — workable; the lines carry traceId, not spec FR-2’s trace.id.
F38-e evidence — AC-3: the alert topic, its policy and the failed-deploy rule
The parts of AC-3 and FR-4, FR-5 that need no person: the apply, the topic’s policy, the subscriptions as found, the rule and its target, and the no-change plan. Confirming the subscriptions, and the no-change plan while it is pending, are recorded by F38-e-WI-3.1.
2026-10-25 08:26 UTC — Claude (F38-e-WI-2.1), profile et-dev (svc-harness)
Readiness: the dev.yaml run for the merge of forge-et-iaac#7 applied, and the topic and the rule exist.
$ gh run view 39940862972 --repo altrium-io/forge-et-iaac --json conclusion,headSha,headBranch,event,createdAt,updatedAt,workflowName
{"conclusion":"success","createdAt":"2026-10-23T10:41:20Z","event":"push","headBranch":"dev","headSha":"1590009f6abdceef37f188636ae4b9b0bc1d105a","updatedAt":"2026-10-23T10:42:39Z","workflowName":"dev"}
EXIT=0
$ gh run view 39940862972 --repo altrium-io/forge-et-iaac --log > AC-3-2-2.log
EXIT=0
$ grep -nE 'Plan:|Apply complete!' AC-3-2-2.log
587:… 2026-10-23T10:42:30.9224149Z ^[[1mPlan:^[[0m 8 to add, 0 to change, 0 to destroy.
627:… 2026-10-23T10:42:36.9319344Z Apply complete! Resources: 8 added, 0 changed, 0 destroyed.
EXIT=0
$ aws sns list-topics --profile et-dev --region us-east-1 --query "Topics[?ends_with(TopicArn, ':et-dev-alerts')]" --output text
arn:aws:sns:us-east-1:570417736903:et-dev-alerts
EXIT=0
$ aws events list-rules --profile et-dev --region us-east-1 --name-prefix et-dev-backend-deployment-failed --query 'Rules[].[Name,State]' --output text
et-dev-backend-deployment-failed ENABLED
EXIT=0
Result: met — run 39940862972, the push of 1590009 to dev, succeeded and applied the eight resources (the topic, its policy, four subscriptions, the rule and its target); the topic and the rule are in the account. The rest of the run’s log is not quoted, because its plan output prints each subscription’s address (spec NFR-5).
2026-10-25 08:28 UTC — Claude (F38-e-WI-2.1), profile et-dev (svc-harness)
The topic’s policy and encryption.
$ aws sns get-topic-attributes --profile et-dev --region us-east-1 --topic-arn arn:aws:sns:us-east-1:570417736903:et-dev-alerts --query 'Attributes.{Policy:Policy,Kms:KmsMasterKeyId,Confirmed:SubscriptionsConfirmed,Pending:SubscriptionsPending,Deleted:SubscriptionsDeleted}' --output json > AC-3-4-1.log
EXIT=0
$ jq '{Kms,Confirmed,Pending,Deleted}' AC-3-4-1.log
{
"Kms": null,
"Confirmed": "4",
"Pending": "0",
"Deleted": "0"
}
$ jq '.Policy | fromjson' AC-3-4-1.log
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AlarmsAndRulesOfThisAccount",
"Effect": "Allow",
"Principal": {
"Service": [
"cloudwatch.amazonaws.com",
"events.amazonaws.com"
]
},
"Action": "sns:Publish",
"Resource": "arn:aws:sns:us-east-1:570417736903:et-dev-alerts",
"Condition": {
"StringEquals": {
"aws:SourceAccount": "570417736903"
}
}
}
]
}
EXIT=0
Result: met — the policy has one statement, which allows only cloudwatch.amazonaws.com and events.amazonaws.com to sns:Publish to this topic, when aws:SourceAccount is the dev account 570417736903; the topic has no KMS key, so it is not encrypted (F38-e-WI-1.1 plan D-2, D-3). This is AC-3’s policy half.
2026-10-25 08:28 UTC — Claude (F38-e-WI-2.1), profile et-dev (svc-harness)
The subscriptions, as found. The query leaves out each subscription’s endpoint, so no address is read or recorded.
$ aws sns list-subscriptions-by-topic --profile et-dev --region us-east-1 --topic-arn arn:aws:sns:us-east-1:570417736903:et-dev-alerts --query 'Subscriptions[].[Protocol,SubscriptionArn]' --output text
email arn:aws:sns:us-east-1:570417736903:et-dev-alerts:38ee84af-cdfb-45e1-a910-996642bc48e9
email arn:aws:sns:us-east-1:570417736903:et-dev-alerts:3df8a6bd-f3eb-467b-bd70-23c0138eab2f
email arn:aws:sns:us-east-1:570417736903:et-dev-alerts:4398e436-83f2-4504-952b-17d3e2c00c80
email arn:aws:sns:us-east-1:570417736903:et-dev-alerts:23a9a45f-c01f-4e1b-8503-893e788eed84
EXIT=0
Result: found — four email subscriptions, each with an ARN ending in a UUID (none PendingConfirmation), and the topic counts 4 confirmed, 0 pending, 0 deleted (the entry above). AC-3’s confirmation half is ticked by F38-e-WI-3.1.
2026-10-25 08:28 UTC — Claude (F38-e-WI-2.1), profile et-dev (svc-harness)
The failed-deploy rule, its target, and the alarms.
$ aws events describe-rule --profile et-dev --region us-east-1 --name et-dev-backend-deployment-failed --query '{State:State,Bus:EventBusName,Pattern:EventPattern}' > AC-3-4-3.log
EXIT=0
$ jq '{State,Bus}' AC-3-4-3.log
{
"State": "ENABLED",
"Bus": "default"
}
$ jq '.Pattern | fromjson' AC-3-4-3.log
{
"detail": {
"eventName": [
"SERVICE_DEPLOYMENT_FAILED"
]
},
"detail-type": [
"ECS Deployment State Change"
],
"resources": [
"arn:aws:ecs:us-east-1:570417736903:service/et-dev/backend"
],
"source": [
"aws.ecs"
]
}
EXIT=0
$ aws events list-targets-by-rule --profile et-dev --region us-east-1 --rule et-dev-backend-deployment-failed --query 'Targets[].{Id:Id,Arn:Arn,Paths:InputTransformer.InputPathsMap,Template:InputTransformer.InputTemplate}'
[
{
"Id": "alerts-topic",
"Arn": "arn:aws:sns:us-east-1:570417736903:et-dev-alerts",
"Paths": {
"deployment": "$.detail.deploymentId",
"reason": "$.detail.reason",
"service": "$.resources[0]",
"time": "$.time"
},
"Template": "{\n \"summary\": \"A deployment of the backend service on dev failed. ECS's deployment circuit breaker stopped it.\",\n \"service\": <service>,\n \"deployment\": <deployment>,\n \"reason\": <reason>,\n \"time\": <time>,\n \"check\": \"aws ecs describe-services --cluster et-dev --services backend\"\n}\n"
}
]
EXIT=0
$ aws cloudwatch describe-alarms --profile et-dev --region us-east-1 --query 'length(MetricAlarms)'
0
EXIT=0
Result: met — the rule is ENABLED on the default bus, with the pattern of F38-e-WI-1.1 plan D-5 (source aws.ecs, detail type ECS Deployment State Change, event SERVICE_DEPLOYMENT_FAILED, resource the et-dev/backend service); it has one target, alerts-topic, the topic, through the input transformer of D-6 (the four paths and the JSON-object template, each placeholder a bare JSON value); and dev has no CloudWatch alarm before F38-e-WI-3.1’s test alarm (D-8). Whether the email arrives is AC-4, F38-e-WI-3.1’s.
2026-10-25 08:28 UTC — Claude (F38-e-WI-2.1)
The no-change plan: a dev.yaml run after the merge run that plans without a Terraform change.
$ gh run list --repo altrium-io/forge-et-iaac --workflow dev.yaml --branch dev --limit 10 --json databaseId,headSha,event,conclusion,createdAt > AC-3-5-1.log
EXIT=0
$ jq -r '.[] | "\(.databaseId) \(.headSha[0:7]) \(.event) \(.conclusion) \(.createdAt)"' AC-3-5-1.log
39940862972 1590009 push success 2026-10-23T10:41:20Z
39940853448 acd1cf7 push success 2026-10-23T10:41:15Z
39936834361 d86d2ca push success 2026-10-23T10:07:22Z
…
Result: pending — no dev.yaml run on dev since the merge run 39940862972 that plans without a Terraform change; F38-e-WI-3.1 records it from the next such push to dev or a lead-approved manual run. No run was started.
2026-11-21 20:16 UTC — Claude for Tharinda (F38-e-WI-3.1), profile et-dev (svc-harness) — the subscriptions (R-1).
$ aws sns get-topic-attributes --topic-arn arn:aws:sns:us-east-1:570417736903:et-dev-alerts --query 'Attributes.{Confirmed:SubscriptionsConfirmed,Pending:SubscriptionsPending,Deleted:SubscriptionsDeleted}'
{"Confirmed": "4", "Pending": "0", "Deleted": "0"}
EXIT=0
Read again at 20:45 UTC, after both test emails: the same.
2026-11-21 20:17 UTC — Claude for Tharinda (F38-e-WI-3.1); approved by Tharinda Rajapaksha at 20:16 UTC — the no-change plan (R-2). The newest push run (44678328363, fd846ff) changed overlays/dev/locals.tf and overlays/uat/locals.tf, so a manual run was made.
$ gh workflow run dev.yaml --repo altrium-io/forge-et-iaac --ref dev → run 44945483585 (workflow_dispatch, fd846ff), success
terraform::plan Plan: 0 to add, 1 to change, 0 to destroy.
1 module.ses.aws_sesv2_configuration_set.this will be updated in-place (- max_delivery_seconds = 0 -> null)
terraform::apply Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
EXIT=0
The same diff is in the plans of runs 44678328363 (2026-11-20) and 44488259851 (2026-11-19): it repeats on every run.
Result: AC-3 not met — the policy (F38-e-WI-2.1, 2026-10-25) and four confirmed subscriptions (F38-e-WI-3.1, 2026-11-21) are met, but no plan on dev can read No changes. while the SES configuration set’s diff repeats: BUG-532. Met once the forge-et-iaac fix merges and one manual run reads No changes. (F38-e-WI-3.1 R-2, attempt 2).
F38-e evidence — AC-4: the failed-deploy email
Recorded 2026-11-21 by F38-e-WI-3.1; the full record is .forge/plans/F38-e/F38-e-WI-3.1-record.md.
2026-11-21 20:26–20:45 UTC — Claude for Tharinda (F38-e-WI-3.1), profile et-dev (svc-harness); approved by Tharinda Rajapaksha at 20:16 UTC — source B: the throwaway forge-et-be branch chore/F38-e-WI-3.1-failing-health-check (head d968581, only a container health check exit 1 added to .aws/td.dev.json) deployed by a manual build run (44946286489).
$ aws ecs describe-services --cluster et-dev --services backend --query 'services[0].events[:40].[createdAt,message]'
02:01:54 (task 70837a54496246c686d7844c641530ac) failed container health checks.
02:07:02 (task 0d165230542843d2bf9415e529102bd4) failed container health checks.
02:12:02 (task ab5bc7b905a841b39ccb9cc8fe0ab869) failed container health checks.
02:12:31 (deployment ecs-svc/7652176713461752993) deployment failed: tasks failed to start.
02:12:31 rolling back to deployment ecs-svc/4666656741329097645.
$ aws cloudwatch get-metric-statistics --namespace AWS/SNS --dimensions Name=TopicName,Value=et-dev-alerts … (02:12 +05:30 = 20:42 UTC)
NumberOfMessagesPublished 1.0 · NumberOfNotificationsDelivered 4.0 · NumberOfNotificationsFailed 0.0
EXIT=0
The previous task (1f6af523be9c4afead8e933ff800c90e) served throughout: 169 liveness reads from 20:26:43 to 20:43:10 UTC, none other than 200. Put back: backend:178 deregistered (INACTIVE, 20:44:28 UTC, under a temporary temp-f38e-deregister policy removed at 20:44:39 UTC), the branch deleted; no main build ran in the window.
Result: met on delivery — the failed deploy’s email was published once and delivered to all four subscriptions. Each recipient’s report of the email’s service, deployment, reason and time lines is owed (owed.md).
F38-e evidence — AC-5: the test alarm
Recorded 2026-11-21 by F38-e-WI-3.1; the full record is .forge/plans/F38-e/F38-e-WI-3.1-record.md.
2026-11-21 20:20–20:24 UTC — Claude for Tharinda (F38-e-WI-3.1), profile et-dev (svc-harness); the policy run by the lead; the alarm calls approved by Tharinda Rajapaksha at 20:16 UTC
20:20:36 the lead: aws iam put-user-policy --user-name svc-harness --policy-name temp-f38e-test-alarm
20:23:09 aws cloudwatch put-metric-alarm --alarm-name et-dev-f38e-test-alarm … --alarm-actions arn:aws:sns:us-east-1:570417736903:et-dev-alerts (attempt 2; attempt 1 at 20:22:39 was refused while the policy took effect)
20:23:23 aws cloudwatch set-alarm-state --alarm-name et-dev-f38e-test-alarm --state-value ALARM
20:23:24 history: Successfully executed action arn:aws:sns:us-east-1:570417736903:et-dev-alerts
AWS/SNS at 01:53 +05:30: NumberOfMessagesPublished 1.0 · NumberOfNotificationsDelivered 4.0 · NumberOfNotificationsFailed 0.0
20:24 aws cloudwatch delete-alarms --alarm-names et-dev-f38e-test-alarm → describe-alarms: 0
20:24:30 aws iam delete-user-policy --user-name svc-harness --policy-name temp-f38e-test-alarm → inline: spike-invoke-gateway
EXIT=0
Result: met on delivery — the alarm’s email was delivered to all four subscriptions, and the alarm and its policy are gone. Each recipient’s report is owed (owed.md).
F38-e evidence — AC-6: the alarms handed to UAT and production
2026-10-23 05:35 UTC — Claude (F38-e-WI-1.2)
The register rows spec FR-7 names, found by their text on the harness main at 57ffb91c (before this work item’s edits), each with its Owner and Status cells as read. Each row also carries a dated Update 2026-10-23 naming the lead’s decision.
$ for t in "${row_texts[@]}"; do grep -nF "$t" .forge/cloud-gated-validation.md; done
EXIT=0 for each of the 10 commands, one line each
| Line | Row (as found) | Owner |
Status |
|---|---|---|---|
| 443 | §B FS-5 “Feature-owned alarms that need this substrate” | F38-h (UAT), F38-i (production) | open |
| 462 | §B FS-6 “Entra certificate expiry.” | Tharinda; F38-h, F38-i (the alarm) | open |
| 471 | §B FS-7 “DLQ-arrival and queue-depth alarms” | F38-h, F38-i | open |
| 482 | §B FS-9 “Failed / missed scheduled-job alarm” | F38-h, F38-i | open |
| 657 | §B F11-d “7. A dead-letter alarm on the lead-triage DLQ” |
F38-h, F38-i | closed 2026-11-08 — superseded by F38-i decision 26 |
| 691 | §B F12 “6. The CloudWatch alarms” | F38-h, F38-i | open |
| 725 | §B F20-a “CloudWatch alarm on the RECONNECT_REQUIRED gauge” |
F38-h, F38-i | open |
| 747 | §B F20-b “The lapsed-subscription alarm” | F38-h, F38-i | open |
| 752 | §B F20-b “A dashboard for the unrecognised-client_state counter” |
F38-h, F38-i | open |
| 958 | §C-5 “AI-spend alarm” | billing owners | open |
$ grep -n 'Decide the UAT and production alarm sets' .forge/owed.md
125:- **Decide the UAT and production alarm sets** (Tharinda; raised by the F38-e spec, 2026-10-23, F38 Rev 23 — before the F38-h and F38-i plans): …
148:- **The dev database fixes** (… — *decided 2026-10-23 (the lead, F38-e spec decision 1): not in dev; whether UAT and production get them is the row "Decide the UAT and production alarm sets"* …
EXIT=0
Result: met — every row names F38-h and F38-i, and the AI-spend row the billing owners, each with the 2026-10-23 decision; the owed.md row exists (line 125; line 148 only points at it). Its costs per candidate are added by this work item from the prices in “F38-e evidence — NFR-1”.
2026-10-23 05:30–05:31 UTC — Claude (F38-e-WI-1.2), profile et-dev (svc-harness)
$ for r in "${regions[@]}"; do
aws logs describe-metric-filters --profile et-dev --region "$r" --query 'metricFilters[].[logGroupName,filterName]' --output text
done
EXIT=0 for each of the 17 commands; every output empty
Result: met — dev has no metric filter in any Region.
F38-e evidence — AC-7: the system mail sent with the backend’s task role
Recorded 2026-11-21 by F38-e-WI-3.2; the full record is .forge/plans/F38-e/F38-e-WI-3.2-record.md.
2026-11-21 20:22–20:24 UTC — Claude for Tharinda (F38-e-WI-3.2), profile et-dev (svc-harness); approved by Tharinda Rajapaksha at 20:16 UTC — the probe et-dev-ses-probe:1 (AWS CLI 2.37.12, ARM64, 0.25 vCPU) run in et-dev with overrides.taskRoleArn = et-dev-backend-task, assignPublicIp=ENABLED on subnet-02fb26b2351e546d1 (route 0.0.0.0/0 to an internet gateway, no NAT gateway).
$ aws logs get-log-events --log-group-name /ecs/dev/backend --log-stream-name ses-probe/ses-probe/2487094ed18a4d17933dd550ffdeb803 --start-from-head …
PROBE_START 2026-11-21T20:23:15Z
arn:aws:sts::570417736903:assumed-role/et-dev-backend-task/2487094ed18a4d17933dd550ffdeb803
STS_EXIT=0
010001a1ffc713d2-1b7d4498-5bbd-4066-a749-22d1107f484a-000000
SES_EXIT=0
PROBE_END
EXIT=0
The task stopped with exit code 0; the credential search over its lines printed nothing.
2026-11-21 20:31 UTC — Claude for Tharinda (F38-e-WI-3.2), the CRM01 mailbox in Outlook on the web (Inbox, message details)
Authentication-Results: mx.microsoft.com 1; spf=pass (sender IP is 54.240.8.53) smtp.mailfrom=bounce.dev.crm.empire.lk; dkim=pass (signature was verified) header.d=dev.crm.empire.lk;dkim=pass (signature was verified) header.d=amazonses.com;dmarc=pass action=none
Message-ID: <010001a1ffc713d2-1b7d4498-5bbd-4066-a749-22d1107f484a-000000@email.amazonses.com>
2026-11-21 20:24 UTC — removal, read back: et-dev-ses-probe:1 deregistered (INACTIVE) and deleted (DELETE_IN_PROGRESS), no ACTIVE revision left; the temporary policy temp-f38e-ses-probe (added by the lead at 20:20 UTC) removed at 20:24 UTC, svc-harness’s inline policies back to spike-invoke-gateway.
Result: met — the mail sent under the backend’s task role reached the CRM01 Inbox with SPF, DKIM (dev.crm.empire.lk) and DMARC passing, and the probe and its permission are gone.
F38-e evidence — AC-8: the error reporter, the exporter and the stale comments
2026-10-23 05:31–05:33 UTC — Claude (F38-e-WI-1.2)
The decision half. The FS-5 rows “Where the frontend / PWA error reporter sends errors” and “Trace sampling is set to 1.0 but no span exporter is configured” record the lead’s decision of 2026-10-23 — none for V1 (F38 W-9, Rev 23; F38-e spec decision 3). For a decision of “none” the evidence that can be checked is that nothing was built, so the code is read in place of the environment.
$ git -C forge-et-fe show origin/main:src/app/telemetry.ts # forge-et-fe at 3935f926
…
* Where it will send errors is an open decision (harness `cloud-gated-validation.md` §B FS-5).
…
export function reportError(_report: ErrorReport): void {
// Intentionally empty until a reporter is chosen and wired. See the note above.
}
EXIT=0
$ git -C forge-et-pwa show origin/main:src/app/telemetry.ts # forge-et-pwa at 578013f — the same 17 lines
…
export function reportError(_report: ErrorReport): void {
// Intentionally empty until a reporter is chosen and wired. See the note above.
}
EXIT=0
$ git -C forge-et-be grep -nE 'zipkin-reporter|zipkin2|opentelemetry-exporter|otlp|micrometer-registry-|xray' origin/main -- gradle.lockfile # forge-et-be at 2efd29cd
EXIT=1
$ git -C forge-et-fe grep -nE 'web-vitals|@sentry|aws-rum' origin/main -- package.json
EXIT=1
$ git -C forge-et-pwa grep -nE 'web-vitals|@sentry|aws-rum' origin/main -- package.json
EXIT=1
Result: met — both reportError bodies hold only their comment and send nowhere; neither frontend depends on an error-reporting or RUM package; the backend’s lockfile has no span exporter or meter registry (the Brave tracer and io.zipkin.aws:brave-propagation-aws are on it on purpose — FS-5’s tracer). The code matches “none in V1”; both FS-5 rows are set validated. The two telemetry.ts comments that still call the destination an open decision are on §E already.
2026-10-23 05:31–05:33 UTC — Claude (F38-e-WI-1.2)
The searches of spec FR-9, re-run on the day after a fetch.
$ for r in forge-et-be forge-et-fe forge-et-pwa; do
git -C "$r" fetch -q origin; echo "$r EXIT=$? $(git -C "$r" rev-parse --short origin/main)"
done
forge-et-be EXIT=0 2efd29cd
forge-et-fe EXIT=0 3935f926
forge-et-pwa EXIT=0 578013f
$ git -C forge-et-be grep -nE 'F38-e|FS-5|CloudWatch alarm|alarms? .{0,40}read' origin/main -- src docs
… 48 hits, each in the table below
EXIT=0
$ git -C forge-et-fe grep -n 'open decision' origin/main
origin/main:CLAUDE.md:274: no-op; where it will send errors is an open decision (harness `cloud-gated-validation.md` §B FS-5).
origin/main:src/app/telemetry.ts:6: * Where it will send errors is an open decision (harness `cloud-gated-validation.md` §B FS-5).
EXIT=0
$ git -C forge-et-pwa grep -n 'open decision' origin/main
origin/main:CLAUDE.md:231: no-op; where it will send errors is an open decision (harness `cloud-gated-validation.md` §B FS-5).
origin/main:src/app/telemetry.ts:6: * Where it will send errors is an open decision (harness `cloud-gated-validation.md` §B FS-5).
EXIT=0
Each hit was read in its file (git -C … show origin/main:<path>, a few lines around the hit). A hit is stale when it says F38-e or “FS-5’s cloud half” builds, exports or alarms on something, or that a destination, an exporter or a dashboard is still to be decided; a hit about FS-5’s tracer, its MDC wiring or its tests, or one that says an alarm could read a signal, is still true; on §E row means the file and line are on an existing §E row. Paths under forge-et-be src/main/java/com/empireteas/ are shortened.
| Repo | File and line | Verdict | §E row, or why |
|---|---|---|---|
forge-et-be |
docs/ai/README.md 179 |
on §E row | AiCircuitBreaker.java row (179–181) |
forge-et-be |
docs/ai/bedrock-cloud-contract.md 39 |
on §E row | AiCircuitBreaker.java row (36–39) |
forge-et-be |
docs/api/errors.md 43 |
still true | the FS-5 tracer that fills traceId |
forge-et-be |
docs/messaging/README.md 87 |
on §E row | the dead-letter, queue-depth and scheduler alarms row |
forge-et-be |
docs/messaging/scan-result-cloud-contract.md 49, 125, 131 |
on §E row | AttachmentMetrics.java row (49, 125–131) |
forge-et-be |
docs/messaging/scan-result-cloud-contract.md 260 |
on §E row | the dead-letter, queue-depth and scheduler alarms row |
forge-et-be |
docs/messaging/scheduler-cloud-contract.md 8 |
still true | says FS-9’s producer leg was deferred like FS-5’s and FS-7’s cloud halves; names no alarm, export or dashboard |
forge-et-be |
docs/messaging/scheduler-cloud-contract.md 228 |
on §E row | the dead-letter, queue-depth and scheduler alarms row |
forge-et-be |
ai/service/AiCircuitBreaker.java 27 |
on §E row | its own row (27–28) |
forge-et-be |
attachment/service/AttachmentMetrics.java 12, 13, 47 |
on §E row | its own row (12–13, 47) |
forge-et-be |
attachment/service/ScanResultHandler.java 23 |
on §E row | its own row (23) |
forge-et-be |
attachment/service/ScanResultHandler.java 59 |
stale (uncertain — given a row, plan R-3) | the paragraph 56–60 calls the INFECTED ERROR line “a security signal for an alarm” that “alerts exactly once”; the scan alarms were withdrawn with GuardDuty — new §E row |
forge-et-be |
attachment/service/SkippedScanSeries.java 10 |
stale | “an alarm or a dashboard can read” the skipped-scan series “as it reads AttachmentMetrics’ other series”; no meter is exported (decision 6) and the §E AttachmentMetrics.java row records that nothing reads those series outside the process — new §E row |
forge-et-be |
mailbox/service/MailboxMetrics.java 15, 47, 52, 61, 70 |
on §E row | its own row (15–16, 47–48, 52, 61, 70) |
forge-et-be |
mailbox/service/MailboxRenewalService.java 69 |
on §E row | the MailboxMetrics.java row |
forge-et-be |
platform/async/config/AsyncConfig.java 29 |
still true | FS-5’s MDC task decorator |
forge-et-be |
platform/error/DependencyFailedException.java 16 |
still true | “the alarm for the dependency reads its own metrics” says where such an alarm would look rather than at this WARN line; it names no owner, builder or export |
forge-et-be |
platform/observability/MdcTaskDecorator.java 20, 22; package-info.java 4 |
still true | FS-5’s MDC wiring |
forge-et-be |
platform/scan/GuardDutyFileScanProvider.java 25 |
on §E row | its own row (25) |
forge-et-be |
platform/web/error/ProblemFactory.java 36 |
still true | the FS-5 tracer that fills traceId |
forge-et-be |
src/main/resources/application.yml 216 |
still true | the FS-5 tracer’s log correlation |
forge-et-be |
src/main/resources/application.yml 405 |
on §E row | the management.tracing row (403–409) |
forge-et-be |
src/test/groovy/…/api/ErrorEnvelopeIT.groovy 27, 28; LogFormatIT.groovy 20; TracePropagationIT.groovy 25, 27; probe/ProbeFilter.groovy 20; probe/TraceProbeController.groovy 10, 17 |
still true | FS-5’s tracer and its tests |
forge-et-be |
src/test/groovy/…/mailbox/MailboxObservabilityIT.groovy 32, 48 |
on §E row | the MailboxMetrics.java row |
forge-et-be |
src/test/groovy/…/mailbox/MailboxObservabilityIT.groovy 266 |
still true | a test name: the gauge carries a description “so an alarm author can read it” — a property of the gauge, not a claim that an alarm is built |
forge-et-be |
src/test/groovy/…/mailbox/MailboxRenewalIT.groovy 41 |
on §E row | the MailboxMetrics.java row |
forge-et-be |
src/test/groovy/…/mailbox/MailboxTokenSecrecyIT.groovy 43 |
still true | “the FS-5 cloud log pipeline” — the awslogs shipping that exists (AC-1) |
forge-et-be |
src/test/groovy/…/platform/async/MdcDecoratorCarrySpec.groovy 29; …/platform/observability/MdcTaskDecoratorSpec.groovy 9, 16 |
still true | FS-5’s MDC decorator tests |
forge-et-fe |
CLAUDE.md 274 |
on §E row | the two frontends’ CLAUDE.md row |
forge-et-fe |
src/app/telemetry.ts 6 |
on §E row | its own row (6–8, 16) |
forge-et-pwa |
CLAUDE.md 231 |
on §E row | the two frontends’ CLAUDE.md row |
forge-et-pwa |
src/app/telemetry.ts 6 |
on §E row | its own row (6–8, 16) |
Result: met — two new stale hits, SkippedScanSeries.java 10–11 and ScanResultHandler.java 56–60, get one new §E row (after the last F38-e row), and SkippedScanSeries.java joins the forge-et-be bullet of the .forge/owed.md row “Correct the stale observability comments in the app repos” (ScanResultHandler.java is on it already). No existing F38-e §E row’s line numbers moved. grep -n 'Correct the stale observability comments' .forge/owed.md finds the row (line 137 before this work item’s edits).
F38-e evidence — NFR-1: the cost figures
2026-10-23 05:33–05:35 UTC — Claude (F38-e-WI-1.2), profile et-dev (svc-harness) for the pricing calls
AWS’s price pages were read with a fetch of the page text on 2026-10-23. Where a page’s table is filled by script and its text shows no figure, the figure was read from the AWS Price List API (aws pricing get-products, served from us-east-1); one aws pricing get-attribute-values --profile et-dev --region us-east-1 --service-code AmazonSNS --attribute-name endpointType --output text call (a get, outside the plan’s D-3 list) found the SNS endpoint name to filter on. Each get-products output was saved to its own file with its own EXIT=, then read with jq -r '.PriceList[] | fromjson | .product.attributes as $a | .terms.OnDemand[].priceDimensions[] | "\($a.usagetype)\t\(.beginRange)-\(.endRange)\t\(.pricePerUnit.USD)\t\(.description)"'; the lines under each command are that extract (one line per price dimension, the ones that show the result).
$ aws pricing get-products --profile et-dev --region us-east-1 --service-code AmazonSNS --filters 'Type=TERM_MATCH,Field=location,Value=US East (N. Virginia)' 'Type=TERM_MATCH,Field=endpointType,Value=SMTP' --output json
DeliveryAttempts-SMTP 1000-Inf 0.0000200000 $2.00 per 100,000 Amazon SNS Email/Email-JSON Notifications thereafter
DeliveryAttempts-SMTP 0-1000 0.0000000000 First 1,000 Amazon SNS Email/Email-JSON Notifications per month are free
EXIT=0
$ aws pricing get-products --profile et-dev --region us-east-1 --service-code AmazonSNS --filters 'Type=TERM_MATCH,Field=location,Value=US East (N. Virginia)' 'Type=TERM_MATCH,Field=usagetype,Value=Requests-Tier1' --output json
Requests-Tier1 1000000-Inf 0.0000005000 $0.50 per 1,000,000 Amazon SNS API Requests per month thereafter
Requests-Tier1 0-1000000 0.0000000000 First 1,000,000 Amazon SNS API Requests per month are free
EXIT=0
$ aws pricing get-products --profile et-dev --region us-east-1 --service-code AmazonCloudWatch --filters 'Type=TERM_MATCH,Field=location,Value=US East (N. Virginia)' --output json
USE1-DataScanned-Bytes 0-Inf 0.0050000000 $0.005 per GB log data scanned by CloudWatch Logs Insights queries - US East (Northern Virginia)
CW:AlarmMonitorUsage 0-Inf 0.1000000000 $0.10 per alarm metric month (standard resolution) - US East (Northern Virginia)
CW:MetricMonitorUsage 0-10000 0.3000000000 $0.30 per metric-month for the first 10,000 metrics - US East (Northern Virginia)
USE1-DataProcessing-Bytes 0-Inf 0.5000000000 $0.50 per GB custom log data ingested in Standard log class - US East (Northern Virginia)
…
EXIT=0
$ aws pricing get-products --profile et-dev --region us-east-1 --service-code AmazonCloudWatch --filters 'Type=TERM_MATCH,Field=usagetype,Value=DashboardsUsageHour' --output json
DashboardsUsageHour 0-Inf 3.0000000000 $3.00 per Dashboard per Month (>50 metrics)
EXIT=0
$ aws pricing get-products --profile et-dev --region us-east-1 --service-code AWSEvents --filters 'Type=TERM_MATCH,Field=location,Value=US East (N. Virginia)' --output json
USE1-Invocation-64K-Chunks 0-Inf 0.0000010000 $1.00 per million cross-account invocations with AWS management events in US East (N. Virginia)
USE1-Event-64K-Chunks 0-Inf 0.0000010000 $1.00 per million EventBridge partner events received in US East (N. Virginia)
USE1-Event-64K-Chunks 0-Inf 0.0000010000 $1.00 per million EventBridge custom events received in US East (N. Virginia)
… (22 dimensions; none prices AWS service events delivered to a rule on the same account's default bus)
EXIT=0
$ aws pricing get-products --profile et-dev --region us-east-1 --service-code AmazonECS --filters 'Type=TERM_MATCH,Field=location,Value=US East (N. Virginia)' 'Type=TERM_MATCH,Field=usagetype,Value=USE1-Fargate-ARM-vCPU-Hours:perCPU' --output json
USE1-Fargate-ARM-vCPU-Hours:perCPU 0-Inf 0.0323800000 AWS Fargate - ARM - vCPU - US East (N. Virginia)
EXIT=0
$ aws pricing get-products --profile et-dev --region us-east-1 --service-code AmazonECS --filters 'Type=TERM_MATCH,Field=location,Value=US East (N. Virginia)' 'Type=TERM_MATCH,Field=usagetype,Value=USE1-Fargate-ARM-GB-Hours' --output json
USE1-Fargate-ARM-GB-Hours 0-Inf 0.0035600000 AWS Fargate - ARM - Memory - US East (N. Virginia)
EXIT=0
$ aws pricing get-products --profile et-dev --region us-east-1 --service-code awskms --filters 'Type=TERM_MATCH,Field=location,Value=US East (N. Virginia)' --output json
us-east-1-KMS-Keys 0-Inf 1.0000000000 $1 per customer managed KMS key version in US East (N. Virginia)
…
EXIT=0
| Item | Price as read (us-east-1) |
Source | Read |
|---|---|---|---|
| SNS standard topic | no charge of its own: “You pay only for what you use”, billed by API requests and deliveries; the first 1,000,000 API requests a month free, then $0.50 per million | https://aws.amazon.com/sns/pricing/ (the page text); Price List API AmazonSNS Requests-Tier1 |
2026-10-23 |
| SNS email notifications | the first 1,000 a month free, then $2.00 per 100,000 | Price List API AmazonSNS DeliveryAttempts-SMTP (the page text shows no figure) |
2026-10-23 |
| EventBridge — AWS service events on the default bus, matched by a rule in the same account | no charge line: the Price List API charges custom, partner and cross-account events, API destinations, schema discovery, Pipes and Scheduler, and has no dimension for these; the page says “All event buses carry no minimum charge” and “AWS management events are ingested by the event bus for free”. At most, at the custom-event rate of $1.00 per million, a few failed-deploy events a month cost under $0.01 | https://aws.amazon.com/eventbridge/pricing/; Price List API AWSEvents |
2026-10-23 |
| CloudWatch alarm (standard resolution) | $0.10 per alarm metric a month, “pro-rated based on the number of hours that your alarm exists in a month” | https://aws.amazon.com/cloudwatch/pricing/ (the worked examples); Price List API CW:AlarmMonitorUsage |
2026-10-23 |
| CloudWatch custom metric (what a metric filter produces) | $0.30 per metric a month for the first 10,000 | the page’s examples; Price List API CW:MetricMonitorUsage |
2026-10-23 |
| CloudWatch dashboards | three custom dashboards free, each up to 50 metrics; then $3.00 per dashboard a month | the page text (the free tier); Price List API DashboardsUsageHour (the price) |
2026-10-23 |
| CloudWatch Logs Insights | $0.005 per GB scanned | Price List API USE1-DataScanned-Bytes (the page text shows no figure) |
2026-10-23 |
| CloudWatch Logs ingestion (Standard class) | $0.50 per GB | the page’s examples; Price List API USE1-DataProcessing-Bytes |
2026-10-23 |
| SES outbound email | $0.10 per 1,000 emails (à la carte) | https://aws.amazon.com/ses/pricing/ |
2026-10-23 |
| Fargate, Linux/ARM | $0.03238 per vCPU-hour and $0.00356 per GB-hour ($0.0000089944 per vCPU-second and $0.0000009889 per GB-second on the page), “calculated per second with a 1-minute minimum” | https://aws.amazon.com/fargate/pricing/ (Example 2, “Linux/ARM in US East (N. Virginia)”); Price List API AmazonECS |
2026-10-23 |
| KMS customer-managed key (for an encrypted alert topic, if production chooses one) | $1 per key a month | Price List API awskms us-east-1-KMS-Keys |
2026-10-23 |
Result: met — the figures match the spec’s NFR-1 and the plan’s D-10. F38-e adds no recurring charge: the topic and its four subscriptions cost nothing at a handful of emails a month, and the failed-deploy rule has no charge line; the one-off items are the test alarm for about a day (under $0.01 at $0.10 a month prorated by the hour) and the SES probe (one minute of the smallest task, 0.25 vCPU and 0.5 GB, about $0.0002, and one message at $0.0001). F38-e stays within the cost annexe’s development line “CloudWatch and data transfer”, $4 a month (.forge/client/sow-cost-annexe.md §3).
F38-g — The smoke tier and the retest on dev (spec drafted 2026-10-19)
The items below are the parts of F38-g that dev cannot prove today (F38-g spec FR-6). The smoke
tier and the retest themselves run on dev as it is; these rows carry what waits on a capability dev
does not have yet. When a row’s capability arrives, its checks are run, recorded in F38-g’s retest
report, and the row is set validated.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The capture app’s configurable target and its smoke check — forge-et-pwa’s Playwright target pointed at the capture app’s dev host (no server, no specs), and a capture-app check added to forge-et-fe’s smoke suite for after a forge-et-pwa deploy-dev: its dev host loads and reports its version, with any sign-in it needs chosen then |
The capture app has no dev hosting or deploy pipeline (F38-d W-2; .forge/owed.md “The rest of the dev estate and the capture app’s hosting”). Update 2026-10-26: it has both — hosting since forge-et-iaac#5 (22 Oct) and the deploy since forge-et-pwa build.yaml (F38-d wave 5, first runs 25 Oct); https://capture.dev.crm.empire.lk/ answers 200. The target and the smoke check can be built now; the row stays open until they are. Update 2026-11-03: the configurable target was built by F38-g-WI-1.2; the smoke check is in no work item, so F38-g’s decomposition plan (## Notes, first item) and its tracker note now say it is owed before F38-g closes |
F38-g spec FR-1, FR-2, FR-6; F38 AC-31, AC-34 | With forge-et-pwa’s target set to the capture app’s dev host, no local server starts and no spec runs; after a forge-et-pwa deploy, forge-et-fe’s smoke run by hand, with the capture-app check added, is green in under five minutes |
F38-g | open |
| Retest checks that need the file bucket — record attachments, the weekly workbook upload, the Pipedrive import file (the features the retest report names) | The task definition’s ET_FILES_STORAGE_BUCKET is still a placeholder (F38-d FR-6, W-1); the bucket et-dev-attachments exists since forge-et-iaac#3 was applied on 20 Oct. Update 2026-10-25: the value is set since forge-et-be#300 (22 Oct; et-dev-attachments in the running backend:62), so these checks can be walked now; the row stays open until they are |
F38-g spec FR-4, FR-6; F38 FR-33 | Walk those checks on dev and record them in the retest report | F38-g | open |
| Retest checks that need timed jobs — work EventBridge Scheduler starts, such as F26-g’s price-expiry sweep and the mailbox’s scheduled work (the features the retest report names) | No scheduled job has yet been seen to run on dev (W-1); EventBridge Scheduler’s schedules exist since forge-et-iaac#3 was applied on 20 Oct. Update 2026-10-25: they run — on 24 Oct the backend consumed all 173 scheduler messages, none dead-lettered, and logged the price-expiry sweep at 18:35 UTC (environments-and-identity.md §4, base/sqs), so these checks can be walked now; the row stays open until they are |
F38-g spec FR-4, FR-6; F38 FR-33; AD #12 | Walk those checks on dev once the schedules run, and record them in the retest report | F38-g | open |
| Retest checks that need the mailbox — connecting a mailbox, sync, sending, templates and the weekly price email (the features the retest report names) | The task definition lacks the mailbox token key’s and ingest queue’s values (F38-d FR-6, W-1); the key and the queue exist since forge-et-iaac#3 was applied on 20 Oct. Update 2026-10-25: both values, and the mailbox’s public address, are set since forge-et-be#300 (22 Oct; read in the running backend:62), so these checks can be walked now; no mailbox has been connected on dev yet, and the row stays open until they are walked |
F38-g spec FR-4, FR-6; F38 FR-33 | Walk those checks on dev as a test account with its own mailbox and record them in the retest report | F38-g | open |
| Retest checks of AI features configured for Sonnet 5.5 or Haiku 5.5 | Bedrock on dev serves Haiku 4.5 and refuses Sonnet 5.5 and Haiku 5.5 (quota 0; §C-5) | F38-g spec FR-4, FR-6; F38 FR-33; AD #19 | Walk those checks on dev once Bedrock serves the model, and record them in the retest report | F38-g | open |
F38-f — Proving the platform on dev (spec approved 2026-10-25)
The items below are the parts of F38-f that dev cannot prove today (F38-f spec FR-6). The rest of F38-f — the register walk, the Haiku 4.5 proof and AC-40’s other checks — runs on dev as it is. When a row’s wait ends before F38-f closes, F38-f walks it and sets it; only the Sonnet 5.5 row is expected to outlive F38-f (F38 Rev 25). Its evidence is below the table.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The Sonnet 5.5 half of the live Bedrock proof — one call to Sonnet 5.5 through its Global profile from the task role; email-draft (F11-h), account-briefing (F13) and lead-research (F11-c) measured on Sonnet 5.5 for quality, latency and cost per call, as F38-f FR-3 measures them on Haiku 4.5 |
Bedrock refuses Sonnet 5.5 in the dev account (quota 0; §C-5), so dev’s task definition overrides the three features to Haiku 4.5 | F38-f spec FR-2, FR-3, FR-6; F38 AC-29, Rev 25; AD #19 | Once AWS raises the quota above 0 (.forge/owed.md, Shyamin Ayesh): remove the three overrides from forge-et-be .aws/td.dev.json (each override’s own row in the F11-c, F11-h and F13 blocks), make the call, measure the three features, and record them as F38-f’s record does |
Tharinda (quota: Shyamin Ayesh) | open |
The capture app’s AC-40 check — it installs in Chromium on its dev host, and its /api/ requests come from the network, never the cache |
The capture app has hosting but no deploy workflow (F38-d wave 5). Update 2026-10-26: it has both — forge-et-pwa build.yaml (F38-d wave 5) deploys it since 25 Oct, and https://capture.dev.crm.empire.lk/ answers 200 (“Start here” step 13). The check can be run now; the row stays open until it is Update 2026-11-21 (F38-f WI-2.5): on capture.dev.crm.empire.lk (build ec0eec5), sw.js controls the page, the manifest is installable in form (name, standalone, 192 and 512 icons), a same-origin /api/ request came over the network and the cross-origin API answered 401, and the worker’s cache (62 entries) holds no /api/ URL; the install itself was not run (the browser pane cannot install an app) — .forge/owed.md, “Install the capture app from dev for F38-f WI-2.5”; record .forge/plans/F38-f/F38-f-WI-2.5-record.md. |
F38-f spec FR-8 (g); F38 AC-40 | After F38-d wave 5 deploys forge-et-pwa to dev, install it in Chromium and record the network panel for one /api/ request |
F38-f | open |
AC-40’s database checks — btree_gist present, every application table in public, Flyway’s first entry, and the one USER attachment.scan_skipped audit row for F38-f’s upload |
Nobody can reach the dev database (environments-and-identity.md §4 item 15) |
F38-f spec FR-7, FR-8 (b), FR-8 (e); F38 AC-40, Rev 25 | Once F38-f’s ECS Exec tunnel and dev_readonly login exist, a person runs the plan’s read-only queries and records their output. Update 2026-10-26 (F38-f spec Rev 4, F38 Rev 28): no tunnel is built. The schema half is read from Flyway’s startup lines and the migrations on main (F38-f WI-3.1); the audit row from the audit service’s Audit Event Recorded line, switched to DEBUG through /actuator/loggers for a short, announced window (F38-f WI-2.6). |
F38-f | open |
Evidence. The subsections below are F38-f’s evidence for its acceptance criteria (spec .forge/specs/F38-f-prove-platform-on-dev-spec.md). The rules: every AWS command is a read — describe, list or get, CloudWatch Logs reads and Cost Explorer reads, plus sts get-caller-identity — made with the CLI profile et-dev (the IAM user svc-harness) in the development account 570417736903, in us-east-1 unless the command names another Region, except a call the lead approved in that session, which its entry names together with the approval; secretsmanager get-secret-value is never called; times are UTC; account IDs, ARNs, resource names, task IDs and log lines are recorded, but no secret value, token, password, session cookie, access-key ID or personal email address (spec NFR-1, NFR-2). Log lines are recorded as their @timestamp, level, logger and message only. Every entry has the same shape: a bold line **<YYYY-MM-DD HH:MM> UTC — <who> (<wi-id>), profile et-dev (svc-harness)** (the profile part is left out for a read that is not an AWS call); a text block holding $ <the command>, the output lines that show the result (trimmed with … where lines were cut, never edited) and EXIT=<n>; then Result: — met, not met — <why>, or reported to the lead — .forge/owed.md. Where one command was run in a loop, the block shows it once and a table gives each item’s output. A register row updated from this evidence gets *Update <YYYY-MM-DD> (F38-f WI-<x.y>): <one-sentence result>; evidence: §B F38-f "F38-f evidence — <AC>".* appended to its Validate / finish when cloud is up cell, after the row’s existing updates. Row statuses: a work item appends its update to each row its evidence bears on; it may change the Status text to record a half its evidence meets, and sets a final status only when its evidence meets the row’s whole Validate / finish when cloud is up step; a row another work item or sibling owns keeps that owner’s status, and F38-f-WI-3.1’s walk judges what is left (spec FR-1). Each subsection is a #### heading F38-f evidence — <AC> — <title>, added only with content and inserted in the order of the table below; a later work item that records more of a subsection already here appends entries to it. A manual work item’s full record is its record file .forge/plans/F38-f/F38-f-WI-<n.m>-record.md, written only by the person who ran it; its subsection quotes the lines that show the result and points at the file. Each AWS change F38-f makes (AC-10) is listed in environments-and-identity.md §4 under ### F38-f — changes made on dev, and put back, placed after ### F38-d — the dev deploy pipelines, checked and added by the first work item that makes a change, one entry per change with the lead’s approval, the time it was made and the time it was put back.
| Subsection | What it records | Added by |
|---|---|---|
| AC-1 | the register walk | F38-f-WI-3.1 |
| AC-2 and AC-3 | Haiku 4.5 from the task role, and each AI feature | F38-f-WI-2.1 |
| AC-4 | the manual path | F38-f-WI-2.2 |
| AC-5 | the Bedrock spend | F38-f-WI-3.1 |
| AC-6 | the FR-6 rows at close | F38-f-WI-3.1 |
| AC-8 (a) | no direct path to a task | F38-f-WI-1.1 |
| AC-8 (b) | the schema, from Flyway’s startup lines | F38-f-WI-3.1 |
| AC-8 © | a password changed by hand | F38-f-WI-2.3 |
| AC-8 (d) | the schedules | F38-f-WI-1.1 |
| AC-8 (e) | an upload with scanning off | F38-f-WI-1.1 (the startup WARN), F38-f-WI-2.6 (the upload and its audit line) |
| AC-8 (f) | the mailbox key grant | F38-f-WI-1.1 (the expected outcome), F38-f-WI-2.4 (the run) |
| AC-8 (g) | the capture app | F38-f-WI-2.5 |
| AC-8 (h) | alternating users | F38-f-WI-1.1 |
| AC-9 | the spec’s records | F38-f-WI-1.1 |
| AC-10 | the AWS changes, checked | F38-f-WI-3.1 |
| AC-11 | the run’s summary | F38-f-WI-3.1 |
Renumbered 2026-10-26 (spec Rev 4). No tunnel is built, so AC-7 is withdrawn and has no subsection, and the work items were renumbered. Entries dated before that change keep the numbers they were written with: their WI-3.1 – WI-3.5 are now WI-2.1 – WI-2.5, their WI-5.1 is WI-3.1, and their WI-2.1 (the tunnel PR) and WI-4.1 (the tunnel in use) were withdrawn — the upload and its audit row moved to WI-2.6.
F38-f evidence — AC-2 and AC-3 — Haiku 4.5 from the task role, and each AI feature
Recorded 2026-11-21 by F38-f-WI-2.1; the full record is .forge/plans/F38-f/F38-f-WI-2.1-record.md.
2026-11-21 21:33 UTC — Claude, in Tharinda Rajapaksha’s session (F38-f-WI-2.1), profile et-dev (svc-harness)
$ aws logs filter-log-events --profile et-dev --region us-east-1 --log-group-name /ecs/dev/backend --start-time 1795295324000 --filter-pattern '"AI Call Completed"' --query 'events[].message' --output json
…
{"@timestamp":"2026-11-21T21:09:08.658522085Z","level":"INFO","logger":"com.empireteas.ai.service.AiInvocationRecorder","message":"AI Call Completed - [invocationId=550, feature=inquiry-extract, outcome=SUCCESS] [provider=bedrock, model=us.anthropic.claude-haiku-4-5-20251001-v1:0, promptId=inquiry-extract, promptVersion=v1, inputTokens=2353, outputTokens=429, costUsd=0.004948, latencyMs=2090, attempts=1]"}
…
EXIT=0
$ aws cloudtrail lookup-events --profile et-dev --region us-east-1 --lookup-attributes AttributeKey=EventName,AttributeValue=Converse --start-time 2026-11-21T21:08:44Z --output json
(projected to EventTime, EventName, userIdentity.arn, requestParameters.modelId, awsRegion)
…
{"EventTime":"2026-11-22T02:39:08+05:30","EventName":"Converse","arn":"arn:aws:sts::570417736903:assumed-role/et-dev-backend-task/1f6af523be9c4afead8e933ff800c90e","modelId":"us.anthropic.claude-haiku-4-5-20251001-v1:0","awsRegion":"us-east-1"}
…
EXIT=0
Result: met (AC-2) — the first Extract (21:09:01 UTC) was answered by Haiku 4.5 through the us. profile, SUCCESS in 2 090 ms, and CloudTrail records the Converse call at 21:09:08 UTC from the backend task role, task 1f6af523be9c4afead8e933ff800c90e (the running task of backend:177).
2026-11-21 21:36 UTC — Claude, in Tharinda Rajapaksha’s session (F38-f-WI-2.1), profile et-dev (svc-harness) — each AI feature on forge-et-be main (abc9f368) exercised through the app on test data, from the same log read (R-7-7.2-1.log, 41 calls, EXIT=0) and the screens; every call matched to an action of the run.
| Feature | Pinned model | Model run on (dev) | Calls | Highest latencyMs |
Total costUsd |
Quality, in one line | Exa searches |
|---|---|---|---|---|---|---|---|
inquiry-extract (F12) |
Haiku 4.5 us. |
Haiku 4.5 us. |
3 | 2 204 | 0.014299 | all three evaluation cases match expected; the injected instruction was not followed |
— |
lead-triage (F11-d) |
Haiku 4.5 us. |
Haiku 4.5 global. |
11 | 1 531 | 0.028382 | the three triage cases meet their spam flag and summary terms; jobs finished within 1.6 s | — |
lead-research (F11-c) |
Sonnet 5.5 | Haiku 4.5 global. (dev override) |
10 | 5 258 | 0.090133 | invented companies nothing_found; Twinings done with three working, supporting links; one invented name matched a real firm and cited a differently named one |
16 (10 company, 6 person), all SUCCESS, no pause |
email-draft (F11-h) |
Sonnet 5.5 | Haiku 4.5 us. (dev override) |
3 | 1 337 | 0.009837 | no forbidden term or price; signed by the test account; first name only; none sent | — |
account-briefing (F13) |
Sonnet 5.5 | Haiku 4.5 us. (dev override) |
3 | 4 698 | 0.011568 | no forbidden term; the price in the notes did not leak; the injected note was not followed | — |
lead-ranking (F11-g) |
Haiku 4.5 us. |
Haiku 4.5 global. |
11 | 3 523 | 0.040831 | 9 SUCCESS; 2 INVALID_OUTPUT (NO_TOOL_USE, 2 attempts) on the two leads with no inquiry; P1 on an injection inquiry |
— |
The busiest minute held 6 of the run’s calls (under 10 a minute); no call was throttled. The three Sonnet-pinned features ran on Haiku 4.5, so they are not the pinned model’s measurement (the Sonnet 5.5 row above). The capture app’s five AI features (card-read, plan-research, plan-ranking, capture-research, capture-ranking) were not exercised: they run only through the capture app (record R-6c; .forge/owed.md).
Result: met (AC-3) for the five features of the spec — each answered on the model dev runs it on, with a quality note, every call’s latency and cost; no synchronous call came near 10 000 ms. lead-ranking’s two failures are reported to the lead — .forge/owed.md.
F38-f evidence — AC-4 — the manual path
Recorded 2026-11-21 by F38-f-WI-2.2; the full record is .forge/plans/F38-f/F38-f-WI-2.2-record.md. The six CRM AI features — the spec’s five and lead-ranking (F11-g), which forge-et-be main abc9f368 also runs on every new lead — were switched off on dev by one manual build run of the unmerged branch chore/F38-f-manual-path-check; main was deployed again straight after. The five capture-app features were left on. The run was not announced, by the lead’s decision.
2026-11-21 22:08 UTC — Claude, in Tharinda Rajapaksha’s session (F38-f-WI-2.2), gh as tharinda-altrium; approved by Tharinda Rajapaksha at 20:16 UTC
$ git -C forge-et-be push -u origin chore/F38-f-manual-path-check # head b679b6e60b1956fc821f2380fa870aaaf6772dc5, one change: six ET_AI_FEATURES_*_ENABLED=false lines in .aws/td.dev.json (6 0)
* [new branch] chore/F38-f-manual-path-check -> chore/F38-f-manual-path-check
EXIT=0
$ gh workflow run build.yaml --repo altrium-io/forge-et-be --ref chore/F38-f-manual-path-check # 22:08:15
https://github.com/altrium-io/forge-et-be/actions/runs/44955151808
EXIT=0
2026-11-21 22:13 UTC — Claude, in Tharinda Rajapaksha’s session (F38-f-WI-2.2), profile et-dev (svc-harness)
$ aws ecs describe-services --profile et-dev --region us-east-1 --cluster et-dev --services backend --query 'services[0].deployments[].[status,rolloutState,taskDefinition,runningCount]' --output text
PRIMARY COMPLETED arn:aws:ecs:us-east-1:570417736903:task-definition/backend:179 1
EXIT=0
$ aws ecs describe-task-definition --profile et-dev --region us-east-1 --task-definition backend:179 --query 'taskDefinition.containerDefinitions[0].environment[?ends_with(name, `_ENABLED`)]' --output text
ET_AI_FEATURES_EMAILDRAFT_ENABLED false
ET_AI_FEATURES_LEADTRIAGE_ENABLED false
ET_AI_FEATURES_ACCOUNTBRIEFING_ENABLED false
ET_AI_FEATURES_INQUIRYEXTRACT_ENABLED false
ET_AI_FEATURES_LEADRANKING_ENABLED false
ET_AI_FEATURES_LEADRESEARCH_ENABLED false
EXIT=0
Result: met — run 44955151808 ended success, and the branch was deployed at 22:13:40 UTC on backend:179.
2026-11-21 22:18 UTC — Claude, in Tharinda Rajapaksha’s session (F38-f-WI-2.2), profile et-dev (svc-harness) — each feature’s manual path, from the screens and one log read
$ aws logs filter-log-events --profile et-dev --region us-east-1 --log-group-name /ecs/dev/backend --start-time 1795299222000 --filter-pattern '?"AI Call Completed" ?"Lead Research Completed" ?"Lead Triage Completed" ?"Lead Ranking Completed"' --query 'events[].message' --output json
…
EXIT=0
$ aws logs filter-log-events --profile et-dev --region us-east-1 --log-group-name /ecs/dev/backend --start-time 1795299222000 --end-time 1795299476000 --filter-pattern '"Search Call Completed"' --query 'events[].message' --output json # counted with jq length
0
EXIT=0
| Feature | Input | What the screen showed | Log line |
|---|---|---|---|
inquiry-extract (F12) |
evaluation-set — inquiry-extract-v1.json case importer-full |
“AI extraction isn’t available right now. You can fill in the lead yourself.” with Continue Without AI | 22:14:13 invocationId=591, feature=inquiry-extract, outcome=DISABLED, attempts=0 |
lead-triage (F11-d) |
evaluation-set — lead-triage-v3.json case bulk-importer-full (lead 146) |
“The AI summary isn’t available for this lead right now.” with the duplicate match shown | 22:14:52 invocationId=592, feature=lead-triage, outcome=DISABLED; Lead Triage Completed - [leadId=146, outcome=DISABLED …] |
lead-research (F11-c) |
lead 146 | “Research isn’t available right now. It will be retried automatically.” | 22:14:52 Lead Research Completed - [leadId=146, state=unavailable, outcome=DISABLED, stage=ai, aiInvocationId=null …]; no search |
lead-ranking (F11-g) |
lead 146 | “The AI ranking isn’t available for this lead right now.” | 22:14:53 invocationId=593, feature=lead-ranking, outcome=DISABLED; Lead Ranking Completed - [leadId=146, outcome=DISABLED …] |
email-draft (F11-h) |
lead 146 | Draft Follow-Up: “AI drafting is unavailable right now. You can write the email yourself.”; Compose Email still opened the compose form (cancelled, nothing sent) | 22:15:20 invocationId=594, feature=email-draft, outcome=DISABLED |
account-briefing (F13) |
evaluation-set, adapted — account-briefing-v1.json case busy-account (account 45, one note) |
“The AI briefing isn’t available right now. Everything else on this page works as usual.”; profile, contacts, notes and timeline loaded | 22:16:33 invocationId=595, feature=account-briefing, outcome=DISABLED |
Result: met — each feature showed its manual path with the rest of the screen working; every DISABLED line has attempts=0 and no cost, and no search was made while the switches were off.
2026-11-21 22:17 UTC — Claude, in Tharinda Rajapaksha’s session (F38-f-WI-2.2), gh as tharinda-altrium; approved by Tharinda Rajapaksha at 20:16 UTC
$ gh workflow run build.yaml --repo altrium-io/forge-et-be --ref main # 22:17:56
https://github.com/altrium-io/forge-et-be/actions/runs/44956014556
EXIT=0
$ aws ecs describe-services --profile et-dev … (as above) # 22:22:22
PRIMARY COMPLETED arn:aws:ecs:us-east-1:570417736903:task-definition/backend:180 1
EXIT=0
$ aws ecs describe-task-definition --profile et-dev --region us-east-1 --task-definition backend:180 --query 'taskDefinition.containerDefinitions[0].environment[?starts_with(name, `ET_AI_FEATURES_`)]' --output text
… eight _MODEL lines, no _ENABLED line …
EXIT=0
Result: met — run 44956014556 on main abc9f368 ended success, and main was deployed again at 22:22:22 UTC on backend:180, with no _ENABLED override.
2026-11-21 22:27 UTC — Claude, in Tharinda Rajapaksha’s session (F38-f-WI-2.2), profile et-dev (svc-harness) — each feature answering again
$ aws logs filter-log-events --profile et-dev --region us-east-1 --log-group-name /ecs/dev/backend --start-time 1795298817000 --filter-pattern '"AI Call Completed"' --query 'events[].message' --output json
…
EXIT=0
| Feature | Input | Log line after 22:22:22 UTC |
|---|---|---|
inquiry-extract |
the same case | 22:23:02 invocationId=596, SUCCESS, us. Haiku 4.5, 2 501 ms |
lead-triage |
a second lead from the same case (lead 147) | 22:23:39 invocationId=597, SUCCESS, global. Haiku 4.5, 1 295 ms |
lead-research |
lead 147 | 22:23:44 invocationId=598, SUCCESS, global., 3 771 ms; one COMPANY search (searchCallId=151) between its Enqueued (22:23:38) and Completed (22:23:44, nothing_found) lines |
lead-ranking |
lead 147 | 22:23:46 invocationId=599, SUCCESS, global., 1 656 ms |
email-draft |
lead 146 | 22:25:22 invocationId=600, SUCCESS, us., 1 326 ms; the draft discarded, nothing sent |
account-briefing |
account 45 | 22:25:45 invocationId=601, SUCCESS, us., 1 692 ms |
Result: met — each feature answered through Bedrock again, and each screen showed its answer.
2026-11-21 22:26 UTC — Claude, in Tharinda Rajapaksha’s session (F38-f-WI-2.2); approved by Tharinda Rajapaksha at 20:16 UTC
$ git -C forge-et-be push origin --delete chore/F38-f-manual-path-check
- [deleted] chore/F38-f-manual-path-check
EXIT=0
$ git -C forge-et-be ls-remote --heads origin chore/F38-f-manual-path-check
EXIT=0
$ gh pr list --repo altrium-io/forge-et-be --head chore/F38-f-manual-path-check --state all --json number
[]
EXIT=0
$ git -C forge-et-be merge-base --is-ancestor b679b6e60b1956fc821f2380fa870aaaf6772dc5 origin/main; echo "is-ancestor=$?"
is-ancestor=1
EXIT=0
Result: met (AC-4) — the branch chore/F38-f-manual-path-check (head b679b6e6) was deleted unmerged and was never in a PR. The two leads (146, 147) and the account (45) the run created are archived. The two runs billed 2 minutes each. The AWS changes are listed in environments-and-identity.md §4 under #### F38-f-WI-2.2 — the manual path.
F38-f evidence — AC-8 (a) — no direct path to a task
FR-8 (a): a request to a running task’s public IP on the application port is refused, and the same request through the load balancer succeeds. The path is the target group’s health check (forge-et-iaac origin/dev 1590009 base/alb/alb.tf, path = "/actuator/health/liveness"); the port is td.dev.json’s containerPort 8080.
2026-10-26 04:49 UTC — Claude (F38-f-WI-1.1), profile et-dev (svc-harness)
$ aws ecs list-tasks --profile et-dev --region us-east-1 --cluster et-dev --service-name backend --desired-status RUNNING --output json
{
"taskArns": [
"arn:aws:ecs:us-east-1:570417736903:task/et-dev/0d5b878055164e81a40992775c68dae0"
]
}
EXIT=0
$ aws ecs describe-tasks --profile et-dev --region us-east-1 --cluster et-dev --tasks 0d5b878055164e81a40992775c68dae0 --query 'tasks[].[taskArn,startedAt,taskDefinitionArn,attachments[0].details[?name==`networkInterfaceId`].value | [0]]' --output json
…
"arn:aws:ecs:us-east-1:570417736903:task/et-dev/0d5b878055164e81a40992775c68dae0",
"2026-10-26T08:36:13.746000+05:30",
"arn:aws:ecs:us-east-1:570417736903:task-definition/backend:66",
"eni-08379dbf60a05c7b1"
…
EXIT=0
$ aws ec2 describe-network-interfaces --profile et-dev --region us-east-1 --network-interface-ids eni-08379dbf60a05c7b1 --query 'NetworkInterfaces[0].[Association.PublicIp,Groups[].GroupId]' --output json
[
"100.54.218.165",
[
"sg-0097bf895557b7716"
]
]
EXIT=0
$ aws ec2 describe-security-groups --profile et-dev --region us-east-1 --group-ids sg-0097bf895557b7716 --query 'SecurityGroups[].[GroupId,GroupName,IpPermissions]' --output json
…
"sg-0097bf895557b7716",
"et-dev-backend",
[
{
"IpProtocol": "tcp",
"FromPort": 8080,
"ToPort": 8080,
"UserIdGroupPairs": [
{
"Description": "HTTP from the load balancer",
"UserId": "570417736903",
"GroupId": "sg-0cdf10021add79056"
}
],
"IpRanges": [],
"Ipv6Ranges": [],
"PrefixListIds": []
}
]
…
EXIT=0
$ aws ec2 describe-security-groups --profile et-dev --region us-east-1 --group-ids sg-0cdf10021add79056 --query 'SecurityGroups[].[GroupId,GroupName]' --output text
sg-0cdf10021add79056 et-dev-alb
EXIT=0
2026-10-26 04:50 UTC — Claude (F38-f-WI-1.1), from the laptop (not an AWS call)
$ curl -sS -o /dev/null -w '%{http_code}\n' --connect-timeout 10 --max-time 15 http://100.54.218.165:8080/actuator/health/liveness
curl: (28) Failed to connect to 100.54.218.165 port 8080 after 10006 ms: Timeout was reached
000
EXIT=28
$ curl -sS -w '\n%{http_code}\n' --max-time 15 https://api.dev.crm.empire.lk/actuator/health/liveness
{"components":{"livenessState":{"status":"UP"}},"status":"UP"}
200
EXIT=0
| Task ID | Started (UTC) | Task definition | Public IP | Security groups | Direct request: curl exit, HTTP code |
|---|---|---|---|---|---|
0d5b878055164e81a40992775c68dae0 |
2026-10-26 03:06:13 | backend:66 |
100.54.218.165 |
sg-0097bf895557b7716 (et-dev-backend: inbound TCP 8080 from sg-0cdf10021add79056 et-dev-alb only, no IP range) |
28 (timeout), 000 |
Result: met — the one running task has a public IP, but its only security group admits port 8080 from the load balancer’s group alone, so the direct request gets no HTTP response (the packets are dropped, hence the timeout); the same path through https://api.dev.crm.empire.lk answers 200 UP. The public IP is the task’s own and changes with every deployment.
F38-f evidence — AC-8 (d) — the schedules
FR-8 (d): one real EventBridge tick of each schedule on dev reaches the worker, shown by each schedule’s own completion line in CloudWatch. The schedules are those of forge-et-iaac origin/dev on the day, checked against the account. scheduled-demo-job has a handler on forge-et-be and no schedule, and is left out.
2026-10-26 04:51 UTC — Claude (F38-f-WI-1.1)
$ git -C forge-et-iaac rev-parse --short origin/dev
1590009
$ git -C forge-et-iaac show origin/dev:base/scheduler/locals.tf
…
jobs = {
mailbox-subscription-renewal = {
expression = "rate(6 hours)" # et.mailbox.renewal.cadence — change both together
timezone = "UTC"
}
mailbox-delta-poll = {
expression = "rate(15 minutes)"
timezone = "UTC"
}
price-expiry-sweep = {
expression = "cron(5 0 * * ? *)" # 00:05 Colombo, just after the business day changes
timezone = "Asia/Colombo"
}
email-upload-cleanup = {
expression = "rate(1 hour)"
timezone = "UTC"
}
lead-triage-sweep = {
expression = "rate(1 hour)" # requeues leads with no current triage (F11-d FR-J7)
timezone = "UTC"
}
lead-research-sweep = {
expression = "rate(1 hour)" # queues leads with no current research, and paces the Pipedrive backlog (F11-c FR-J5)
timezone = "UTC"
}
}
…
EXIT=0
$ git -C forge-et-be rev-parse --short origin/main
30fd601c
$ git -C forge-et-be grep -n 'JOB_KIND = "' origin/main -- src/main
origin/main:src/main/java/com/empireteas/lead/service/LeadResearchSweepJobHandler.java:20: public static final String JOB_KIND = "lead-research-sweep";
origin/main:src/main/java/com/empireteas/lead/service/LeadTriageSweepJobHandler.java:23: public static final String JOB_KIND = "lead-triage-sweep";
origin/main:src/main/java/com/empireteas/mailbox/service/EmailUploadCleanupJobHandler.java:23: public static final String JOB_KIND = "email-upload-cleanup";
origin/main:src/main/java/com/empireteas/mailbox/service/MailboxDeltaPollJobHandler.java:85: public static final String JOB_KIND = "mailbox-delta-poll";
origin/main:src/main/java/com/empireteas/mailbox/service/MailboxRenewalJobHandler.java:30: public static final String JOB_KIND = "mailbox-subscription-renewal";
origin/main:src/main/java/com/empireteas/platform/scheduling/service/ScheduledDemoJobHandler.java:36: public static final String JOB_KIND = "scheduled-demo-job";
origin/main:src/main/java/com/empireteas/pricing/service/PriceExpirySweepJobHandler.java:30: public static final String JOB_KIND = "price-expiry-sweep";
EXIT=0
$ git -C forge-et-be grep -n -E '"(Mailbox Subscription Renewal|Mailbox Delta Poll|Price Expiry Sweep|Email Upload Cleanup|Lead Triage Sweep|Lead Research Sweep) (Completed|Failed|All Failed|Deferred)' origin/main -- src/main
…
origin/main:src/main/java/com/empireteas/lead/service/LeadResearchSweepService.java:146: "Lead Research Sweep Completed - [idempotencyKey={}, selected={}, queued={}, backlog={}]",
origin/main:src/main/java/com/empireteas/lead/service/LeadTriageSweepService.java:74: "Lead Triage Sweep Completed - [idempotencyKey={}, aiEnabled={}, selected={}, queued={}]",
origin/main:src/main/java/com/empireteas/mailbox/service/EmailUploadCleanupService.java:85: "Email Upload Cleanup Failed - [idempotencyKey={}] [job={}, candidates={}, failed={}]",
origin/main:src/main/java/com/empireteas/mailbox/service/EmailUploadCleanupService.java:92: "Email Upload Cleanup Completed - [idempotencyKey={}] [candidates={}, deleted={}, failed={}]",
origin/main:src/main/java/com/empireteas/mailbox/service/MailboxDeltaPollJobHandler.java:155: "Mailbox Delta Poll Completed - [active={}, reached={}, succeeded={}]",
origin/main:src/main/java/com/empireteas/mailbox/service/MailboxRenewalService.java:166: "Mailbox Subscription Renewal Completed - [due={}, reached={}] [outcomes={}]",
origin/main:src/main/java/com/empireteas/pricing/service/PriceExpirySweepService.java:127: "Price Expiry Sweep Completed - [idempotencyKey={}, sweepId={}, currentSaleId={}, closeDate={}]"
…
EXIT=0
Each Completed line above is logged with log.info (the line before it in each file); Email Upload Cleanup Failed is log.error, logged instead of Completed when every candidate fails.
2026-10-26 04:51 UTC — Claude (F38-f-WI-1.1), profile et-dev (svc-harness)
$ aws scheduler list-schedules --profile et-dev --region us-east-1 --group-name et-dev --query 'Schedules[].[Name,State]' --output text
mailbox-subscription-renewal ENABLED
mailbox-delta-poll ENABLED
email-upload-cleanup ENABLED
lead-triage-sweep ENABLED
price-expiry-sweep ENABLED
lead-research-sweep ENABLED
EXIT=0
$ START=1792896689000 # 2026-10-25 02:51 UTC, 26 hours before the read
$ for l in "${lines[@]}"; do # each completion line below, plus "Email Upload Cleanup Failed"
aws logs filter-log-events --profile et-dev --region us-east-1 --log-group-name /ecs/dev/backend --start-time $START --filter-pattern "\"$l\"" --query 'events[].[logStreamName,message]' --output json
done
EXIT=0 for each of the 7 commands
$ jq -c '[.[] | {stream: .[0]} + (.[1] | fromjson | {"@timestamp", level: .log.level, logger: .log.logger, message})] | sort_by(."@timestamp") | last' <each saved output>
| Schedule | Expression (Terraform) | Account state | Handler (JOB_KIND) |
Completion line | Lines in the window | Latest line: @timestamp, level, logger, message |
|---|---|---|---|---|---|---|
mailbox-subscription-renewal |
rate(6 hours) UTC |
ENABLED |
MailboxRenewalJobHandler |
Mailbox Subscription Renewal Completed |
4 (06:35, 12:35, 18:35 on 25 Oct; 00:35 on 26 Oct) | 2026-10-26T00:35:15.685701338Z INFO com.empireteas.mailbox.service.MailboxRenewalService “Mailbox Subscription Renewal Completed - [due=0, reached=0] [outcomes={}]” |
mailbox-delta-poll |
rate(15 minutes) UTC |
ENABLED |
MailboxDeltaPollJobHandler |
Mailbox Delta Poll Completed |
104 | 2026-10-26T04:50:23.189842410Z INFO com.empireteas.mailbox.service.MailboxDeltaPollJobHandler “Mailbox Delta Poll Completed - [active=2, reached=2, succeeded=2]” |
price-expiry-sweep |
cron(5 0 * * ? *) Asia/Colombo |
ENABLED |
PriceExpirySweepJobHandler |
Price Expiry Sweep Completed |
1 | 2026-10-25T18:35:19.245901185Z INFO com.empireteas.pricing.service.PriceExpirySweepService “Price Expiry Sweep Completed - [idempotencyKey=price-expiry-sweep#2026-10-25T18:35:00Z, sweepId=2c7bdcac-cc00-4604-a091-3b8027cafd3c, currentSaleId=3, closeDate=2026-10-26] [closed=0, held=0, elapsedMs=152]” |
email-upload-cleanup |
rate(1 hour) UTC |
ENABLED |
EmailUploadCleanupJobHandler |
Email Upload Cleanup Completed (Email Upload Cleanup Failed: 0 lines) |
26 | 2026-10-26T04:35:25.702331845Z INFO com.empireteas.mailbox.service.EmailUploadCleanupService “Email Upload Cleanup Completed - [idempotencyKey=email-upload-cleanup#2026-10-26T04:34:58Z] [candidates=0, deleted=0, failed=0]” |
lead-triage-sweep |
rate(1 hour) UTC |
ENABLED |
LeadTriageSweepJobHandler |
Lead Triage Sweep Completed |
26 | 2026-10-26T04:35:12.701831240Z INFO com.empireteas.lead.service.LeadTriageSweepService “Lead Triage Sweep Completed - [idempotencyKey=lead-triage-sweep#2026-10-26T04:34:58Z, aiEnabled=true, selected=0, queued=0]” |
lead-research-sweep |
rate(1 hour) UTC |
ENABLED |
LeadResearchSweepJobHandler |
Lead Research Sweep Completed |
26 | 2026-10-26T04:08:07.322494419Z INFO com.empireteas.lead.service.LeadResearchSweepService “Lead Research Sweep Completed - [idempotencyKey=lead-research-sweep#2026-10-26T04:08:02Z, selected=0, queued=0, backlog=0]” |
Result: met — the six schedules in forge-et-iaac origin/dev are the six in the account’s group et-dev, each ENABLED, and each has its own completion line at INFO in /ecs/dev/backend in the 26 hours read (no re-read over 48 hours was needed); the clean-up job logged no Failed line. The lines come from several task streams because dev was redeployed during the window. The renewal job ticks at 00:35, 06:35, 12:35 and 18:35 UTC, and the delta poll at about 05, 20, 35 and 50 minutes past each hour (AC-8 (f) uses both).
F38-f evidence — AC-8 (e) — an upload with scanning off
FR-8 (e)'s startup half: each running task logged one Malware Scanning Switched Off WARN at startup (F38 Rev 7). The upload stored NOT_SCANNED, its download and its one USER attachment.scan_skipped audit row are F38-f-WI-4.1’s, appended below.
2026-10-26 04:55 UTC — Claude (F38-f-WI-1.1)
$ git -C forge-et-be grep -n 'Malware Scanning Switched Off' origin/main -- src/main
origin/main:src/main/java/com/empireteas/platform/scan/ScanOffFileScanProvider.java:31: "Malware Scanning Switched Off - [{}={}] uploads are stored NOT_SCANNED and are downloadable"
EXIT=0
$ git -C forge-et-be show origin/main:src/main/java/com/empireteas/platform/scan/ScanOffFileScanProvider.java
…
/** Logs, once per context, that uploads are stored {@code NOT_SCANNED}. */
public ScanOffFileScanProvider() {
log.warn(
…
EXIT=0
The line is the only one, logged at WARN by the constructor of a singleton @Component created only when et.files.scan.provider=none, so once per Spring context, which is once per task (forge-et-be origin/main 30fd601c). Each task writes to its own stream backend/backend/<task-id> (td.dev.json: awslogs-stream-prefix backend, container backend).
2026-10-26 04:55 UTC — Claude (F38-f-WI-1.1), profile et-dev (svc-harness)
$ aws logs describe-log-streams --profile et-dev --region us-east-1 --log-group-name /ecs/dev/backend --log-stream-name-prefix backend/backend/0d5b878055164e81a40992775c68dae0 --query 'logStreams[].[logStreamName,firstEventTimestamp,lastEventTimestamp]' --output json
…
"backend/backend/0d5b878055164e81a40992775c68dae0",
1792983957239,
1792987522737
…
EXIT=0
$ aws logs filter-log-events --profile et-dev --region us-east-1 --log-group-name /ecs/dev/backend --log-stream-names backend/backend/0d5b878055164e81a40992775c68dae0 --filter-pattern '"Malware Scanning Switched Off"' --query 'events[].message' --output json
EXIT=0
$ jq -c '.[] | fromjson | {"@timestamp", level: .log.level, logger: .log.logger, message}' <the saved output>
{"@timestamp":"2026-10-26T03:06:58.477255883Z","level":"WARN","logger":"com.empireteas.platform.scan.ScanOffFileScanProvider","message":"Malware Scanning Switched Off - [et.files.scan.provider=none] uploads are stored NOT_SCANNED and are downloadable without a malware scan"}
| Task ID (running, AC-8 (a)) | Stream | First event (UTC) | Malware Scanning Switched Off lines |
The line’s time and level |
|---|---|---|---|---|
0d5b878055164e81a40992775c68dae0 |
backend/backend/0d5b878055164e81a40992775c68dae0 |
2026-10-26 03:05:57 | 1 | 2026-10-26 03:06:58 UTC, WARN |
Result: met — the one running task’s stream holds exactly one Malware Scanning Switched Off line, at WARN, from its startup; the task started inside the log group’s 30-day retention.
Recorded 2026-11-21 by F38-f-WI-2.6; the full record is .forge/plans/F38-f/F38-f-WI-2.6-record.md.
2026-11-21 22:02–22:04 UTC — Claude for Tharinda (F38-f-WI-2.6), as CRM01 in the browser pane’s console; the DEBUG window approved by Tharinda Rajapaksha with the dev windows (20:16, 20:35 UTC) — the audit logger read INFO (no configured level), set to DEBUG at 22:02:05 (204), one 1×1 PNG uploaded through POST /api/v1/email/attachments (201, id 21, NOT_SCANNED), downloaded (200, SHA-256 identical to the upload), discarded (DELETE, 204), and the level put back to null at 22:04:25 (204, read back INFO), on task 384b6f575dd44f1396474f13e83b6a38.
2026-11-21T22:02:16.157Z INFO EmailUploadService Email File Uploaded - [attachmentId=21, userId=1] [sizeBytes=70, scanStatus=NOT_SCANNED]
2026-11-21T22:02:16.144Z DEBUG DefaultAuditService Audit Event Recorded - [action=attachment.uploaded, entityType=attachment, entityId=21, actorKind=USER]
2026-11-21T22:02:16.152Z DEBUG DefaultAuditService Audit Event Recorded - [action=attachment.scan_skipped, entityType=attachment, entityId=21, actorKind=USER]
2026-11-21T22:04:23.559Z DEBUG DefaultAuditService Audit Event Recorded - [action=attachment.deleted, entityType=attachment, entityId=21, actorKind=USER]
EXIT=0
The task’s startup line: WARN Malware Scanning Switched Off - [et.files.scan.provider=none] … — one. No cleanup or price sweep ran inside the window; Part B (the cleanup’s SYSTEM deletion 24 hours later) was not run.
Result: met — the upload stored NOT_SCANNED and downloaded, exactly one attachment.scan_skipped line as USER, the level put back, and one scanning-off WARN per task. The F20-h and F26-g rows’ database halves stay open (no scheduled run in the window).
F38-f evidence — AC-8 (f) — the mailbox key grant
FR-8 (f): the DirectKeys statement of the task role’s inline policy kms is removed while a test mailbox’s token must be refreshed, then put back unchanged. The entry below is the expected outcome, read from the code and the account before the run; F38-f-WI-3.4 appends its run, and the check passes when the run matches this entry and the failure was not silent (a WARN or ERROR line names it).
Expected outcome, read before the run — 2026-10-26 04:56–05:00 UTC — Claude (F38-f-WI-1.1)
The code, at forge-et-be origin/main 30fd601c; each file saved whole with git -C forge-et-be show origin/main:src/main/java/com/empireteas/<path> (EXIT=0 for each), quoted by line number.
mailbox/service/MailboxTokenService.java
115 private static final Duration EXPIRY_SKEW = Duration.ofMinutes(2);
226 @Nullable CachedToken cached = cache.get(connectionId);
227 if (cached != null && cached.serves(neededScopes)) {
228 return Optional.of(cached.token());
296 return refresh(connection, connectionId, neededScopes);
301 String refreshToken = cipher.decrypt(
310 TokenRefreshResult result = provider.refreshToken(connectionId, refreshToken);
470 return Instant.now().isBefore(expiresAt.minus(EXPIRY_SKEW));
mailbox/service/TokenCipher.java
136 public String decrypt(
138 byte[] plaintextKey = keys.unwrap( (before the try at line 141, whose catch is
GeneralSecurityException | IllegalArgumentException)
mailbox/KmsTokenKeyProvider.java
130 response = kms.decrypt(DecryptRequest.builder()
135 } catch (InvalidCiphertextException e) { (the only exception it translates)
mailbox/service/MailboxIngestService.java
190 public String accessToken(long connectionId) {
191 return tokens.accessTokenFor(connectionId)
mailbox/service/MailboxDeltaPollJobHandler.java
140 } catch (RuntimeException e) {
145 log.warn(
146 "Mailbox Delta Poll Failed - [connectionId={}] [reason={}]",
151 log.debug("Mailbox Delta Poll Failure Detail - [connectionId={}]", connectionId, e);
154 log.info(
155 "Mailbox Delta Poll Completed - [active={}, reached={}, succeeded={}]",
162 log.error("Mailbox Delta Poll All Failed - [reached={}, failed={}]", reached, failed);
210 String accessToken = ingest.accessToken(connectionId);
mailbox/service/ProviderFailures.java
29 return e.getClass().getSimpleName(); (for anything but an HTTP response exception)
mailbox/service/MailboxRenewalService.java
219 accessToken = tokens.accessTokenFor(connectionId);
220 } catch (RuntimeException e) {
224 log.warn(
225 "Mailbox Subscription Renewal Deferred - [connectionId={}] [reason={}]",
173 log.error("Mailbox Subscription Renewal All Failed - [reached={}, failed={}]", reached, failed);
mailbox/service/MailboxIngestJobHandler.java
62 : ingest.ingestNotified(live.get(), ingest.accessToken(connectionId), graphMessageId);
$ git -C forge-et-be grep -n 'accessTokenFor' origin/main -- src/main (callers outside MailboxTokenService)
…/mailbox/service/EmailSendService.java:765: token = tokens.accessTokenFor(connectionId);
…/mailbox/service/MailboxConnectionService.java:449: Optional<String> accessToken = tokens.accessTokenForBestEffort(connectionId);
…/mailbox/service/MailboxIngestService.java:191: return tokens.accessTokenFor(connectionId)
…/mailbox/service/MailboxRenewalService.java:219: accessToken = tokens.accessTokenFor(connectionId);
…/mailbox/service/MessageAttachmentService.java:134: token = tokens.accessTokenFor(connectionId);
EXIT=0
$ git -C forge-et-be grep -n -i 'log\.\(info\|warn\|error\)' origin/main -- …/GraphMailboxProvider.java …/KmsTokenKeyProvider.java …/TokenCipher.java …/MailboxTokenService.java
…/MailboxTokenService.java:197 (Mailbox Access Token Unavailable), :266 (Mailbox Token Absent), :320 and :334 (Mailbox Token Scopes Short), :418 (Mailbox Connection Status Changed)
EXIT=0
The account — profile et-dev (svc-harness). The key is td.dev.json’s ET_MAILBOX_TOKEN_KEY_ID (a key ID, not a secret).
$ aws kms describe-key --profile et-dev --region us-east-1 --key-id ce1dbe4f-711e-4dd3-ab6e-9ce1930e9040 --query 'KeyMetadata.[Arn,KeyState,KeyUsage,KeyManager,Description]' --output json
"arn:aws:kms:us-east-1:570417736903:key/ce1dbe4f-711e-4dd3-ab6e-9ce1930e9040",
"Enabled",
"ENCRYPT_DECRYPT",
"CUSTOMER",
"Encrypts the et-dev backend's stored mailbox refresh tokens"
EXIT=0
$ aws kms get-key-policy --profile et-dev --region us-east-1 --key-id ce1dbe4f-711e-4dd3-ab6e-9ce1930e9040 --policy-name default --output text
"Statement" : [ {
"Sid" : "EnableIamPolicies",
"Effect" : "Allow",
"Principal" : {
"AWS" : "arn:aws:iam::570417736903:root"
},
"Action" : "kms:*",
"Resource" : "*"
} ]
EXIT=0
$ aws kms list-grants --profile et-dev --region us-east-1 --key-id ce1dbe4f-711e-4dd3-ab6e-9ce1930e9040 --output json
{
"Grants": []
}
EXIT=0
$ aws iam list-role-policies --profile et-dev --region us-east-1 --role-name et-dev-backend-task --output json
"PolicyNames": [ "bedrock", "kms", "s3", "ses", "sqs", "sqs-send" ]
EXIT=0
$ aws iam list-attached-role-policies --profile et-dev --region us-east-1 --role-name et-dev-backend-task --output json
"AttachedPolicies": []
EXIT=0
$ for p in bedrock kms s3 ses sqs sqs-send; do
aws iam get-role-policy --profile et-dev --region us-east-1 --role-name et-dev-backend-task --policy-name $p --query PolicyDocument --output json
done
EXIT=0 for each of the 6 commands
| Inline policy (IAM name) | Statements and actions | A kms: action on the token key? |
|---|---|---|
kms |
DirectKeys: kms:GenerateDataKey, kms:Decrypt on key/ce1dbe4f-711e-4dd3-ab6e-9ce1930e9040; KeysThroughS3: the same two actions on key/676c9326-b585-48f1-88d3-dbd2fd9d4d43, only kms:ViaService s3.us-east-1.amazonaws.com |
DirectKeys only |
bedrock |
bedrock:InvokeModel on the Haiku 4.5 and Sonnet 5.5 models and profiles |
no |
s3 |
s3:ListBucket, s3:ListBucketVersions on et-dev-attachments; s3:PutObject, GetObject, DeleteObject, DeleteObjectVersion on its objects |
no |
ses |
ses:SendEmail, ses:SendRawEmail on dev.crm.empire.lk and et-dev-mail |
no |
sqs |
receive, delete, change visibility and read attributes on the four queues | no |
sqs-send |
sqs:SendMessage on three queues |
no |
The expected outcome:
-
The key and its only path. The mailbox token key is
ce1dbe4f-711e-4dd3-ab6e-9ce1930e9040(its alias isalias/et-dev-mailbox-tokenbyforge-et-iaacorigin/devbase/kms/keys.tfandlocals.tf). Its key policy names only the account root, which hands the decision to IAM; it has no grants; the task role has no attached policy; and of the role’s six inline policies onlykms’sDirectKeysstatement allows akms:action on it. RemovingDirectKeystherefore removes the task role’s only path to the key, and the check cannot pass for the wrong reason. -
The policy’s IAM name is
kms(task_kmsis its Terraform resource name,base/ecs/task_role.tfline 104–105, spec Rev 2). F38-f-WI-3.4’sget-role-policyandput-role-policymust namekms:put-role-policywith a name the role does not have would add a second policy and leaveDirectKeysin place. The document to put back is the one F38-f-WI-3.4 reads first; on this read it heldDirectKeysandKeysThroughS3as shown above. -
Which connections fail while the statement is removed. Every connection whose access token must be decrypted, not the test mailbox only: the grant covers the key for every connection, and the delta poll read on this day shows two
ACTIVEconnections (AC-8 (d):active=2). A token is decrypted when its cached copy is within two minutes of expiry (EXPIRY_SKEW, lines 115 and 470), or on a cache miss: the cache is in memory, per task (desired_count = 1,base/ecs/service.tf), so the first poll after a task starts decrypts every connection’s token whatever its expiry. A connection whose cached token is still live gets it from the cache (lines 226–228) and is unaffected. -
The call path and the lines, per failing connection, on the delta poll (every 15 minutes, at about 05, 20, 35 and 50 minutes past the hour).
MailboxDeltaPollJobHandler.pollDeclared(line 210) →MailboxIngestService.accessToken(line 190) →MailboxTokenService.accessTokenFor→obtainDeclared→refresh→TokenCipher.decrypt(line 301) →KmsTokenKeyProvider.unwrap→ KMSDecrypt(line 130), which is refused.unwraptranslates onlyInvalidCiphertextException(line 135) andTokenCiphercallsunwrapbefore its owntry(line 138), so the AWS SDK exception leavesMailboxTokenServiceuntranslated, before any Microsoft call (line 310) and before any write;accessTokenFor’s transaction rolls back with nothing to undo.pollAll’scatch (RuntimeException e)(line 140) logs:- WARN, logger
com.empireteas.mailbox.service.MailboxDeltaPollJobHandler:Mailbox Delta Poll Failed - [connectionId=<id>] [reason=<the exception's simple class name>]— most likelyKmsException, the AWS SDK v2 base class an unmodelled KMSAccessDeniedExceptionarrives as; the exact name is confirmed on the run. The throwable itself goes to DEBUG only (line 151), which dev does not log; - INFO, the same logger:
Mailbox Delta Poll Completed - [active=<n>, reached=<n>, succeeded=<n minus the connections that needed a decrypt>]; - ERROR, the same logger:
Mailbox Delta Poll All Failed - [reached=<n>, failed=<n>], only when every connection reached needed a decrypt.
The poll’s own SQS message completes normally (the catch is per connection), so it is not redelivered.
- WARN, logger
-
The connection status, the row and the audit trail while it is removed. Each connection stays
ACTIVE:reconnectRequiredis called only for missing scopes, an absent token or the provider’s permanent refusal, none of which this path reaches. No row changes, no refresh token is rotated, and no audit row is written (nomailbox.token_rotated, nomailbox.reconnect_required). The refresh token stored in the database is the one from before the window, unused by Microsoft, so it stays valid. -
The other paths that make the same call — each fails for a connection that needs a decrypt:
- The notification path:
MailboxIngestJobHandler(line 62) callsMailboxIngestService.accessToken, so a message arriving at a connected mailbox in the window fails its ingest job, which SQS redelivers up tomaxReceiveCount = 3and then moves toet-dev-mailbox-ingest-dlq(forge-et-iaacorigin/devbase/sqs/mailbox_ingest.tf). F38-f-WI-3.4 sends no mail to a connected mailbox during the window; a dead-lettered message that arrives anyway is a record it created (spec NFR-5). - The renewal job (
rate(6 hours), ticking at 00:35, 06:35, 12:35 and 18:35 UTC — AC-8 (d)) asks for a token only for a subscription due within its threshold (four cadences, 24 hours,MailboxProperties; every run in the 26 hours read haddue=0). When one is due and needs a decrypt: WARNMailbox Subscription Renewal Deferred - [connectionId=<id>] [reason=<simple class name>](loggerMailboxRenewalService), itsCompletedline counting itDEFERRED, and ERRORMailbox Subscription Renewal All Failedif every renewal failed. A window between those ticks avoids it. - User actions in the CRM (not in the reading of
.forge/plans/F38-f/F38-f-WI-1.1-plan.mdD-8; found on this day): sending email (EmailSendService, line 765) and reading a message’s attachments (MessageAttachmentService, line 134) callaccessTokenForand catch only the rate-limit exception, so either fails its request for such a connection; a mailbox back-fill (MailboxBackfillService, throughMailboxIngestService.accessToken) fails its job; a disconnect uses the best-effort call and logs WARNMailbox Access Token Unavailable. The window’s announcement asks the team not to send email, open attachments, or connect or disconnect a mailbox on dev until it ends.
- The notification path:
-
After the statement is put back. The next delta poll decrypts, refreshes with Microsoft and rotates each such connection’s refresh token, writing one
mailbox.token_rotatedaudit row per connection, and the poll’sCompletedline showssucceededequal toreachedagain. No user action is needed: the status staysACTIVE, neverRECONNECT_REQUIRED. -
Timing the window. A successful refresh writes no log line on dev: the token path logs at INFO or above only on failures and status changes (the
grepabove), and dev runs at INFO (application.yml: a cloud task runs with no profile, so the local profile’scom.empireteas: DEBUGdoes not apply;td.dev.jsonsets noLOGGING_LEVEL_*). Afilter-log-eventsread of"MailboxTokenService"over the same 26 hours found no line (below). So the last refresh cannot be read from the logs, as spec Q-2 assumed. What can be read: a task’s start (ECSstartedAt, or its stream’s first event), because the first poll after it decrypts every connection; and themailbox.token_rotatedaudit rows, which only the F38-f tunnel can read (F38-f-WI-4.1). The access token’s lifetime is not logged either. This is reported to the lead (.forge/owed.md, “FR-8 (f)'s refresh cannot be timed from the logs”), for F38-f-WI-3.4’s run sheet to choose its trigger.
2026-10-26 05:00 UTC — Claude (F38-f-WI-1.1), profile et-dev (svc-harness)
$ aws logs filter-log-events --profile et-dev --region us-east-1 --log-group-name /ecs/dev/backend --start-time 1792896689000 --filter-pattern '"MailboxTokenService"' --query 'events[].message' --output json
[]
EXIT=0
Result: reported to the lead — .forge/owed.md (item 8, the trigger for the window). Items 1–7 are the expected outcome F38-f-WI-3.4 runs against; they differ from the plan’s reading only by item 6’s user actions.
Recorded 2026-11-21 by F38-f-WI-2.4; the full record is .forge/plans/F38-f/F38-f-WI-2.4-record.md.
2026-11-21 21:40–22:05 UTC — Claude for Tharinda (F38-f-WI-2.4), profile et-dev (svc-harness); the removal, the forced deployment and the restore approved by Tharinda Rajapaksha at 20:16 UTC; the temporary temp-f38f-wi24-redeploy policy (21:40:08–21:51:19) run with his permission of 20:24 UTC
21:40:12 aws iam put-role-policy --role-name et-dev-backend-task --policy-name kms --policy-document file://kms-without-directkeys.json → Sids: KeysThroughS3
21:41:24 aws ecs update-service --cluster et-dev --service backend --force-new-deployment → task 384b6f575dd44f1396474f13e83b6a38, COMPLETED 21:45:33
21:50:22 WARN Mailbox Delta Poll Failed - [connectionId=20] [reason=KmsException]
21:50:22 WARN Mailbox Delta Poll Failed - [connectionId=16] [reason=KmsException]
21:50:22 INFO Mailbox Delta Poll Completed - [active=2, reached=2, succeeded=0]
21:50:22 ERROR Mailbox Delta Poll All Failed - [reached=2, failed=2]
Email › Mailbox as CRM01: Connected
21:51:09 aws iam put-role-policy … --policy-document file://kms-before.json → cmp kms-before.json kms-after.json: identical
22:05:25 INFO Mailbox Delta Poll Completed - [active=2, reached=2, succeeded=2]
Email › Mailbox as CRM01: Connected, no reconnect
EXIT=0
The ingest dead-letter queue held 2 messages before and after; no renewal was deferred; no dev.yaml or build run was created in the window.
Result: met — items 1–7 each match: every connection’s refresh failed with a WARN while the grant was gone, nothing was marked broken, and refresh worked again on the next poll with the document put back identical.
F38-f evidence — AC-8 (h) — alternating users
FR-8 (h): not applicable, because OQ-K chose no rotation and no alternating users (F38 Rev 14). The record says so, from the parent spec on main.
2026-10-26 05:02 UTC — Claude (F38-f-WI-1.1)
$ git -C forge-et-harness rev-parse --short origin/main
3eb371a0
$ git -C forge-et-harness show 3eb371a0:.forge/specs/F38/cloud-bring-up-spec.md
EXIT=0
$ grep -n 'not applicable — OQ-K chose no rotation' <the saved output>
399: - *(Rev 25: not applicable — OQ-K chose no rotation and no alternating users, Rev 14.)* If OQ-K chose alternating users: after a rotation, a deploy carrying a new migration that alters an existing table succeeds as the other user, and every table in `public` is still owned by the owner role (the `\dt+` output is recorded).
$ grep -n 'Rev 14 (built in dev on 2026-10-20' <the saved output>
75:**FR-10 [F38-b]** … Rev 14 (built in dev on 2026-10-20; each later environment gets it from the same module): the database password is not rotated on a schedule. …
$ grep -n '^### Rev 14' <the saved output>
675:### Rev 14 — 2026-10-16
676:- **Changed:** FR-10, AC-10, AC-40's rotation check and OQ-K (answered). The database password is not rotated on a schedule, in any environment. … No restart on rotation and no alternating users are built. …
Result: not applicable — no scheduled rotation and no alternating users exist on dev or are built for any environment (F38 FR-10, Rev 14; AC-40’s rotation bullet, Rev 25 note), so there is nothing to check.
F38-f evidence — AC-9 — the spec’s records
AC-9 (FR-9; parent AC-30): the spec’s PR carries the parent’s Rev 25, §4 item 15’s update, the FR-6 rows and the register’s ## Log line; the tracker’s SP-ARCH-004 row records 2026-10-13 as the day stage 2 ran, and the register’s §C-5 row is validated. Parent Rev 27 records the §E correction (spec Rev 1). Each file read on harness origin/main with git show 3eb371a0:<path> (EXIT=0 for each), then grep -n.
2026-10-26 05:02–05:03 UTC — Claude (F38-f-WI-1.1)
$ git -C forge-et-harness fetch -q origin
EXIT=0
$ git -C forge-et-harness rev-parse --short origin/main
3eb371a0
$ yq '.setup.architecture.spikes[] | select(.id == "SP-ARCH-004") | .stage_2' <3eb371a0:.forge/tracker.yaml>
RAN 2026-10-13 in the dev account. AgentCore Web Search (through et-dev-gateway, signed as svc-harness) searched the same 25 customers in the same four query shapes: 82 searches, no errors. …
EXIT=0
$ gh pr view 908 --repo altrium-io/forge-et-harness --json state,mergedAt,mergeCommit
{"mergeCommit":{"oid":"b20b288b64f9c84e873ebb868e2212300c9cbbac"},"mergedAt":"2026-10-25T09:47:45Z","state":"MERGED"}
EXIT=0
$ gh pr view 919 --repo altrium-io/forge-et-harness --json state,mergedAt,mergeCommit
{"mergeCommit":{"oid":"31702688154ad2bed399cf5926ae17530ac04e03"},"mergedAt":"2026-10-25T20:07:46Z","state":"MERGED"}
EXIT=0
| Item | File on origin/main 3eb371a0 |
Line found |
|---|---|---|
| The parent’s Rev 25 | .forge/specs/F38/cloud-bring-up-spec.md |
772: ### Rev 25 — 2026-10-25 |
| The parent’s Rev 27 (the §E correction) | .forge/specs/F38/cloud-bring-up-spec.md |
790: ### Rev 27 — 2026-10-26 |
| FR-30 met, AC-30’s note | .forge/specs/F38/cloud-bring-up-spec.md |
390: - [ ] **AC-30 (FR-30)** The spike row records the date stage 2 was unblocked. *(Rev 25: stage 2 ran on 2026-10-13 — met, the evidence is the tracker's SP-ARCH-004 row and the register's §C-5 row.)* |
| §4 item 15’s update | .forge/design/environments-and-identity.md |
283: item 15, “*Update 2026-10-25: the need has come — F38-f builds it (F38-f spec FR-7, decided by the lead the same day; F38 Rev 25). …” |
| The F38-f block | .forge/cloud-gated-validation.md |
1447: ### F38-f — Proving the platform on dev (spec approved 2026-10-25) |
| FR-6 row: the Sonnet 5.5 half | .forge/cloud-gated-validation.md |
1456: owner Tharinda (quota: Shyamin Ayesh), open |
| FR-6 row: the capture app’s AC-40 check | .forge/cloud-gated-validation.md |
1457: owner F38-f, open |
| FR-6 row: AC-40’s database checks | .forge/cloud-gated-validation.md |
1458: owner F38-f, open |
The register’s ## Log line for the spec |
.forge/cloud-gated-validation.md |
2337: - 2026-10-25 — F38-f spec approved (Claude, for Tharinda; no AWS call): §B gains the F38-f block, three rows — … |
| §C-5’s SP-ARCH-004 stage 2 row | .forge/cloud-gated-validation.md |
1549: ends | Dinil | validated |, with “Validated 2026-10-13: the gateway (et-dev-gateway, built from forge-et-iaac) was called as svc-harness …” |
| The tracker’s SP-ARCH-004 row | .forge/tracker.yaml |
stage_2 begins RAN 2026-10-13 |
| The spec PR and the records PR | GitHub | #908 MERGED 2026-10-25 09:47 UTC; #919 MERGED 2026-10-25 20:07 UTC |
Result: met — every record AC-9 names is on main; AC-9 is ticked in the spec.
F42 — Bedrock call pacing under the account’s request quota
The five items F42 spec FR-H4 names (AC-14), numbered as FR-H4 numbers them. Item 5 is wave-1 behaviour — a Bedrock throttle ends RATE_LIMITED and never opens the circuit breaker (FR-B1, FR-B2) — and was added with wave 1 by F42-WI-1.3; items 1–4 depend on wave 2’s pacer and are added with it by F42-WI-2.4. Every F42 AC runs on a fake clock, a fake Bedrock client (FakeBedrockRuntimeClient) or the AIProvider stub; these are what only real Bedrock and a deployed environment can prove. Each item makes real Bedrock calls, so each run needs the lead’s OK (.claude/CLAUDE.md § Boundaries › Ask first). What the backend expects of the cloud is forge-et-be/docs/ai/bedrock-cloud-contract.md.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
1. A burst on dev stays under the quota — with dev’s one backend task at its shipped rows (10 Haiku 4.5 requests a minute, 8 of them for background work), a burst of lead triage is paced: no call ends PROVIDER_ERROR or opens the circuit breaker, Bedrock records no more than 10 invocations and no throttle in any minute, no triage message is redelivered while its call waits for a slot, and every lead is triaged by the run or by a later hourly sweep |
Only real Bedrock enforces the account’s quota, and only the deployed triage queue carries Terraform’s settings; every test runs on a fake clock and a fake Bedrock client, and the listener’s receive visibility is proved against the local queue only | F42 spec FR-H4 item 1, FR-P1 – FR-P4, FR-W1, FR-W2, NFR-1, AC-1, AC-2, AC-15; F42-WI-2.1, F42-WI-2.2 and F42-WI-2.3 plans; forge-et-be ai/service/AiPacer.java, ai/service/AiCallRunner.java, lead/service/LeadTriageJobListener.java |
Once both F42 waves are on dev, with the lead’s OK for the Bedrock calls and no live evaluation running (spec W-4): create at least 30 leads on dev within a minute. Each creation queues its triage (F11-d FR-J1) and its research (F11-c), which spends one Exa search of dev’s credit per lead. Then read, for the run and the next hourly sweep: the ai_invocation rows for lead-triage and lead-research — none PROVIDER_ERROR, and any RATE_LIMITED with attempts = 0 (handed back to the sweep) followed by a later row for the same lead; the backend’s log — no AI Circuit Opened line; CloudWatch AWS/Bedrock Invocations and InvocationThrottles for the Haiku 4.5 profile at a one-minute period (aws cloudwatch get-metric-statistics, a read; record which dimension was read) — at most 10 and 0 in every minute; SQS NumberOfMessagesReceived against NumberOfMessagesSent for the dev triage queue over the run (a redelivered message is received twice) — equal; and the leads — each shows its triage result. Record the counts, the window and the metric reads here |
Tharinda (F42) | validated 2026-10-29 on dev (backend:85, one task, shipped rows) — 30 leads created at 04:51:44 UTC, all 201. Read from the backend’s AI Call Completed log lines (the AiInvocationRecorder line written with each ai_invocation row): the run gave 17 lead-triage SUCCESS (attempts=1) and 13 RATE_LIMITED (attempts=0, INFO, handed back), and 30 lead-research SUCCESS by 04:57; the 05:35 hourly sweep triaged 8 of the 13 and handed 5 back again (attempts=0) after the background wait, and the 06:35 sweep triaged those 5, so every lead was triaged. No PROVIDER_ERROR, no AI Circuit Opened line. CloudWatch AWS/Bedrock, dimension ModelId=us.anthropic.claude-haiku-4-5-20251001-v1:0, period 60: Invocations 8, 8, 9, 4, 8, 8, 5 a minute from 04:51 to 04:57 (50 = 17 + 30 + 3 email drafts), 8 in the 05:35 minute and 5 in the 06:35 minute; InvocationThrottles no datapoints from 04:45 to 08:00. SQS from 04:45 to 05:10: et-dev-lead-triage and et-dev-lead-research each sent 30, received 30 and deleted 30, so no message was redelivered |
2. An interactive call during the burst — an email draft asked for while item 1’s burst runs answers inside the 9 s deadline, or ends RATE_LIMITED with no attempt and shows the manual path; background work never takes the slots kept for it |
The reserve and the order in which a freed slot is given are proved on a fake clock; only a deployed backend under a real burst shows a person’s call getting a slot | F42 spec FR-H4 item 2, FR-P4, FR-W1, FR-O3, NFR-2, AC-1, AC-3, AC-5; F11-h | During item 1’s run, under the same OK: sign in as CRM01 on https://dev.crm.empire.lk, open one of the burst’s leads and ask for a follow-up draft (F11-h). Record whether the draft or the manual-path message appeared, and the ai_invocation row for email-draft in that window — SUCCESS with latency_ms under 9,000, or RATE_LIMITED with attempts = 0 |
Tharinda (F42) | validated 2026-10-29 on dev, during item 1’s run — the lead asked for drafts as CRM01 and the draft appeared; the log shows 3 email-draft SUCCESS rows with latency 1,260, 1,385 and 1,366 ms, and the triage calls that found no slot were handed back rather than taking the reserved ones |
| 3. The effective quota — whether dev’s Bedrock throttles below 10 Haiku 4.5 requests a minute, or within a minute at a burst the sliding window allows (10 starts in one second) | AWS says a new account’s request quotas can be lower than Service Quotas shows (spec W-2), and whether the quota is enforced over intervals shorter than a minute is not documented (spec Q-1); only real Bedrock answers either | F42 spec FR-H4 item 3, decision 6, W-2, Q-1; environments-and-identity.md §4 AC-4 (L-CCA5DF70, 10 a minute) |
Read from item 1’s run, with no Bedrock call of its own: an InvocationThrottles value above 0, or an AI Call Rate Limited line at WARN (attempts ≥ 1), means the effective quota is below the row or is enforced over shorter intervals. Then either lower dev’s rows (a task-definition override restating every ET_AI_PACING_LIMITS_<n>_* row, §D) and repeat item 1, or add a minimum spacing between starts (60 s ÷ the limit) by an F42 spec revision (Q-1). Record the reads and the choice here |
Tharinda (F42) | validated 2026-10-29 on dev, from item 1’s run — InvocationThrottles had no datapoints and no AI Call Rate Limited line was logged at WARN, at up to 9 calls a minute including the run’s opening seconds, so no change to dev’s rows and no spec revision. Not shown: whether Bedrock throttles at exactly 10 in a minute, because the pacer never sent 10 |
4. The per-task values in each task definition — each environment’s running task definition gives each et.ai.pacing.limits row the account’s quota for that model divided by the backend tasks drawing on it, rounded down, under the variable names §D gives, with two exceptions: the Sonnet 5.5 row is a placeholder of at least 1, not quota ÷ tasks, while no feature uses that model (its quota is 0 in dev and expected to be 0 in production, F38-i W-6), and a feature is never moved to a model whose quota is 0; and where a model’s share is at or below the background reserve (2 by default), the task definition also sets ET_AI_PACING_BACKGROUNDRESERVE below every row, or startup refuses |
The values depend on each environment’s task count and each account’s quota, and production’s quota and task count are not known until its account exists (spec W-3); the tests prove only that the names bind (AC-19) | F42 spec FR-H4 item 4, decision 2, FR-P3, AC-19, W-3; §D (the ET_AI_PACING_* rows); F42-WI-2.1 plan C-4; F38-h § Constraints and Dependencies (“Shared account limits”); F38-i |
Dev: read the running task definition (aws ecs describe-task-definition) and the service’s desired count (aws ecs describe-services), both reads: one task and no ET_AI_PACING_* variable, so the shipped Haiku 4.5 row of 10 is dev’s share with one task, and the shipped Sonnet 5.5 row of 10 is a placeholder (no feature uses Sonnet 5.5, whose quota is 0); read again whenever dev’s task count or quota changes. UAT: a UAT instance runs at the shipped rows; it shares the development account’s quota with dev (AD #4), so during a round the two tasks together can ask for twice the quota, and Bedrock’s throttles end RATE_LIMITED; F38-h asks the team to keep dev light during the round; read UAT’s running task definition. Production: F38-i’s task definition sets the rows from production’s quotas (read by its FR-2) and its task count, with the two exceptions above; read the running one after the first production deploy. Record each environment’s values, the quota and the task count they come from here Update 2026-11-04 (F43-WI-4.2): since F43 application.yml carries six rows at 10 — Haiku 4.5’s us. and Global profiles, dev’s share with one task of each profile’s quota, and four placeholders for the 5.5 models — and lead triage and lead research use the Global Haiku 4.5 row where §D sets it (dev since F43’s wave 4); read dev’s again after that deploy (§B F43 item 5). A task definition that restates the rows restates all six (§D). |
Tharinda (F42); production with whoever takes F38-i | open — dev read 2026-10-29: backend:85, desired count 1, no ET_AI_PACING_* variable, so the shipped rows apply (Haiku 4.5 10 a minute, dev’s share with one task; Sonnet 5.5 a placeholder of 10, no feature using it); UAT and production not yet read |
5. A deploy’s overlap — a burst of AI calls while a deploy replaces dev’s backend task (two tasks, each pacing to the whole per-task share, for about a minute and a half) is throttled by Bedrock to RATE_LIMITED, never PROVIDER_ERROR or CIRCUIT_OPEN, and every lead it hands back is triaged by the run or a later sweep |
Two backend tasks sharing one account quota exist only while ECS replaces a task; every test runs one JVM against a fake Bedrock client that throttles only when scripted to | F42 spec FR-H4 item 5, decisions 2 and 5, FR-B1, FR-B2, FR-O2, AC-7, AC-8; F42-WI-1.1 plan Decisions 4–7; environments-and-identity.md §4 (base/ecs: one task, two while a deploy replaces it); forge-et-be ai/service/AiCallRunner.java, ai/service/AiCircuitBreaker.java |
Once both F42 waves are on dev, and with the lead’s OK for the Bedrock calls: while a merge to main deploys the backend (the build workflow), queue at least 30 leads for triage on dev about a minute after the run starts, before dev’s backend service shows two running tasks (aws ecs describe-services, a read). Starting when the second task appears is too late: on 2026-10-29 the overlap lasted about 80 s (18:19:54 – 18:21:12 UTC, backend:86 to backend:87), the second task appeared about 2.5 minutes after the run started, and the 30 lead-research calls of a burst take about 6 minutes to finish, so a burst started a minute in still has calls waiting when both tasks run. Then read, for the deploy’s window: the ai_invocation rows — any throttled call ends RATE_LIMITED with attempts ≥ 1, and none ends PROVIDER_ERROR or CIRCUIT_OPEN; both tasks’ logs — one AI Call Rate Limited line at WARN per such row, and no AI Circuit Opened line; and the leads — each is triaged by the run or by a later hourly sweep (a lead at et.lead-triage.max-attempts waits for the daily retry, F11-d FR-J7). Record the counts, the deploy’s run link and the window here |
Tharinda (F42) | validated 2026-11-02 on dev — a manual dev deploy of main (build run 41394125023, started 05:27:22 UTC, backend:88 to backend:89) ran two backend tasks from 05:29:36 to about 05:30:58 UTC; 30 leads were created at 05:28:50, all 201. Bedrock Invocations for the Haiku 4.5 us. profile (period 60): 8, 8, 10, 12, 6, 2 a minute from 05:28 to 05:33, so the 05:31 minute went over one task’s share, and InvocationThrottles 4 in that minute, none in any other. Those 4 throttled attempts ended as 2 lead-research rows RATE_LIMITED with attempts 1 and 3, one on each task, each with one AI Call Rate Limited line at WARN; no PROVIDER_ERROR, no CIRCUIT_OPEN and no AI Circuit Opened line from 05:28 on. Triage: 18 at once across both tasks, 12 handed back (attempts=0), then 8 by the 05:35 sweep and the last 4 by the 06:35 sweep, so all 30 were triaged; research: 28 at once, and the 2 throttled leads by the 06:08 research sweep. SQS from 05:25 to 06:45: et-dev-lead-triage sent, received and deleted 46 each (30 + 12 + 4), et-dev-lead-research 32 each (30 + 2), so no message was redelivered. The 29 Oct attempt during forge-et-be#336’s deploy was missed, because the burst was to start when two tasks showed and the overlap ended first |
F43 — AI load under the Bedrock quota: backlogs that drain, and a busy answer for people
The five items F43 spec FR-H3 names (AC-16), numbered as FR-H3 numbers them. Each is added by the F43 wave that defers it — item 1 with wave 1 (F43-WI-1.2), item 3 with wave 2 (F43-WI-2.4), item 4 with wave 3 (F43-WI-3.1), and items 2 and 5 with wave 4 (F43-WI-4.2) — so the table lacks the items of any F43 wave not yet merged. Every F43 AC runs on the AI stub, a fake Bedrock client and a fake clock (AC-18); these are what only real Bedrock, the account’s quotas and a deployed environment can prove. All five are proved together by one planned run after the last F43 wave is on dev (F43-WI-5.1; spec § Constraints and Dependencies, “The cost of testing it”). The lead approved that one run on 2026-11-02 — the burst’s Bedrock calls, the temporary pacing override and its removal, and a manual dev deploy if one is needed; a second run, or anything beyond that list, needs the lead’s OK again (.claude/CLAUDE.md § Boundaries › Ask first). The run’s Bedrock cost, from its costUsd lines, is recorded in this block (AC-18). What the backend expects of the cloud is forge-et-be/docs/ai/bedrock-cloud-contract.md.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
1. A burst drains in minutes — about 30 leads queued for triage on dev, with lead triage and lead research on Haiku 4.5’s Global profile, are all triaged within about 10 minutes of the first: no lead-triage call ends RATE_LIMITED with attempts = 0 (a first attempt refused for want of a slot), and no triage message is received more than once. A lead whose call failed after an attempt (spec decision 2 — a throttle, or a retry that found no slot) is left to the hourly sweep and recorded here, and does not fail the run |
Only real Bedrock enforces the account’s quota, and only the deployed triage queue carries Terraform’s settings; the 60 s wait and a burst draining at the background share are proved on a fake clock (AC-4, AC-5), and the listener’s own receive visibility against the local queue only (AC-2) | F43 spec FR-H3 item 1, FR-D1 – FR-D4, decisions 1 and 2, AC-1, AC-2, AC-4, AC-5; F43-WI-1.1 plan; §B F42 item 1 (the 29 Oct burst, 13 of 30 handed back); forge-et-be lead/service/LeadTriageJobListener.java, lead/service/LeadTriageReceiveVisibility.java, ai/service/AiPacer.java |
In F43-WI-5.1’s run, once F43’s waves 1, 2 and 4 are on dev (wave 4 puts triage and research on the Global profile, after wave 3’s grants), under the lead’s approval of 2026-11-02 for that run: create about 30 leads on dev within a minute. Each creation queues its triage (F11-d FR-J1) and its research (F11-c), which spends one Exa search of dev’s credit per lead. Then read, for the 15 minutes from the first creation: the backend’s AI Call Completed and AI Call Rate Limited log lines for lead-triage and lead-research — each lead’s triage SUCCESS or a failure after an attempt (recorded), none RATE_LIMITED with attempts=0, the time of the last triage, and the model they name (the Global Haiku 4.5 profile); SQS NumberOfMessagesReceived against NumberOfMessagesSent for et-dev-lead-triage over the run (a redelivered message is received twice) — equal; and the costUsd of the run’s AI Call Completed lines. Record the counts, the window, the reads and the cost here. Update 2026-11-20 (F43-WI-5.1, Dinil): 30 leads created at 08:38:53 UTC, all 201, all on backend:169 (main); all 30 Lead Triage Completed SUCCESS between 08:38:57 and 08:45:59 UTC, each lead-triage call on the Global profile with attempts=1, no RATE_LIMITED line at all, no sweep and no AI Circuit Opened in the 15 minutes; et-dev-lead-triage sent, received and deleted 30 each (no redelivery), et-dev-lead-research the same; all 30 researched; record: .forge/plans/F43/F43-WI-5.1-record.md. |
Dinil (F43) | validated 2026-11-20 on dev — 30 leads triaged within 7 minutes, none refused for a slot, none received twice |
2. The two quotas are separate — on dev, during item 1’s burst, with lead triage and lead research on Haiku 4.5’s Global profile (global.anthropic.claude-haiku-4-5-20251001-v1:0) and the calls a person waits for on its us. profile, a person makes at least 3 AI calls on the us. profile within one minute, and in that minute Bedrock’s CloudWatch metrics show calls on the two profiles together above 10 with no throttle on either |
Service Quotas lists Haiku 4.5’s two request quotas separately — L-CCA5DF70 for the us. profile and L-E5084BBA for the Global profile, 10 a minute each (environments-and-identity.md §4) — but that Bedrock enforces them separately is seen only under load; locally the pacer’s one window per inference profile is proved on a fake clock (F42), and every test runs on the AI stub or a fake Bedrock client |
F43 spec FR-H3 item 2, point 4, FR-R1, FR-R3, decision 4, AC-12, AC-14; F43-WI-3.1 and F43-WI-4.1 plans; register §B F42 items 1 and 3 (the same metric reads) | In F43-WI-5.1’s run, once F43’s waves 1, 2 and 4 are on dev and item 4’s grant is applied, under the lead’s approval of 2026-11-02 for that run: while item 1’s burst runs, sign in as a test user and make at least 3 AI calls within one minute in one place on the us. profile (for example Draft follow-up and Draft again on one lead). Read aws cloudwatch get-metric-statistics --namespace AWS/Bedrock for Invocations and InvocationThrottles, period 60, once with the dimension ModelId=us.anthropic.claude-haiku-4-5-20251001-v1:0 and once with ModelId=global.anthropic.claude-haiku-4-5-20251001-v1:0 (reads; record the dimension read, as §B F42 item 1 does), and the backend’s AI Call Completed lines (lead-triage and lead-research naming the Global profile, the person’s calls the us. profile). It passes when, in one minute, Invocations on the two profiles sum to more than 10 and InvocationThrottles is 0 on both. A throttle on either at that load means the two quotas are not separate: record it here, leave the row open, and the lead decides the fallback the spec names (point 4: the reserve sized against the busiest real case instead). Record the minute, the per-profile counts and the dimension read here. Update 2026-11-20 (F43-WI-5.1, Dinil): four email drafts on burst lead 116 at 08:40:00–08:40:10 UTC, all SUCCESS on us.anthropic.claude-haiku-4-5-20251001-v1:0; read by ModelId (each profile has its own series, ModelId=us.… and ModelId=global.anthropic.claude-haiku-4-5-20251001-v1:0): in the 08:40 minute Invocations were 4 on us. and 7 on Global, 11 together, with no InvocationThrottles datapoint on either in any minute of the window (the Global profile took 3–9 a minute throughout); record: .forge/plans/F43/F43-WI-5.1-record.md. |
Dinil (F43) | validated 2026-11-20 on dev — 11 calls in one minute across the two profiles, no throttle |
3. The busy answer on screen — on dev, with dev’s pacing rows restated in its task definition and the us. Haiku 4.5 row at 3 (above the background-reserve of 2, or startup refuses) for the run, a person’s fourth AI call on the us. profile within one minute answers 429 AI_BUSY, and the place shows “The AI is busy right now. Try again in N seconds.” with N the pacer’s wait to its next free slot (1–60), the button enabled and the place’s manual path still offered (in the inquiry-extraction dialog, Continue without AI), in at least one of the four AI places — the inquiry-extraction dialog, the lead’s follow-up draft panel, the compose dialog’s AI draft button, the account briefing panel. The override is then removed, so dev runs on main’s pacing rows again (two deploys) |
Only the real pacer, in a deployed backend calling Bedrock, computes the wait from the starts in its window; the stub takes no slot and gives fixed waits of 5 s and 10 s (FR-A1). Locally the wait is proved on a fake clock (AC-7), the 429 against the stub’s triggers (AC-8), and the sentence, the button and the manual path in component and browser tests with the stub’s waits (AC-9, AC-10) |
F43 spec FR-H3 item 3, FR-A1 – FR-A4, decisions 5 – 7, AC-7 – AC-10, § Constraints and Dependencies “The cost of testing it”; .forge/plans/F43/F43-Wave-2-contract.md; F43-WI-2.1, F43-WI-2.2 and F43-WI-2.3 plans; Decomposition Plan § Notes “How the dev run overrides dev’s pacing rows”; F42 FR-O4 (the AI Call Rate Limited line); forge-et-be ai/service/AiPacer.java |
In F43-WI-5.1’s run, once F43’s waves 1, 2 and 4 are on dev, under the lead’s approval of 2026-11-02 for that run (the temporary pacing override, its removal and the manual dev deploys): tell the team, and confirm that no forge-et-be PR is about to merge, since a push to main redeploys dev and drops the override. Deploy a throwaway forge-et-be branch, cut from main, whose .aws/td.dev.json restates dev’s pacing rows with the us. Haiku 4.5 row at 3, by a manual run of forge-et-be’s build workflow (AD #24, F38 Rev 18), and read that the backend service runs that task-definition revision (aws ecs describe-services, a read). Signed in as a test user, make four AI calls on the us. profile within one minute in one place (for example Draft follow-up, then Draft again three times, on one lead); the first three succeed and the fourth answers busy. Read: the place, the sentence and its N, the button enabled, the manual path offered; the backend’s AI Call Rate Limited line for the fourth call (attempts=0, the us. Haiku 4.5 model, a waitedMs of about 3000). Then a manual build run of main redeploys dev without the override; read that the service runs main’s revision, and delete the throwaway branch. Record the place, N, the log line, both workflow runs and both task-definition revisions here. Update 2026-11-20 (F43-WI-5.1, Dinil): the throwaway branch chore/F43-busy-answer-check (head 3313e922, twelve lines added to .aws/td.dev.json, never in a PR) deployed by build run 44706650441 as backend:170 (live 09:03 UTC); on lead 123’s follow-up draft panel, four presses at 09:04:00–09:04:07 UTC: three SUCCESS, then “The AI is busy right now. Try again in 51 seconds.” under the kept draft, Draft again enabled and Approve & send still offered; the backend logged AI Call Rate Limited for email-draft, us. Haiku 4.5, attempts=0, waitedMs=3000. main deployed again by build run 44707873719 as backend:171 (live 09:10:54 UTC, no pacing key); the branch deleted at 09:06 UTC; backend:170 left registered (WI-5.1 D-6); record: .forge/plans/F43/F43-WI-5.1-record.md. |
Dinil (F43) | validated 2026-11-20 on dev — the busy sentence shown with its wait, the button and the manual path kept, dev back on main |
4. The grants — the backend’s task role may call all six inference profiles of the three planned models (us. and global. Haiku 4.5, Sonnet 5.5 and Haiku 5.5) and the foundation models they route to, in every environment: on dev, et-dev-backend-task’s bedrock policy allows bedrock:InvokeModel on all six; UAT’s role et-uat-backend-task the same, from overlays/uat/locals.tf; production’s when F38-i writes its overlay |
Only IAM in the account decides what the role may invoke, and only dev.yaml’s apply changes it; locally the backend calls the AI stub, which needs no grant, and terraform validate proves only that the list is well formed |
F43 spec FR-R2, FR-H3 item 4, AC-13; F43-WI-3.1 plan; forge-et-iaac overlays/dev/locals.tf (forge-et-iaac#16), overlays/uat/locals.tf (forge-et-iaac#11, 6596205), base/ecs/task_role.tf (task_bedrock); .forge/owed.md “Allow the backend’s task role to call Haiku 5.5” |
Dev: after forge-et-iaac#16 merges into dev, read its dev.yaml run — the plan 0 to add, 1 to change, 0 to destroy, module.ecs.aws_iam_role_policy.task_bedrock only, and Apply complete! Resources: 0 added, 1 changed, 0 destroyed — then aws iam get-role-policy --role-name et-dev-backend-task --policy-name bedrock as svc-harness (a read): the six inference-profile ARNs and the foundation-model ARNs of overlays/dev/locals.tf; record the run id and the read here. F43’s wave 4 repeats the read before its forge-et-be PR merges (F43 Decomposition Plan § Notes), and F43-WI-5.1’s run relies on it. UAT: when F38-h applies the uat instance, the same read of et-uat-backend-task. Production: when F38-i’s overlay names all six (.forge/owed.md), the same read of production’s task role. Update 2026-11-04 — dev’s part done: |
- forge-et-iaac#16 was approved by Shyamin Ayesh and merged at
b8489c6(17:58 UTC). - Its
dev.yamlrun 41882320196 planned and applied the role policy only: “Apply complete! Resources: 0 added, 1 changed, 0 destroyed”. aws iam get-role-policy --role-name et-dev-backend-task --policy-name bedrock --profile et-devthen listedbedrock:InvokeModelon the six profiles —us.andglobal.of Haiku 4.5, Sonnet 5.5 and Haiku 5.5 — andarn:aws:bedrock:*::foundation-model/<id>for each of the three models, nothing else.
Update 2026-11-20 (F43-WI-5.1, Dinil): the same read before the run returned the same single statement, the six inference-profile ARNs and the three foundation-model ARNs, nothing else; record: .forge/plans/F43/F43-WI-5.1-record.md.
UAT’s and production’s parts are still open. | Dinil (F43) | open — dev validated 2026-11-20 (F43-WI-5.1); UAT and production open |
| 5. The values in each task definition — each environment’s running task definition carries the background route register §D gives it. On dev: ET_AI_FEATURES_LEADTRIAGE_MODEL and ET_AI_FEATURES_LEADRESEARCH_MODEL set to global.anthropic.claude-haiku-4-5-20251001-v1:0, ET_AI_FEATURES_EMAILDRAFT_MODEL and ET_AI_FEATURES_ACCOUNTBRIEFING_MODEL left on the us. profile, and no ET_AI_PACING_* or ET_AI_PRICES_* key, so application.yml’s six rows apply. On UAT and production: the values §D gives, the lower ET_LEADTRIAGE_MAXCONCURRENT and ET_LEADRESEARCH_MAXCONCURRENT included where a profile’s per-task share is not above the waiting background callers | Only the registered task definition shows what a deployed task runs with; the tests prove that the names bind (AiPropertiesBindingSpec, F11-d AC-9, F11-c AC-34) and that a cloud-shape boot accepts a feature on each of the six profiles (F43 AC-12) | F43 spec FR-H3 item 5, FR-R1, FR-R3, AC-14; F43-WI-4.1 plan (D-5); register §D; forge-et-be .aws/td.dev.json | Dev: after the forge-et-be wave 4 PR merges and its build run deploys dev, read the service’s task-definition revision (aws ecs describe-services --cluster et-dev --services backend, then aws ecs describe-task-definition, both reads): record its four ET_AI_FEATURES_*_MODEL values and that it sets no ET_AI_PACING_* or ET_AI_PRICES_* key; then read one AI Call Completed line each for lead-triage and lead-research naming the Global profile (a CloudWatch Logs read). F43-WI-5.1’s run reads them again before its burst. UAT: when F38-h writes UAT’s task definition from §D and applies the uat instance, the same reads. Production: when F38-i writes production’s task definition from its Bedrock scan (.forge/owed.md, “Scan production’s Bedrock setup before its task definition is written …”) and §D, the same reads after its first deploy. Record each environment’s values and revision here. Update 2026-11-20 (F43-WI-5.1, Dinil): dev’s backend:169 before the run and backend:171 after it each carry five ET_AI_FEATURES_*_MODEL values — LEADTRIAGE, LEADRESEARCH and LEADRANKING (added by F11-g) on global.anthropic.claude-haiku-4-5-20251001-v1:0, EMAILDRAFT and ACCOUNTBRIEFING on the us. profile — and no ET_AI_PACING_* or ET_AI_PRICES_* key; the run’s lead-triage and lead-research AI Call Completed lines name the Global profile; record: .forge/plans/F43/F43-WI-5.1-record.md. | Dinil (F43); UAT with F38-h, production with whoever takes F38-i | open — dev validated 2026-11-20 (F43-WI-5.1); UAT and production open |
The dev run’s cost (AC-18). F43-WI-5.1’s run on 2026-11-20 (08:22:30 – 09:11:10 UTC) made 98 AI calls, costing USD 0.42 (0.420033) from the costUsd of their AI Call Completed lines: lead-triage Global Haiku 4.5 SUCCESS 30 calls USD 0.077; lead-research Global SUCCESS 30 calls USD 0.218 (one on a second attempt); lead-ranking Global SUCCESS 28 calls USD 0.090 and INVALID_OUTPUT after two attempts 2 calls USD 0.011; email-draft us. Haiku 4.5 SUCCESS 7 calls USD 0.023 and RATE_LIMITED 1 call USD 0 — and 30 Exa searches from the free credit. Record: .forge/plans/F43/F43-WI-5.1-record.md.
Backlog features whose stub spec already names a cloud dependency
Not yet rows — the feature’s spec adds them when it is written. Listed so the walk-list shows
what is coming and so /forge-spec-review can check the FR is present.
| Feature | Cloud dependency the stub already names | Source | Owner |
|---|---|---|---|
| F11-c Lead research job | Replaced 2026-10-21 by the ### F11-c — Lead research job block above (spec FR-H3, written by F11-c-WI-1.3): its eight items are the Exa key, the lead-research queue and DLQ, the lead-research-sweep schedule, a live run, the evaluation on dev, dev’s model override, Sonnet 5.5 on dev, and Empire’s own Exa account. |
F11-c-lead-research-job-spec.md FR-H3 |
Dinil |
info@; the .Shared scopes are unconfirmed (F20-a W-4)info@ reaches the CRM by forwarding to a rep’s own mailbox, so there is no shared-mailbox consent or .Shared scope to validate (environments-and-identity.md §2.4) |
F20-e-shared-inquiries-inbox-spec.md (its Context records the drop); F20-a spec W-4 |
— | |
| F21-b Calendar sync | Live Graph calendar subscription + renewal on FS-9 (schedule rows as F20-b’s). No new reply URL while it adds its scope to the microsoft-mailbox registration; a separate registration would owe one per environment and a new Empire IT ask |
F21-b-calendar-sync-spec.md; environments-and-identity.md §2.2 |
— |
| F22 Contact sync | Replaced 2026-11-21 by the ### F22 — Contact sync block above (spec FR-17, written by F22-WI-2.1): its items are the real Graph contacts read, the hourly contact-sync-sweep schedule, the queue Check now uses (none added), a large first read’s run time, four Graph behaviours F22-WI-1.1 proved only against a mock server, and four run behaviours F22-WI-1.2 proved only locally. |
F22-contact-sync-spec.md FR-17 |
Tharinda |
F33-b File attachments on deals, quotes, samples and briefs (split from F33, 2026-11-06; F33-a’s half moved to the ### F33-a block above on 2026-11-06) |
New owner types (deals, quotes, samples, product briefs) on F33-a’s attachment store and the attachments bucket (§B F33-a; A-1); since 2026-10-09 with scanning switched off, so uploads are NOT_SCANNED rather than scanned (§B FS-11 block note). Each new owner type’s visibility joins A-1’s boot check, as F33-a’s contact did (§B F33-a row (d)) |
F33-b-deal-attachments-spec.md |
— |
| F36-a Capture and upload | Replaced 2026-11-13 by the ### F36-a — Capture and upload (card photo, badge QR) block above (spec FR-CG1, written by F36-a-WI-1.4): its items are the capture redirect address per environment, the sign-in round trip from the capture app on dev, the capture origins in the UAT and production CORS lists, the card photo on dev’s S3, the camera and badge reader on real phones, blob: in the capture app’s Content-Security-Policy, and the install prompt, standalone mode and safe areas on real phones. |
F36-a-capture-and-upload-spec.md FR-CG1 |
Nanndish |
| F36-b AI card reading | Replaced 2026-11-13 by the ### F36-b — AI card reading block above (spec FR-CG1, written by F36-b-WI-1.4): its eight items are the card-read queue and its key per environment, a live vision call and its cost, the evaluation run on dev, the call’s time against the 9-second deadline, the startup check of card reading’s visibility, the end-to-end case from a real phone, the time from upload to read, and a burst of 20 cards. |
F36-b-ai-card-reading-spec.md FR-CG1 |
Nanndish |
| F36-c Capture research + notify | Replaced 2026-11-16 by the ### F36-c — Capture research, scoring and in-app notification block above (spec FR-CG1, written by F36-c-WI-1.6): its items are the two research queues and their keys, the capture-research-sweep schedule, the research models’ overrides, the visibility check in the cloud shape, a live capture run, a 100-exhibitor import, the live evaluations, the notifications on two phones and a conversion’s copy on dev, and from spec Rev 6 a focus edit on dev and the v2 evaluation sets. Its SES row was withdrawn on 2026-11-12 by AD #42: the capture app sends no email. |
F36-c-capture-research-notify-spec.md FR-CG1 |
Dinil |
| — | F36-f-same-day-follow-up-email-spec.md |
— | |
| F15-a, F18-a (F18-b merged into it, 2026-10-11), F25-b | Each names an FS-9 scheduled job without naming EventBridge; each will probably owe a schedule row (verify at spec time) | their stub specs | — |
| F20-f Mail auto-link | Links and stores mail that Graph notifies, attachments in S3; the offline path only sees stubbed messages. Phase 1 | F20-f-mail-auto-link-spec.md |
— |
| F20-i Live inbox (added 2026-10-17, split from F20-f; widened 2026-10-28, 2026-11-05) | Replaced 2026-11-19 by the ### F20-i — Live inbox block above (spec FR-25, written by F20-i-WI-5.1). |
F20-i-live-inbox-spec.md |
— |
| F20-j Past-mail linking (added 2026-10-17, split from F20-f) | The 12-month header scan’s run time on a real, filled test mailbox, and the per-record $search. Phase 1 |
F20-j-past-mail-linking-spec.md |
— |
F26-c-weekly-workbook-upload-spec.md |
— | ||
Mail.Send. Phase 1 |
F26-d-send-weekly-prices-spec.md |
— | |
| F4-b Pipedrive migration (lead writes) | Replaced 2026-11-06 by the ### F4-b — Pipedrive migration: lead writes block above (spec FR-21, written by F4-b-WI-1.4): its three items are the write run with leads on dev, the triage backlog on dev, and the production first load and final run with the lead writes. |
F4-b-migrate-leads-spec.md FR-21 |
Dinil |
| F23-b Spreadsheet import | Imports upload through FS-11 — real S3 (A-1). F23-d and F23-a need none: a list export is built and returned in its request, with no stored file (F23-d FR-E7, the lead’s decision of 2026-11-15). F23-b is Phase 2 | F23-b-spreadsheet-import-spec.md |
— |
AI features on the AIProvider seam — Phase 1: F11-d (summary, spam/duplicate — now its own §B block, F11-d), F11-g (ranking — now its own §B block, F11-g), F11-h (drafted follow-up); Phase 2: F6-c (feedback-chase draft), F14 (quote assistant), F15-a (morning briefing). F7-e and F16 were dropped on 2026-10-11 (PRD Rev 61) |
Each calls Bedrock through the seam F12 builds; locally each runs on the stub or the manual path. Live model quality, latency (under 10 s or a background job) and cost are only measurable on real Bedrock (§C-5) | their stub specs; §C-5; AD #19 | — |
C. By capability — the view for whoever stands the environment up
Each capability lists the engagement-level rows that have no single feature owner, then points at the §B blocks that carry the rest. Do not duplicate a §B row here.
C-1. Authentication & identity — Microsoft Entra (biggest gap; risk W-1 / R-FND-003)
The whole real login path is untested locally; only the profile-scoped stub login runs. Rows:
§B FS-6 (certificate, app registration, forwarded headers, domains, cookies, CORS, roles,
stub absence), §B F5-a (the round trip, from-route, ?login=failed, storageState; its first-admin row
is withdrawn), §B F5-c (the identity link, the deactivated user, the mailbox release), §B F5-k (the seeded
admin per environment, the production email in Secrets Manager, the unset-placeholder refusal),
§B F20-a (mailbox reply URL, consent, refresh, subscription, KMS),
§B F20-b (notifications, delta, renewal).
One engagement-level row — a way to prove part of C-1 before the cloud exists:
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
Built: real Microsoft sign-in and mailbox consent from a developer laptop, so the F5-a round trip and F20-a’s consent, same-user check and token refresh can be proved before AWS exists. It uses the two laptop reply URLs and the non-prod certificate (shared with dev and UAT — no laptop certificate of its own). Laptop mode is switched on by the backend entra-laptop profile, together with local (SPRING_PROFILES_ACTIVE=local,entra-laptop ./gradlew bootRun; refused without local or with test), and the frontend VITE_ENTRA_LAPTOP flag under pnpm dev (F5-i-WI-1.2); both are off by default. The how-to is forge-et-be/docs/auth/laptop-sign-in.md |
The deployed environments do not exist (A-5); this is the stop-gap, not a substitute | .forge/design/environments-and-identity.md §2.2, §2.5; forge-et-be/docs/auth/laptop-sign-in.md; forge-et-be/docs/auth/README.md § “The profile shapes”; the F5-i Wave 1 code PRs (backend forge-et-be#86, WI-1.1; frontend forge-et-fe#65, WI-1.2 — per-WI review PRs, shipped through the wave PR) |
Not a cloud step. Follow the how-to, sign in as CRM01@empire.lk on a laptop (the seeded admin — needs F5-k on main) and connect its mailbox; record the evidence in .forge/plans/F5-i/F5-i-WI-2.1-record.md and against the §B F5-a, §B F20-a and §B FS-6 certificate-login rows it covers, marked “laptop, not dev”. The rows stay open until they pass on dev too — a laptop proves nothing about cookies across hosts, CORS, KMS, Graph notifications or the ALB |
Tracked as F5-i (Phase 1, owner Tharinda); the Entra side is ready — both laptop reply URLs registered and the non-prod certificate uploaded on 2026-09-24; the Wave 1 code is built (the backend profile and its startup check, the frontend flag and the dev-proxy paths); the live run as CRM01 (F5-i-WI-2.1) was done on 2026-09-29 and 2026-09-30 (Nanndish; .forge/plans/F5-i/F5-i-WI-2.1-record.md, merged in forge-et-harness#322), and the rows it covers carry its “Laptop, not dev” notes and stay open |
implemented |
C-2. Cross-host cookies, CSRF & CORS
A-4 is decided (AD #33 — the apps call the backend cross-site); the cookie code change is built
(F5-j, forge-et-be#78 and forge-et-fe#59) and only the per-environment values remain. Rows:
§B FS-6 (CSRF readability, CORS list and preflight max-age, Secure cookies,
login-redirect-uri), §B F5-j (both sides set, with matching names) and §B F5-a
(VITE_API_BASE_URL).
C-3. Infrastructure — FS-1 cloud half (Terraform / the AWS estate) — see A-2
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
The Terraform estate: ECS Fargate + ALB + ACM, RDS, S3, SQS (+ DLQ), EventBridge Scheduler, KMS, Secrets Manager, CloudFront for both frontends, per-environment modules (dev, prod, temporary uat), state in S3 locked by S3’s own lock file (AD #23) |
No AWS account | AD #22/#23; CLAUDE.md Active Context; architecture.md § Deployment |
terraform apply dev + prod (+ a uat per round); each service comes up and the app deploys; terraform destroy of uat is clean Update 2026-10-21 (F38-b WI-2.1): forge-et-iaac#3 was merged into dev (d230b133) and applied by dev.yaml run 39475850266 on 2026-10-20, adding the KMS keys, the attachments bucket, the ingest queue, EventBridge Scheduler and the task-role grants to dev; the capture app’s hosting, prod and uat are still to come, and the no-drift check is F38-b WI-4.1’s; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-1. Update 2026-10-26 (F38-b WI-4.1): dev’s estate is applied — the capture app’s hosting included, serving its build — and the sweep finds nothing outside Terraform; the no-change dev.yaml plan after c728e06b is not yet recorded (later the same day: run 40278225094 planned No changes. — AC-1 met); prod and uat are still to come; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-1, AC-7. Update 2026-11-13 (F38-h WI-3.1): overlays/uat and uat.yaml merged into dev (forge-et-iaac#11, b175dae); dev unchanged (dev.yaml run 43376614685: one move, 0 to add, change or destroy). Unlike the plan’s expectation, the instance was then applied the same day by Senuthmi Thimansa with service unticked (run 43378306661, 75 resources), before deploy-uat exists (F38-h spec Rev 8). The rehearsal’s destroy and three-step re-apply (WI-5.1, WI-5.3) are still what proves the overlay; evidence: environments-and-identity.md §4, “The uat instance”. |
Tharinda | open |
Backend secrets via task-definition secrets → Secrets Manager; AWS access via the task IAM role (no static keys anywhere) |
No task definitions or IAM locally (LocalStack fakes AWS) | AD #22 | The backend reads secrets from Secrets Manager and every SDK client uses the task role | — | open |
altrium-io/forge-et-iaac, for all DevOps work; AD #23 revised. What remained, how the delivery team reviews that repository, was decided 2026-10-02: every change is a PR to its trunk dev, reviewed by the others with write access (git-conventions.md § forge-et-iaac) |
Team decision | architecture.md revision notes; AD #23 |
Decide before the first apply; update AD #23’s “lives in forge-et-be/terraform/” if it moves |
Tharinda / Shyamin Ayesh | implemented |
DNS for the CRM hosts under empire.lk — revised 2026-09-30 (F38 Rev 2): Empire IT delegates crm.empire.lk once, with four NS records in the empire.lk zone; Terraform then writes the ACM validation records, a CAA record, the host records for every host (crm., api.crm., capture.crm. and the dev. / uat. variants — environments-and-identity.md §1) and the SES records on a sending domain under crm.empire.lk into the Route 53 zone. A sending domain on empire.lk itself would need one further request (F38 OQ-E). Update 2026-10-01: the delegation is live — Kavindu Anthony added the four NS records in Cloudflare, which serves empire.lk DNS (environments-and-identity.md §1.1); the row stays open for the Terraform-written records |
The delegation needed only the zone (live 2026-10-01); the records need ACM, SES and the load balancer and distributions to exist | .forge/client/dependencies.md §1.4 (item 5); environments-and-identity.md §1, §1.1; §B FS-6 “Custom domains” row; F38 FR-6 |
dig NS crm.empire.lk +short returns the zone’s four name servers; each host resolves over HTTPS; SES shows the domain verified. Update 2026-10-15 (F38-a WI-1.1): dev’s half read — dig NS crm.empire.lk +short returns the four name servers (also through 1.1.1.1), the api.dev.crm.empire.lk and dev.crm.empire.lk certificates are ISSUED in us-east-1, dev holds no hosted zone, and dev’s Terraform writes its dev. records into the zone in the management account through dns-dev; still open for dev’s SES records (F38-a WI-3.1) and the capture app’s host and certificate (F38-b), and for production (F38-i); evidence: environments-and-identity.md §4 “F38-a — the development account, checked”, AC-6. Update 2026-10-21 (F38-a WI-3.1): dev’s SES records resolve through the local resolver and 1.1.1.1 — the three DKIM CNAMEs, the MX and SPF TXT on bounce.dev.crm.empire.lk, and _dmarc.dev.crm.empire.lk (v=DMARC1; p=none) — each declared in forge-et-iaac base/ses/records.tf at 0eee623 and written through aws.dns by module "ses" in overlays/dev/main.tf, so the dev SES part of this row is met; still open for the capture app’s host and certificate (F38-b) and for production (F38-i); evidence: environments-and-identity.md §4 “F38-a — the development account, checked”, AC-5. Update 2026-10-26 (F38-b WI-4.1): the capture app’s host is done for dev — capture.dev.crm.empire.lk is answered with aa by the zone’s name server ns-1983.awsdns-55.co.uk, as are dev. and api.dev.crm.empire.lk, and its certificate is ISSUED and in use by its distribution; still open for production (F38-i); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-7. |
Tharinda to ask; Kavindu Anthony (Empire IT) to add the NS records | open |
| Database-credential rotation in Secrets Manager | Only exists in the account | Proposal §A.4 (“with rotation”); architecture.md § Committed security baseline |
Enable rotation on the RDS secret; the backend keeps working across a rotation. Update 2026-10-09: dev uses RDS-managed rotation of the master user (F38 FR-10’s first option), with the backend connecting as the master user app; nothing restarts the task after a rotation yet, so the first scheduled rotation (default every 7 days — about 16 Oct) leaves the running task unable to open new connections. Shyamin Ayesh has it on his list (environments-and-identity.md §4, “Checked against F38” item 1). Update 2026-10-14: still on; the next rotation is 2026-10-16 23:59:59 UTC (17 Oct 05:29:59 in Sri Lanka). Update 2026-10-16: on 15 Oct Shyamin Ayesh lengthened the schedule to every 365 days with the AWS CLI, without an immediate rotation (CloudTrail RotateSecret; the schedule is not in Terraform — environments-and-identity.md §4 item 1); read in the account, the next rotation is 2027-10-15 23:59:59 UTC and the last is still 9 Oct, so the password was not changed. The row stays open: the backend is still not restarted after a rotation, which is needed before production (F38-i) and before the schedule returns to a short interval. Production does not copy the dev workaround: a yearly rotation still breaks a backend nothing restarts, and a schedule set outside Terraform is lost when the database or its secret is recreated. Validated in production by a forced rotation with the backend still serving afterwards (F38 AC-40). Update 2026-10-16 (the lead, F38 Rev 14 — planned, not yet applied): the password is not rotated on a schedule in any environment: the RDS-managed secret’s automatic rotation is set in Terraform, in the module every environment uses, to the longest interval Secrets Manager accepts (or off, if AWS allows that for a secret RDS manages), no restart on rotation is built, and the password is changed by hand with a forced redeploy when needed (.forge/guides/aws-setup-explained.md §7). The row now validates that: the secret’s rotation setting is in Terraform in each environment, and, once in dev, a change by hand followed by a redeploy leaves the backend serving (F38 AC-10, AC-40 as revised). A post-issue note on the proposal records the departure from §A.2.3 and §A.4 Update 2026-10-18: F38-b’s forge-et-iaac#3 (delivery team, open; the DevOps team reviews, plans and merges) sets aws_secretsmanager_secret_rotation with rotation_enabled = false on the RDS-managed secret in the shared base/rds module (F38-b spec Rev 4). Validated after the apply by aws secretsmanager describe-secret showing RotationEnabled: false (F38-b AC-3); a password changed by hand with rotate-secret turns rotation back on until the next apply (.forge/guides/aws-setup-explained.md §7). Update 2026-10-21: #3 was merged into dev on 20 Oct (d230b133) and applied by dev.yaml run 39475850266; aws secretsmanager describe-secret on the RDS-managed secret shows RotationEnabled: false and no next rotation date, so dev’s setting is in Terraform. Still to validate: the change by hand with a redeploy in dev, and the setting in each later environment. Update 2026-10-26 (F38-b WI-4.1): re-read on dev — still RotationEnabled: false with no next rotation date, declared by base/rds/rotation.tf at c728e06b; the change by hand (F38-f) and the later environments remain; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-3. |
F38 | open |
Plus §B FS-1, §B FS-2, §B FS-7, §B FS-9, §B FS-11, and the F20-a KMS key row. §D is the task-definition checklist.
C-4. Observability & mail — FS-5 cloud half
Rows: §B FS-5 (log groups, JSON format, alarm substrate, SES, error reporter, exporter) and the feature-owned alarm rows it points at (F20-a, F20-b, FS-7, FS-9, FS-11).
C-5. AI layer — Amazon Bedrock
The AI layer runs on Bedrock in-account; there is no Bedrock locally, so every live AI feature
is cloud-gated (local degrades to the manual path or a stub, per feature {model, enabled}).
Lead decision 2026-09-23: AI features stay stub-only until F38 — no direct Anthropic API for
local development either (design/ai/ai-provider-strategy.md §5 row 7). Specs for AI features
should say which ACs are asserted against the stub and owe a real-Bedrock check on dev.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
Live AIProvider calls against real Bedrock (Haiku 4.5 / Sonnet 5.5); F12 builds the seam and its Bedrock provider on FS-11’s SeamGuard pattern (R-FND-001), and its own rows are the §B F12 block |
No Bedrock endpoint locally | AD #19; .forge/design/ai/ai-provider-strategy.md; F12 spec |
Exercise each AI feature (extraction, scoring/triage, briefings, drafting, card reading) against real Bedrock; confirm cost and latency are acceptable; confirm the per-feature {model, enabled} switch degrades to the manual path Update 2026-11-21 (F38-f WI-2.1): extraction, triage, research, drafting, briefing and lead ranking were exercised through the app on dev, all on Haiku 4.5 — 41 calls from the task role, 39 SUCCESS and 2 lead-ranking INVALID_OUTPUT, the slowest 5 258 ms, $0.195 in all; card reading and the capture app’s research and ranking were not exercised, and the {model, enabled} switch is F38-f-WI-2.2’s; evidence: §B F38-f “F38-f evidence — AC-2 and AC-3”. |
— | open |
| The anonymised-inquiry evaluation-key comparison (proposal A.3.1) | Needs real Bedrock | ai-provider-strategy.md |
Run the comparison when Bedrock is live. 2026-10-26: the lead declined running it on Anthropic’s API with evaluation keys, for cost; it runs on Bedrock once Sonnet 5.5 and Haiku 5.5 answer there (model-choice-evaluation-plan.md, D-2; ai-provider-strategy.md §4d). |
— | open |
Bedrock usable for Haiku 4.5 and Sonnet 5.5 in us-east-1, and service quotas at the expected volume, in both accounts |
Model access is on by default, but Anthropic’s models need the first-time-use form, the Marketplace subscription permissions on the first caller and a payment method; new accounts may receive reduced quotas (AWS “Request access to models” and “Quotas for Amazon Bedrock”, read 2026-09-27) | .forge/client/dependencies.md §1.1 (Bedrock row: “model access enabled and quotas raised by Altrium during environment setup”); AD #19; F38 FR-4 |
Submit the first-time-use form once; make the first call to each model from an identity with the three aws-marketplace permissions; read both accounts’ quotas and request increases (F38-a FR-7, AC-4); one call to each pinned model succeeds from the task role (F38-f). Update 2026-10-09 (DevOps channel): in dev, svc-harness holds the Marketplace permissions; Haiku 4.5 is subscribed and answers; Sonnet 5.5 and Haiku 5.5 both fail with “… is not available for this account … contact AWS Sales”, so the synthesis model AD #19 pins cannot be called yet. The account holder (Empire Teas — Kavindu Anthony, with Shyamin Ayesh) asks AWS to enable both models for the account; until then F12 and every Sonnet-5.5 feature can be proved only on the stub. The task role holds no Bedrock permission yet. Update 2026-10-14: Shyamin accepted both models’ agreements, which now read AVAILABLE, but calls through both us. and global. profiles are still refused, because Service Quotas gives both models 0 tokens a minute; Haiku 4.5 has 10 requests a minute. Left: a quota increase for both models and for Haiku 4.5’s requests, raised by Shyamin Ayesh (the lead, 14 Oct) (environments-and-identity.md §4 item 4). Quotas are per account and per Region: a request in the management account does not reach dev, while a Service Quotas request template there covers accounts created later (production, a separate UAT account). Update 2026-10-15 (F38 Rev 12, F38-a WI-1.1): F38-a covers the development account only — the form, the quotas and the requests for Sonnet 5.5 and Haiku 5.5 there; production’s half is F38-i’s. Read in dev the same day: a use case is on file, all three models read AVAILABLE and AUTHORIZED, no quota increase has been requested, the 5.5 models’ tokens-per-minute quotas are 0 and Haiku 4.5’s requests-per-minute quotas 10, and Service Quotas has no requests-per-minute quota for either 5.5 model, so only their tokens per minute can be raised; the quota request template cannot be read from a member account. Evidence: environments-and-identity.md §4 “F38-a — the development account, checked”, AC-4. Update 2026-10-16: Shyamin Ayesh filed the six increases in dev on 15 Oct, at the AWS defaults; AWS closed all six cases within half an hour and no quota changed (0 tokens a minute for both 5.5 models, 10 requests a minute for Haiku 4.5), so the models are still refused. Next: Shyamin reads AWS’s replies in the Support Center (environments-and-identity.md §4 item 4, the 16 Oct update). Update 2026-10-22: AWS refused three more requests on 21 Oct (Haiku 4.5’s Global requests per minute, then the Global tokens per minute of Haiku 5.5 and of Sonnet 5.5, each at the default), with “We are unable to approve your request at this time”; read on 21 Oct at 21:13 UTC, every quota is unchanged. The lead’s reply, for Shyamin Ayesh to send through an AWS Support “Service limit increase” case, asks for Haiku 4.5 at 100 requests a minute and Sonnet 5.5 and Haiku 5.5 at 400,000 tokens a minute on both their quotas, or any non-zero value for the 5.5 models; AWS’s answer is not recorded yet (environments-and-identity.md §4 item 4, the 22 Oct update; the case numbers in its AC-4 subsection). Update 2026-10-22, later (read at 04:59 UTC): every quota is still unchanged, Haiku 4.5’s included; the lead reports Shyamin asked AWS again on 21 Oct, presumably through the Support case, whose answer comes through him. A call counts only against the quota of the profile it names: the backend uses Haiku 4.5’s cross-Region L-CCA5DF70 and Sonnet 5.5’s Global L-31AB82D0, and F38-a WI-2.1 also uses Haiku 5.5’s cross-Region L-47520EB7; the two Haiku requests filed in Service Quotas on 21 Oct were the Global ones, which neither uses, while the Support-case reply asks for both (environments-and-identity.md §4 AC-4, the 22 Oct table). Update 2026-10-22, last: the “asked again” was the two Service Quotas requests of 09:24 and 09:25 UTC, not a Support case; AWS emailed the same refusal for the Sonnet 5.5 one (case 179257474300142), and Shyamin Ayesh then sent AWS the lead’s reply. Read at 11:52 UTC: every quota unchanged; AWS’s answer to the reply is awaited (environments-and-identity.md §4 item 4, the last 22 Oct update). Update 2026-10-26: AWS answered the reply with no increase — the account keeps new-account limits (Sonnet 5.5 and Haiku 5.5 at 0, Haiku 4.5 at 10 requests a minute) until it has a few billing cycles of Bedrock usage and spend, and AWS lifts them on its own review, with no date. Two more requests that day (Opus 5.5’s and Sonnet 5.5’s cross-Region tokens a minute) were closed unchanged. Every row in this register that waits for “once its quota is raised” therefore has no date; the lead’s decisions of the same day (Haiku 4.5 everywhere until the limits lift, and the backend pacing its Bedrock calls) are ai-provider-strategy.md §5 row 10, and what is left to do is .forge/owed.md, “The AI layer under Bedrock’s new-account limits” (environments-and-identity.md §4 item 4, the 26 Oct update, and AC-4’s 26 Oct entry). |
F38-a (dev: form, quotas, the requests for Sonnet 5.5 and Haiku 5.5); F38-i (production); F38-f (task-role call) | open |
| Sonnet 5.5 and Haiku 5.5 answer one call each in dev — F38-a’s former AC-4 calls, moved here by F38-a Rev 4 (2026-10-26) so F38-a can close | AWS keeps the dev account at new-account limits (Sonnet 5.5 and Haiku 5.5 at 0 tokens a minute) until it has a few billing cycles of usage and spend, with no date (row above; environments-and-identity.md §4 item 4, the 26 Oct update) |
F38-a FR-7 “The proof” and AC-4 (Rev 4); parent F38 AC-4 (Rev 29); ai-provider-strategy.md §4d |
When the weekly quota read (.forge/owed.md, “The AI layer under Bedrock’s new-account limits”) shows Sonnet 5.5’s Global L-31AB82D0 or Haiku 5.5’s Global L-F26CA8D1 above 0: with the lead’s permission at the time, one bedrock-runtime Converse call to global.anthropic.claude-sonnet-5-5 and one to global.anthropic.claude-haiku-5-5 each return a completion (the Haiku 5.5 call moved from us. and L-47520EB7 to the Global profile chosen on 2026-10-27, .forge/owed.md), made by the lead or by Claude with that permission, from an administrator identity in dev (svc-harness, which keeps broad access in dev by F38 Rev 9, or the lead’s own login); record the date, the endpoint, the profile, the identity, who gave permission, and the quotas read that day in environments-and-identity.md §4 “F38-a — the development account, checked”, AC-4. The run sheet is .forge/plans/F38-a/F38-a-WI-2.1-plan.md, kept for this. The call from the task role is §B F38-f’s row |
Tharinda (the lead), when the quota rises | open |
Claude Haiku 4.5 has no in-Region on-demand route on bedrock-runtime in us-east-1. It needs the us. geo inference profile, which may process a request in us-east-2 or us-west-2, or the bedrock-mantle endpoint, which serves it in-Region through the Messages API only. Sonnet 5 is in-Region. (Update 2026-09-30: Sonnet 5.5 replaces Sonnet 5, and it has no in-Region or us. route either — only the Global profile, which may process a request in any commercial Region; ai-provider-strategy.md §4b, §5 row 9.) The model card also gives Haiku 4.5’s end of life as no sooner than October 2026, then a legacy period of at least six months. Proposal §A.3.1 and §A.6.5 promise “no new region”, and AD #19 and ai-provider-strategy.md said (until 2026-09-30) that every pinned model is native to us-east-1 Update 2026-11-04 (F43-WI-4.2): both of Haiku 4.5’s profiles, us. and global., are ACTIVE in the account (read 2026-11-02, F43 spec § In Scope). Since F43 the calls a person waits for use the us. profile and lead triage and lead research the Global profile, which may process a request in any commercial AWS Region, on the lead’s direction of 2026-11-02 (ai-provider-strategy.md §4b, §5 row 9; OQ-15); the proposal’s post-issue note of 2026-11-04 records it. |
Which route works, and at what quota, is visible only in the account; the route choice is a lead decision | AWS Bedrock “Regional availability by model” and the Claude Haiku 4.5 model card (read 2026-09-28); .forge/specs/F38-a-account-baseline-spec.md FR-7, Q-3 |
The lead chooses a route (the profile plus a post-issue note on the proposal, the bedrock-mantle endpoint, or another model — F38-a Q-3); F12 builds the AIProvider for it; F38-a confirms the route in dev, F38-i in production, and each reads its account’s quotas (F38-a AC-4) |
Tharinda (decision); F38-a (confirm); F12 (build) | open |
SP-ARCH-004 stage 2 — the SearchProvider choice: the AgentCore Gateway needs the account, and Bedrock does the scoring; plus ~30 lines of throwaway Java proving the backend can call AgentCore |
Both halves need the account | .forge/design/ai/lead-research-source-options.md §10; .forge/owed.md |
Run stage 2; record the provider choice before the F11-c spec. Update 2026-10-09: Dinil asked Shyamin Ayesh for the AgentCore Gateway in us-east-1 (a web-search target, IAM SigV4 inbound auth, in Terraform if the provider supports it) and bedrock-agentcore:InvokeGateway for svc-harness; not built yet (environments-and-identity.md §4). Claude as the judge needs Sonnet 5.5, which the account refuses for now (the row above). Validated 2026-10-13: the gateway (et-dev-gateway, built from forge-et-iaac) was called as svc-harness after an inline spike-invoke-gateway policy gave it InvokeGateway; AgentCore ran 82 searches on the same 25 customers with no errors, and Claude Haiku 4.5 — enough for judging, so Sonnet 5.5 was not needed — judged all four providers’ 328 saved responses. AgentCore came last (5 of 25 customers found; Exa 13, Serper 11, Tavily 8). Evidence and totals: lead-research-source-options.md §11, Rev 6. Still owed, and not cloud-gated: the provider choice (owed.md); the Java call only if AgentCore is chosen |
Dinil | validated |
| AI-spend alarm | Needs FS-5’s substrate and a live provider | FS-5 spec Out-of-Scope table; AWS Budgets (AD infra) | Set a budget alarm on Bedrock spend; trip it once. Update 2026-10-23 (the lead; F38 Rev 23): an AWS Budgets alert, which belongs to the billing owners since F38 Rev 18 (Shyamin Ayesh, Kavindu Anthony, Malinda Wellalage, Thilina Abeyrathna); F38 does not build it. The proposal’s AI budget names “a billing alert at $150” | billing owners | open |
C-6. CI / deploy pipeline
Rows: §B FS-1 (CI credentials, deploy workflows), §B FS-4 (smoke/full tiers, smoke set, flakiness decision, the release vulnerability check that replaced the Trivy flip (AD #38), dependency submission).
C-7. End-to-end retest on dev of everything built before the cloud existed — done in F38
Until the dev environment exists (A-5), every feature has run only against the stub login, the
stub mailbox, LocalStack and a laptop Postgres, so done in the tracker means “done locally”.
F38 retests all of it in one pass when the dev environment is ready
(.forge/specs/F38/cloud-bring-up-spec.md). No per-feature rows are kept here in the meantime:
which features are affected is not known until that day, and the tracker already records it.
Features started after dev exists are verified on dev as part of their own delivery.
Since F38 Rev 21 (2026-10-19), F38-g carries the retest as a per-feature checklist walked by hand
on dev; the checks that need something dev does not have yet are its §B F38-g rows, run when each
arrives.
C-8. Commitments in the issued proposal and the AWS SOW that only a real environment can meet
These are binding (L-001, .claude/rules/source-trust.md): the accepted proposal and SOW ET-002
promise them to Empire, and nothing in the local stack can show them. F38 carries them.
Update 2026-09-29: SOW ET-002 is superseded and was never signed (Empire Teas holds the AWS account). Rows below that cite only ET-002 — the quarterly restore test, whole-estate cost oversight and monthly reporting — are no longer contractual on that basis. They stay as open rows, because the proposal’s §A.5 still commits the pre-go-live restore rehearsal and the support terms are still to be agreed. Which of them Altrium carries under the USD 360/month support fee is for the lead to settle with Altrium management.
| Item | Why cloud-gated | Source | Validate / finish when cloud is up | Owner | Status |
|---|---|---|---|---|---|
| Backups: RDS point-in-time recovery at 5-minute granularity, retained 14 days in production and 3 days in development; daily snapshots; S3 versioning on attachments. (Cross-region backup copy is not part of the baseline — it is a client decision in the row below.) | Only exists on real RDS/S3 | Proposal §A.5; SOW ET-002 §3 “Backup and recovery”, §6; architecture.md AD-O-5 |
Terraform sets each; confirm in the console (retention, PITR window, versioning) per environment. Update 2026-10-14: in development, backups are optional, the DevOps team’s call (F38 Rev 11; a post-issue note on the proposal); dev has none today. Production’s 14 days stands (F38-i) | F38 | open |
| A documented restore rehearsal before go-live, then a restore test each quarter | Needs a real backup to restore | Proposal §A.5; SOW ET-002 §3 (“quarterly restore testing”) | Restore production-shaped data to a point in time; time it against the §A.5 targets (5 min loss, 1–2 h restore); write it up; schedule the quarterly test | F38 (rehearsal); Altrium operations (quarterly) | open |
| Security baseline: GuardDuty (account threat detection), CloudTrail to a central archive, KMS for database/storage/backups, IAM Identity Center with MFA and no long-lived keys for people, Shield Standard, dev and prod in separate accounts; standing controls — TLS 1.2+, database not publicly reachable, S3 public access blocked at account level, least-privilege security groups | Account-level controls; none exist without the accounts | Proposal §A.4; architecture.md § Committed security baseline; AD #4 |
Terraform or account setup enables each; check each in the console per account. Update 2026-10-09: GuardDuty threat detection is left out by Altrium management’s decision (§B FS-11 block note) — a departure from proposal §A.4, recorded in its post-issue note of 9 October and the client-facing PRD; on 11 October the lead decided Empire Teas is not told separately; the rest of the row stands Update 2026-10-14: in dev, svc-harness is a long-lived key people use, with MFA on its console password optional, and the team has no personal Identity Center logins (F38 Rev 9, Rev 11; the proposal’s second post-issue note of 14 October); production keeps the row as written Update 2026-10-15: in dev there is also no CloudTrail trail (the free 90-day event history is the record), no root protection, password policy or alternate contacts, and no account-level S3 Block Public Access (F38 Rev 13; the proposal’s post-issue note of 15 October). F38-a records what dev has; production’s half of this row is F38-i’s (F38 Rev 12) Update 2026-10-15 (F38-a WI-1.1): dev’s half read — dev is a member account of o-imaviqd8nu with everything but default VPCs in us-east-1; the only IAM users are svc-github and svc-harness, one key each, and the root user has no access key; Shield Standard is present (Shield Advanced INACTIVE), GuardDuty has no detector, account-level S3 Block Public Access is not set and there is no trail, each as expected in dev; the IAM credential report (generated 09:11 UTC) shows keys only on those two users and none on the root user, and the CloudTrail event history answers a lookup-events read, both run by Claude with the lead’s permission of 09:10 UTC because neither is a describe, list or get call; evidence: environments-and-identity.md §4 “F38-a — the development account, checked”, AC-1 and AC-2. Update 2026-10-17: in dev the database is not encrypted at rest and there is no database or backups KMS key, only the storage and mailbox token keys (F38 Rev 17; the proposal’s post-issue note of 17 October); whether production’s database is encrypted is undecided (.forge/owed.md, F38-i) |
F38 | open |
| Security review before go-live — authentication, session handling, permission enforcement, file upload, the AI prompt boundary | A committed delivery step; meaningful only on the deployed shape | Proposal §A.4 (last paragraph) | Run and record the review on the UAT or production shape before go-live; findings fixed or accepted | Tharinda | open |
| Log retention 30 days for application and infrastructure logs; AI prompt and response bodies never persisted in production logs (development only); AI call metadata kept for the life of the system | Retention is a CloudWatch setting; the prompt-body rule can only be checked on production logs | Proposal §A.6.3; architecture.md AD-O-6 |
Set log-group retention to 30 days; run an AI feature in production and confirm its prompt and response text appear in no log group, while its metadata row exists. Update 2026-10-23 (F38-e WI-1.2): dev’s half of the retention is met — every log group in the dev account, in every Region, keeps 30 days; production’s retention and the prompt-body rule are F38-i’s; evidence: §B F38-e “F38-e evidence — AC-1”. | F38; each AI feature’s spec (§B backlog table) | open |
| Budget alerts and cost oversight for the whole estate (not only AI spend — §C-5 has that row) | AWS Budgets exists only in the account | SOW ET-002 §3 “Capacity and cost management”; .forge/client/sow-cost-annexe.md |
Create an AWS Budget per account at the annexe’s expected monthly figure with alert thresholds to the delivery team; trip one. Update 2026-10-15: dev’s budgets and their recipients are arranged by the DevOps team with Empire Teas, and F38-a only records them (F38 Rev 13); production’s are F38-i’s Update 2026-10-15 (F38-a WI-1.1): dev has no AWS budget (describe-budgets returns none), so there is no amount or recipient to record; evidence: environments-and-identity.md §4 “F38-a — the development account, checked”, AC-7. Update 2026-10-18 (F38 Rev 18): billing settings — budgets, their alerts and recipients, the cost allocation tags — are Shyamin Ayesh’s, Kavindu Anthony’s, Malinda Wellalage’s and Thilina Abeyrathna’s, not the delivery team’s; the delivery team keeps costs down through design and daily use (.claude/CLAUDE.md § Boundaries). |
F38 | open |
| Monthly environment-health and usage reporting to Empire | Needs a running estate to report on | SOW ET-002 §1, §3 “Reporting and vendor coordination” | Agree the report shape; first report after the production environment is live | Altrium operations | open |
| Client decisions the proposal leaves to Empire: whether the single-availability-zone recovery targets are acceptable, or to add the multi-AZ database (priced add-on) that cuts an AZ-failure restore to under five minutes; and whether to add cross-region backup copy (priced add-on), without which the proposal’s region-loss target does not hold | The proposal asks Empire to confirm; each choice changes the RDS and backup Terraform. Cross-region copy is outside SOW ET-002 unless added through Change Control | Proposal §A.5 (“Empire Teas should confirm they are acceptable”; the region-loss row “if cross-region copy is enabled”), §A.2.3; SOW ET-002 §6 (“Disaster recovery boundary”) | Ask Empire before the production RDS is created; record the answer here and in the cost annexe. Update 2026-10-27 (F38 Rev 31, F38-i decision 1): F38 OQ-J is closed — SOW ET-002 §2.2 (Multi-AZ RDS and WAF planned for production) no longer binds, because ET-002 was superseded on 2026-09-29 and never signed; the proposal stands, so both choices remain Empire’s priced add-ons. .forge/owed.md, “Empire IT asks”, the message drafted that day) and, if it or the DevOps and billing owners ask, each is built as a change of its own. |
Tharinda to ask | open |
| Whether to add AWS WAF before go-live (the proposal recommends it “once the system carries real business”); Security Hub, Config and Inspector reviewed at the same point | Priced add-ons decided at go-live | Proposal §A.4 (“Available as priced add-ons”), §A.2.3 | Decide with Empire before production go-live; if taken, Terraform it in front of the ALB and CloudFront. Update 2026-10-27 (F38 Rev 31, F38-i decision 1): F38 OQ-J is closed and the proposal stands — WAF is a priced add-on Empire decides, not a planned part of production’s baseline; |
Tharinda with Empire | open |
D. Task-definition and build checklist — every value a cloud environment must carry
Derived from the code on 2026-09-23 (forge-et-be 6879999, forge-et-fe 3b985d9,
forge-et-pwa 22b4050). Re-derive from EtProperties and application.yml before use; this
list is a snapshot, not the source. Cloud runs with no Spring profile (or dev/prod), never
local/test; values bind from environment variables by relaxed binding.
Backend — must set
| Variable | Purpose | Enforced by |
|---|---|---|
ET_ENVIRONMENT |
environment name (dev, uat, prod); never test, which is reserved for test contexts (BUG-070) |
required, no default (FS-1 R11) |
ET_DATASOURCE_HOST, _DATABASE, _USERNAME, _PASSWORD (_PORT defaults 5432) |
RDS; password from Secrets Manager | FS-2 FR-D5 |
ET_DATASOURCE_SSL_MODE (property et.datasource.ssl-mode) = verify-full in every cloud environment |
the backend checks the RDS certificate against the RDS CA bundle in the image; values prefer and verify-full, default prefer, which a cloud task definition must not leave. From F38-d WI-4.1 (not yet merged); source F38-d FR-7 and .forge/plans/F38-d/F38-d-WI-4.1-plan.md D-3 |
not enforced: an unset key starts with prefer (WI-4.1 D-8), so uat and production copy it from here |
ET_BUSINESS_TIMEZONE |
business clock | FS-2 |
ET_SECURITY_LOGIN_REDIRECT_URI |
SPA origin; base for login and mailbox ?connect= redirects |
@NotBlank; and, since BUG-041, an http or https origin only: no path, no trailing slash, no query or fragment (https://crm.empire.lk, not https://crm.empire.lk/): startup refuses anything else, naming the key |
ET_SECURITY_CAPTURE_REDIRECT_URI (property et.security.capture-redirect-uri) |
the capture app’s origin, where a sign-in started from the capture app returns (F36-a FR-A2): https://capture.dev.crm.empire.lk, https://capture.uat.crm.empire.lk, https://capture.crm.empire.lk; a plain value (AD #40). Unset, a capture-app sign-in returns to the CRM. Added by F36-a-WI-1.1; §B F36-a row (a) |
when set, the same rule as ET_SECURITY_LOGIN_REDIRECT_URI: an http or https origin, no path, no trailing slash |
ET_SECURITY_CORS_ALLOWED_ORIGINS |
the environment’s CRM and capture origins (AD #33); empty means none | FS-6 FR-S5 |
ET_SECURITY_CSRF_COOKIE_DOMAIN, ET_SECURITY_CSRF_COOKIE_NAME |
the CSRF cookie’s Domain = the environment’s CRM host (crm.empire.lk, uat.crm.empire.lk, dev.crm.empire.lk) and its name (XSRF-TOKEN-PROD, -UAT, -DEV — equal to that environment’s VITE_CSRF_COOKIE_NAME). Plain values, not secrets. Unset locally |
AD #33, F5-j: startup refuses a domain without a name or with XSRF-TOKEN, a bad name, or a domain that is not a bare host. Unset starts cleanly as host-only XSRF-TOKEN, so an environment that forgets both is caught only by §B F5-j |
ET_SECURITY_ENTRA_TENANT_ID, _CLIENT_ID, _CERTIFICATE, _PRIVATE_KEY |
Entra login; private key from Secrets Manager, the other three plain task-definition values in every environment (AD #40; F38-i FR-19 moves dev’s off their secrets references — done in forge-et-be .aws/td.dev.json by F38-i wave 3, forge-et-be#374, effective from the deploy of its merge; the three secrets stay in forge-et-iaac base/secrets until F38-i wave 8) |
EntraLoginConfig throws if missing |
ET_SECURITY_SEED_ADMIN_EMAIL (property et.security.seed-admin-email; Flyway placeholder seed_admin_email) — replaces ET_SECURITY_BOOTSTRAP_ADMIN_EMAIL, which F5-k retires |
the email of the IT Admin user the F5-k migration seeds: crm01@empire.lk on dev and UAT (a plain value); in production the address Empire confirms, secrets referenceforge-et-be .aws/td.prod.json, in git once confirmed — F38-i decision 23) (§B F5-k) |
@NotBlank @Email on EtProperties.Security.seedAdminEmail: unset or malformed stops startup naming the key before any migration runs; the migration refuses a bad seed_admin_email too (F5-k FR-8, AD #34) |
ET_FILES_STORAGE_BUCKET |
attachments bucket Dev (F38-b WI-2.1, 2026-10-21): et-dev-attachments — the forge-et-iaac output attachments_bucket_name; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-9. |
@NotBlank |
ET_MAILBOX_PUBLIC_BASE_URL, ET_MAILBOX_TOKEN_KEY_ID |
Graph webhook host; KMS key Dev (F38-b WI-2.1, 2026-10-21): ET_MAILBOX_TOKEN_KEY_ID = ce1dbe4f-711e-4dd3-ab6e-9ce1930e9040 — the forge-et-iaac output mailbox_token_key_id (ET_MAILBOX_PUBLIC_BASE_URL is F38-d’s); evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-9. |
MailboxCloudConfig refuses to boot |
ET_MESSAGING_SCHEDULER_QUEUE |
scheduler consumer (subscription renewal, delta poll) — required | SchedulerConfigGuard refuses to boot (F20-b FR-6b, WI-2.1) |
ET_MESSAGING_SCAN_RESULT_QUEUE |
the scan-result consumer (GuardDuty verdicts via EventBridge) — required unless ET_FILES_SCAN_PROVIDER=none (scanning switched off, F38-j FR-8), when it is optional and, if set, the listener still applies results |
ScanResultQueueGuard refuses to boot (F38-c FR-3); a blank value is refused too |
ET_MESSAGING_MAILBOX_INGEST_QUEUE |
mailbox ingest consumer and producer — without it no mail is ingested, silently (one WARN per dropped job) Dev (F38-b WI-2.1, 2026-10-21): et-dev-mailbox-ingest — the forge-et-iaac output mailbox_ingest_queue_name; evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-9. |
F20-b WI-1.1 plan D18; MailboxIngestJobListener @ConditionalOnProperty |
ET_MESSAGING_LEADTRIAGEQUEUE |
the lead-triage queue, consumer and producer (F11-d) — without it no lead is triaged: a create logs one Lead Triage Not Enqueued WARN and the hourly sweep one Lead Triage Sweep Skipped WARN. ET_MESSAGING_LEADTRIAGEDLQ is read only by the local provisioner and stays unset in cloud, where the DLQ is Terraform’s |
F11-d WI-1.1 plan Decision 19; LeadTriageJobListener @ConditionalOnProperty; no startup guard, so a task definition without it starts (§B F11-d row 1). Dev: et-dev-lead-triage, set by forge-et-be#283 (2026-10-20) |
ET_MESSAGING_LEADRANKINGQUEUE |
the lead-ranking queue, consumer and producer (F11-g) — without it no lead is ranked: a research write logs one Lead Ranking Not Enqueued WARN and the hourly sweep one Lead Ranking Sweep Skipped WARN. ET_MESSAGING_LEADRANKINGDLQ is read only by the local provisioner and stays unset in cloud, where the DLQ is Terraform’s |
F11-g WI-1.1 plan Decision 28; LeadRankingJobListener @ConditionalOnExpression; no startup guard, so a task definition without it starts (LeadRankingCloudShapeIT; §B F11-g row 1) |
ET_MESSAGING_LEADRESEARCHQUEUE |
the lead-research queue, consumer and producer (F11-c) — without it no lead is researched: a create logs one Lead Research Not Enqueued WARN, the hourly sweep one Lead Research Sweep Skipped WARN, and the Research panel stays at Researching this lead…. Set it only once the queue exists, because a queue name with no queue fails the deploy. ET_MESSAGING_LEADRESEARCHDLQ is read only by the local provisioner and stays unset in cloud, where the DLQ is Terraform’s — this departs from spec FR-H3 item 2, which puts both keys in the task definition, because the enqueuer and the listener read only the queue key (F11-c WI-1.3 plan Decision 7) |
LeadResearchJobListener exists only with the key set; no startup guard, so a task definition without it starts (LeadResearchCloudShapeIT; §B F11-c item 2). Dev: et-dev-lead-research, set by forge-et-be#307 (2026-10-23) |
ET_MESSAGING_PLANRESEARCHQUEUE |
the plan-research queue, consumer and producer (F36-c, from its wave 1) — without it no exhibitor on a show plan is researched or ranked, and each reads researching. Set it only once the queue exists, because a queue name with no queue fails the deploy. ET_MESSAGING_PLANRESEARCHDLQ is read only by the local provisioner; FR-CG1 (a) names it for the task definition, where it is unused but harmless (the DLQ is Terraform’s; F36-c-WI-1.1 key table) |
F36-c spec FR-JB1, FR-CG1 (a); §B F36-c item (a) |
ET_MESSAGING_CAPTURERESEARCHQUEUE |
The capture-research queue (F36-c wave 2), consumer and producer — without it no captured lead is researched or ranked, each reads researching, and no Ready to review notification is written. Set it only once the queue exists. ET_MESSAGING_CAPTURERESEARCHDLQ is read only by the local provisioner; FR-CG1 (a) names it for the task definition, where it is unused but harmless (the DLQ is Terraform’s; F36-c-WI-1.1 key table) |
F36-c spec FR-JB1, FR-CG1 (a); §B F36-c item (a) |
ET_MESSAGING_CARDREADQUEUE (property et.messaging.card-read-queue) |
the card-read queue, consumer and producer (F36-b) — without it no card is read: a card capture is not queued (one WARN line) and its lead’s page stays at Reading the card…. A plain value (AD #40); set it only once the queue exists. A dead-letter key, if F36-b-WI-2.1 adds one, is read only by the local provisioner and stays unset in cloud, where the DLQ is Terraform’s. Added by F36-b-WI-2.1 (wave 2) | the listener exists only with the key set (F36-b FR-R3); no startup guard, so a task definition without it starts (§B F36-b item (a)) |
ET_SEARCH_EXA_APIKEY — a task-definition secrets reference to Secrets Manager, never a plain environment value (AD #22) |
the Exa key the web search uses (F11-c). Without it the backend starts, logs one startup ERROR line naming the variable, and every search answers NOT_CONFIGURED, so research stays not available yet and the sweep queues nothing (spec Rev 1) |
F11-c spec FR-Q2, NFR-6; §B F11-c item 1. Never logged; SearchProperties.Exa#toString prints only whether it is set. Dev: from the secret et-dev-exa-api-key (forge-et-iaac#8), referenced by forge-et-be#307 (2026-10-23) |
LOGGING_STRUCTURED_FORMAT_CONSOLE=ecs (+ LOGGING_LEVEL_* as wanted) |
JSON logs for CloudWatch | FS-5 Decision 3 |
Backend — optional, set only to switch scanning off (F38-j; the order of the steps is
forge-et-be/docs/messaging/scan-result-cloud-contract.md § “Switching scanning off and back on
(F38-j)”). Left out, scanning is on. Never set either to an empty value.
| Variable | Purpose | Enforced by |
|---|---|---|
ET_FILES_SCAN_PROVIDER |
none switches malware scanning off: uploads are stored NOT_SCANNED and are downloadable without a scan; left out (or exactly guardduty), GuardDuty scans |
ScanSettingGuard refuses any other value, a blank one included, naming the key (F38-j FR-1) |
ET_FILES_SCAN_OFF_ACKNOWLEDGED |
true, set only together with ET_FILES_SCAN_PROVIDER=none |
ScanSettingGuard refuses none without it in every deployed environment, naming both keys (F38-j FR-9) |
Backend — optional overrides (F12; §B F12 row 7; the pacing rows F42, §B F42 item 4; the background route and the six rows F43, §B F43 item 5). application.yml
carries every required et.ai value — each feature’s model, its price row and its pacing row — so a
task definition sets none of these while the environment’s one backend task is the only one drawing on
the account’s quota; each overrides one value. The full key list is forge-et-be/docs/ai/README.md
§ Configuration. Setting any ET_AI_PRICES_<n>_* replaces the whole price list from
application.yml (Spring’s list rule), so an override states every row, the inquiry model’s included.
Setting any ET_AI_PACING_LIMITS_<n>_* replaces the whole pacing list the same way, so it restates
the row of every model a feature uses: startup refuses a feature whose model has no pacing row.
| Variable | Purpose | Name proven? |
|---|---|---|
ET_AI_FEATURES_INQUIRYEXTRACT_MODEL |
the inquiry extraction model, a Bedrock inference-profile id; startup refuses one with no price row | proven (AiPropertiesBindingSpec, F12 AC-2) |
ET_AI_FEATURES_INQUIRYEXTRACT_ENABLED |
false switches inquiry extraction off — every call DISABLED, the manual path |
proven (AiPropertiesBindingSpec, F12 AC-2) |
ET_AI_FEATURES_EMAILDRAFT_MODEL |
the email draft model (F11-h). Set on dev to us.anthropic.claude-haiku-4-5-20251001-v1:0 (.aws/td.dev.json, forge-et-be#280) — and in every environment while the account has Bedrock’s new-account limits (UAT by F38-h, production by F38-i; the lead, 2026-10-26, ai-provider-strategy.md §5 row 10) — because the default, Sonnet 5.5’s Global profile, is refused by the account (§B F11-h item 7); removed once §B F11-h item 5 passes and the lead moves drafts off Haiku 4.5 on the evaluation’s result (ai-provider-strategy.md §4d) |
proven (AiPropertiesBindingSpec, F11-h AC-7, set to the Haiku 4.5 id) |
ET_AI_FEATURES_EMAILDRAFT_ENABLED |
false switches drafting off — every draft DISABLED, 503 AI_DRAFT_UNAVAILABLE, the person writes the email (F11-h) |
proven (AiPropertiesBindingSpec, F11-h AC-7) |
ET_AI_FEATURES_ACCOUNTBRIEFING_MODEL |
the account briefing model (F13). Set on dev to us.anthropic.claude-haiku-4-5-20251001-v1:0 (.aws/td.dev.json, merged in forge-et-be#296 on 2026-10-22) — and in every environment while the account has Bedrock’s new-account limits (UAT by F38-h, production by F38-i; the lead, 2026-10-26, ai-provider-strategy.md §5 row 10) — because the default, Sonnet 5.5’s Global profile, is refused by the account (§B F13 item 4); removed once Sonnet 5.5 answers a brief on dev and the lead moves the briefing off Haiku 4.5 on its evaluation’s result (ai-provider-strategy.md §4d) |
proven (AiPropertiesBindingSpec, F13 AC-1, set to the Haiku 4.5 id) |
ET_AI_FEATURES_ACCOUNTBRIEFING_ENABLED |
false switches the account briefing off — every brief of an account with timeline entries DISABLED, 503 AI_BRIEFING_UNAVAILABLE, the person reads the account themselves (F13) |
proven (AiPropertiesBindingSpec, F13 AC-1) |
ET_LEADS_COLDAFTERDAYS |
the calendar days after the buyer’s last message beyond which a lead is cold and a draft re-engages (default 30; F11-h spec FR-D3; F11-g reads the same key); startup refuses a value below 1 | proven (LeadPropertiesBindingSpec) |
ET_AI_PRICES_<n>_MODEL, ET_AI_PRICES_<n>_INPUTUSDPERMILLIONTOKENS, ET_AI_PRICES_<n>_OUTPUTUSDPERMILLIONTOKENS |
a model’s price per million input and output tokens | _0_ bound in AiPropertiesBindingSpec beside the AC-2 names; other indexes derived from the relaxed-binding rule — prove on dev before relying on them |
ET_AI_CIRCUITBREAKER_FAILURETHRESHOLD |
failed calls in the window that open the circuit (default 5) | bound in AiPropertiesBindingSpec’s refusal row; not proven on a running task |
ET_AI_CALL_DEADLINE |
the most one call may take (default 9s) |
derived from the relaxed-binding rule and not yet proven by a test — prove on dev before relying on it |
ET_AI_PACING_LIMITS_<n>_MODEL, ET_AI_PACING_LIMITS_<n>_REQUESTSPERMINUTE |
one pacing row: a Bedrock inference-profile id and this task’s share of the account’s requests-per-minute quota for it — the quota divided by the backend tasks drawing on it, rounded down (F42 FR-P3). Two exceptions: a model no feature uses and whose quota is 0 (the four rows of Sonnet 5.5 and Haiku 5.5 in dev, and expected in production, F38-i W-6) keeps a placeholder row of at least 1, not quota ÷ tasks, and no feature is moved to a model whose quota is 0; and where a model’s share is at or below the background reserve (2 by default), the task definition also sets ET_AI_PACING_BACKGROUNDRESERVE below every row, or startup refuses. application.yml carries six rows since F43 (FR-R1), in this order: Haiku 4.5’s us. and Global profiles, Sonnet 5.5’s us. and Global profiles, Haiku 5.5’s us. and Global profiles, each at 10 — the two Haiku 4.5 rows dev’s share with one task of L-CCA5DF70 and L-E5084BBA, the four 5.5 rows placeholders — so dev sets none; set them where a share differs (§B F42 item 4), restating all six rows. Where an environment runs more than one backend task, so that a profile’s per-task row less ET_AI_PACING_BACKGROUNDRESERVE is not above the background callers that can wait at once on that profile (the sum of ET_LEADTRIAGE_MAXCONCURRENT and ET_LEADRESEARCH_MAXCONCURRENT for the profile lead triage and lead research use; two tasks on a quota of 10: a row of 5, a share of 3, against 4 waiting callers), its task definition also lowers those two values until their sum is below the share, or equal to it when every such wait is at least 61 s, or the environment accepts that some background calls are handed back to the hourly sweep (F43 FR-R3, FR-D5, decision 2; forge-et-be docs/ai/README.md § Pacing). Startup refuses a row below 1 or a repeated model |
_0_ and _1_ proven (AiPropertiesBindingSpec, F42 AC-19); higher indexes derived from the relaxed-binding rule — prove on dev before relying on them |
ET_AI_PACING_BACKGROUNDRESERVE |
slots a minute kept for calls a person waits on (default 2): lead triage and lead research take a slot only while more than this many are free; startup refuses a value below 0 or not below every row | proven (AiPropertiesBindingSpec, F42 AC-19) |
ET_AI_PACING_INTERACTIVEMAXWAIT |
the longest an inquiry extraction, email draft or account briefing waits for its first slot (default 3s), inside its ET_AI_CALL_DEADLINE; startup refuses one not below the deadline |
proven (AiPropertiesBindingSpec, F42 AC-19) |
ET_AI_PACING_BACKGROUNDMAXWAIT_LEADTRIAGE |
the longest a lead triage call waits for its first slot (default 60s since F43 FR-D1, from 15s) before the lead is handed back to the hourly sweep; in a cloud shape startup refuses one that, added to ET_AI_CALL_DEADLINE, is not at least 60 s below ET_LEADTRIAGE_VISIBILITYTIMEOUT (60 s + 9 s + 60 s = 129 s ≤ 150 s at the defaults, so a wait of up to 81 s is accepted; F43 FR-D3) |
proven (AiPropertiesBindingSpec, F42 AC-19) |
ET_AI_PACING_BACKGROUNDMAXWAIT_LEADRESEARCH |
the longest a lead research call waits for its first slot (default 60s) before the lead is handed back; startup refuses more than 70s, which keeps the research run inside its 270 s visibility timeout with F42’s 60 s margin, now that F36-c’s plan-research searcher shares the search rate limiter (F42 FR-W2; F36-c WI-1.2 plan Rev 1) |
proven (AiPropertiesBindingSpec, F42 AC-19) |
ET_AI_FEATURES_LEADTRIAGE_MODEL |
the lead triage model (F11-d), a Bedrock inference-profile id; startup refuses one with no price row. Set to Haiku 4.5’s Global profile, global.anthropic.claude-haiku-4-5-20251001-v1:0, so that background work has a request quota of its own (F43 FR-R3, the lead’s direction of 2026-11-02): dev in .aws/td.dev.json (F43-WI-4.1); UAT the same, from this row when F38-h writes its task definition (UAT runs in the development account, whose Global Haiku 4.5 quota L-E5084BBA is 10 a minute); production the same if its account’s Global Haiku 4.5 quota is above 0, and otherwise left unset, so application.yml’s default, the us. profile, applies (F38-i, from its quota reads; .forge/owed.md, “Scan production’s Bedrock setup …”). The calls a person waits for stay on the us. profile. The Global profile may process a request in any commercial AWS Region, which is accepted (OQ-15, settled 2026-11-04); moving back is this value |
proven (AiPropertiesBindingSpec, F11-d AC-9; each of the six profiles accepted at startup, F43 AC-12) |
ET_AI_FEATURES_LEADTRIAGE_ENABLED |
false switches the triage’s AI part off — every lead is written unavailable with its duplicate and match, and the sweep queues no lead for its AI part |
proven (AiPropertiesBindingSpec, F11-d AC-9) |
ET_AI_FEATURES_LEADRANKING_MODEL |
the lead ranking model (F11-g), a Bedrock inference-profile id; startup refuses one with no price row. Set to Haiku 4.5’s Global profile, global.anthropic.claude-haiku-4-5-20251001-v1:0, as ET_AI_FEATURES_LEADTRIAGE_MODEL is, on dev, UAT and (where its account’s Global Haiku 4.5 quota is above 0) production; application.yml’s default is the us. profile |
proven (AiPropertiesBindingSpec, F11-g AC-11; each of the six profiles accepted at startup, AiPropertiesCloudShapeIT) |
ET_AI_FEATURES_LEADRANKING_ENABLED |
false switches the ranking’s AI part off — every queued lead is written unavailable / DISABLED with no attempt counted, and the sweep queues only leads with no row or a row for other inputs |
proven (AiPropertiesBindingSpec, F11-g AC-11) |
ET_AI_PACING_BACKGROUNDMAXWAIT_LEADRANKING |
optional: the longest a lead-ranking call waits for a slot (default 60 s); with et.ai.call.deadline and 60 s it must fit et.lead-ranking.visibility-timeout, or a cloud shape refuses to start |
proven (AiPropertiesBindingSpec; AiPacingCloudConfigSpec, F11-g AC-10) |
ET_AI_FEATURES_LEADRESEARCH_MODEL |
the lead research model (F11-c), a Bedrock inference-profile id; startup refuses one with no price row. Set to Haiku 4.5’s Global profile, global.anthropic.claude-haiku-4-5-20251001-v1:0, in every environment while the account has Bedrock’s new-account limits — Haiku 4.5 because the default, Sonnet 5.5’s Global profile, is refused by the account (§B F11-c item 6; the lead, 2026-10-26, ai-provider-strategy.md §5 row 10), and its Global profile so that background work has a request quota of its own (F43 FR-R3, the lead’s direction of 2026-11-02): dev in .aws/td.dev.json (set to the us. profile by forge-et-be#307 on 2026-10-23, moved to Global by F43-WI-4.1); UAT from this row (F38-h); production the same if its Global Haiku 4.5 quota is above 0, and otherwise us.anthropic.claude-haiku-4-5-20251001-v1:0 (F38-i); removed once §B F11-c item 7 passes and the lead moves research off Haiku 4.5 on the evaluation’s result (ai-provider-strategy.md §4d) |
proven (AiPropertiesBindingSpec, F11-c AC-34, set to the Haiku 4.5 id) |
ET_AI_FEATURES_LEADRESEARCH_ENABLED |
false switches research off — no search is made, every queued lead is written unavailable / DISABLED with no attempt, and the sweep queues no unavailable lead until it is back on (F11-c FR-J6) |
proven (AiPropertiesBindingSpec, F11-c AC-34) |
ET_LEADRESEARCH_MAXCONCURRENT, ET_LEADRESEARCH_MAXATTEMPTS, ET_LEADRESEARCH_SWEEPBATCH |
the lead-research consumer’s concurrency (2 in application.yml), the counted failed runs before the daily retry (3), and the most leads one sweep queues (50); startup refuses a value below 1; where a profile’s per-task share is not above the two listeners’ summed concurrency, the task definition lowers them (the ET_AI_PACING_LIMITS_<n>_* row, F43 FR-R3) |
proven (LeadResearchPropertiesBindingSpec, F11-c; the concurrency also through the listener’s placeholder) |
ET_RESEARCH_MONTHLYSEARCHLIMIT, ET_RESEARCH_BACKLOGMONTHLYSEARCHLIMIT, ET_RESEARCH_SOCIALHOSTS_<n>, ET_SEARCH_MAXTEXTCHARS, ET_SEARCH_MAXPERSECOND, ET_SEARCH_EXA_READTIMEOUT |
the search limits (F11-c): the general and backlog monthly limits (1,300 and 900 in application.yml), the social hosts (setting any replaces the list), the page-text cap, the rate and the Exa read timeout. Set a low backlog limit on a UAT instance that uses dev’s key, so a UAT migration of the backlog does not spend dev’s credit (.forge/owed.md). The person search is no longer a setting: F11-i’s stored switch (§B F11-i), changed on Settings → Lead research |
proven (SearchPropertiesBindingSpec, F11-c WI-1.1) |
ET_AI_FEATURES_PLANRESEARCH_MODEL |
the exhibitor research model (F36-c plan-research, from its wave 1), a Bedrock inference-profile id; startup refuses one with no price row. Set to Haiku 4.5’s us. profile, us.anthropic.claude-haiku-4-5-20251001-v1:0, in dev, UAT and production while the account has Bedrock’s new-account limits — the default, Sonnet 5.5’s Global profile, is refused by the account, and the us. profile puts the plan’s background calls beside card reading’s rather than the CRM’s lead work (F36-c FR-RS2): dev in .aws/td.dev.json, UAT from this row (F38-h), production the same (F38-i); removed once §B F36-c item (g) passes for research and the lead moves it (ai-provider-strategy.md §4d) |
F36-c spec FR-RS2, FR-CG1 ©, AC-18; §B F36-c item © |
ET_AI_FEATURES_CAPTURERESEARCH_MODEL |
The captured-lead research model (capture-research). Set to Haiku 4.5’s Global profile, global.anthropic.claude-haiku-4-5-20251001-v1:0, in dev, UAT and production while the account has Bedrock’s new-account limits, beside lead triage, research and ranking (F43 FR-R3); the us. profile in an environment whose account has no Global Haiku 4.5 quota. Without it the feature calls Sonnet 5.5, which the account refuses |
F36-c spec FR-RS2, FR-CG1 ©; §B F36-c item © |
ET_AI_FEATURES_CAPTURERANKING_MODEL |
The captured-lead ranking model (capture-ranking; Haiku 4.5’s us. profile in application.yml). Set to Haiku 4.5’s Global profile, as ET_AI_FEATURES_LEADRANKING_MODEL is: dev in .aws/td.dev.json, UAT and production from this row (F38-h, F38-i); the us. profile where the account has no Global Haiku 4.5 quota |
F36-c spec FR-RS2, FR-CG1 ©; §B F36-c item © |
ET_AI_FEATURES_PLANRANKING_MODEL, and for each of PLANRESEARCH, PLANRANKING, CAPTURERESEARCH and CAPTURERANKING its ET_AI_FEATURES_<KEY>_ENABLED and ET_AI_PACING_BACKGROUNDMAXWAIT_<KEY> |
optional: plan-ranking’s model (Haiku 4.5’s us. profile in application.yml, left at that default everywhere); false switches that feature’s AI calls off — with a research feature off no search is made, the research is written unavailable / DISABLED, and the subject is still ranked (F36-c FR-JB3, FR-RK2); each feature’s longest wait for a slot (default 60s), which with the 9 s deadline must fit its listener’s visibility timeout — 300 s for the plan features, 330 s for the capture features (F36-c FR-JB2, spec Rev 3). The CAPTURE* keys ship with F36-c’s wave 2 |
F36-c spec FR-RS2, FR-JB2, FR-JB3 |
ET_PLANRESEARCH_MAXCONCURRENT, ET_PLANRESEARCH_MAXATTEMPTS, ET_PLANRESEARCH_SWEEPBATCH, ET_PLANRESEARCH_VISIBILITYTIMEOUT, ET_RESEARCH_PLANMONTHLYSEARCHLIMIT; from F36-c’s wave 2 the same four ET_CAPTURERESEARCH_* keys |
the plan-research consumer’s concurrency, the counted failures after which an exhibitor reads failed and is retried daily, the most exhibitors one sweep sends, and the visibility timeout its listener asks SQS for (F36-c FR-JB2): 1 / 3 / 200 / 300 s in application.yml, and no task definition overrides them — max-concurrent stays 1 in every environment, inside F43 FR-D5’s per-task share on the us. profile (spec W-8, settled by F36-c-WI-1.2 Decision 11); the exhibitor searches’ monthly limit (1,100, inside the general 1,300 — F36-c FR-RS3). Wave 2 (ships with F36-c’s wave 2): the ET_CAPTURERESEARCH_* keys, 1 / 3 / 50 / 330 s in application.yml, and no task definition overrides them — max-concurrent stays 1 in every environment, inside F43 FR-D5’s per-task share on the Global profile (F36-c-WI-2.2 Decision 15); the visibility timeout is 330 s because the captured-lead run’s sum, with the person search on, is 300.4 s (spec Rev 3) |
F36-c spec FR-JB1, FR-JB2, FR-JB5, FR-RS3, Rev 3; W-8; F36-c-WI-1.2 Decision 11; F36-c-WI-2.1 key table, WI-2.2 Decision 15 |
ET_LEADTRIAGE_MAXCONCURRENT, ET_LEADTRIAGE_MAXATTEMPTS, ET_LEADTRIAGE_SWEEPBATCH |
the lead-triage consumer’s concurrency (default 2), the failed attempts before the daily retry (3), and the most leads one sweep queues (200); where a profile’s per-task share is not above the two listeners’ summed concurrency, the task definition lowers them (the ET_AI_PACING_LIMITS_<n>_* row, F43 FR-R3) |
proven (LeadTriagePropertiesBindingSpec, F11-d; the concurrency also through the listener’s placeholder) |
ET_LEADRANKING_MAXCONCURRENT, ET_LEADRANKING_MAXATTEMPTS, ET_LEADRANKING_SWEEPBATCH, ET_LEADRANKING_VISIBILITYTIMEOUT, ET_LEADRANKING_RESEARCHWAIT |
optional: the lead-ranking consumer’s concurrency (default 2), the failed attempts before the daily retry (3), the most leads one sweep queues (200), the receive visibility (150 s) and the research wait (24 h); where a profile’s per-task share is not above the three background listeners’ summed concurrency, the task definition lowers the concurrency (the ET_AI_PACING_LIMITS_<n>_* row, F43 FR-R3) |
proven (LeadRankingPropertiesBindingSpec, F11-g; the concurrency also through the listener’s placeholder) |
ET_LEADTRIAGE_VISIBILITYTIMEOUT |
the visibility timeout the lead-triage listener asks SQS for on every receive (default 150s, et.lead-triage.visibility-timeout, F43 FR-D2), in place of ET_MESSAGING_VISIBILITYTIMEOUT, which overrides the triage queue’s own visibility in every environment; startup refuses a value below 1 s, above 12 hours or not whole seconds, and in a cloud shape one that is not at least 60 s above ET_AI_PACING_BACKGROUNDMAXWAIT_LEADTRIAGE plus ET_AI_CALL_DEADLINE (129 s ≤ 150 s at the defaults; F43 FR-D3). No environment sets it while the triage wait and the deadline are at their defaults |
proven (LeadTriagePropertiesBindingSpec, F43 AC-1) |
ET_AI_FEATURES_CARDREAD_MODEL |
the card reading model (F36-b), a Bedrock inference-profile id; startup refuses one with no price row. Set in no environment: card reading runs on Haiku 4.5’s us. profile, application.yml’s default, as a background call (F36-b decision D-3); moving it to the Global profile is a configuration change (ai-provider-strategy.md §4d) |
proven (AiPropertiesBindingSpec, F36-b AC-35; F36-b-WI-1.1) |
ET_AI_FEATURES_CARDREAD_ENABLED |
false switches card reading off — every queued read ends not_read with the reason DISABLED and no model request, Read Again is not offered, and the rep fills the fields with Edit Details (F36-b FR-A7) |
proven (AiPropertiesBindingSpec, F36-b AC-35; F36-b-WI-1.1) |
ET_AI_PACING_BACKGROUNDMAXWAIT_CARDREAD |
optional: the longest a card-read call waits for a slot (default 60 s); with et.ai.call.deadline and 60 s it must fit et.card-read.visibility-timeout, or a cloud shape refuses to start |
proven (AiPropertiesBindingSpec, F36-b AC-8; F36-b-WI-1.1) |
ET_CARDREAD_VISIBILITYTIMEOUT, ET_CARDREAD_MAXCONCURRENT, ET_CARDREAD_MAXATTEMPTS |
optional: the visibility timeout the card-read listener asks SQS for on every receive (default 150s, et.card-read.visibility-timeout; a cloud shape refuses one that is not at least 60 s above ET_AI_PACING_BACKGROUNDMAXWAIT_CARDREAD plus ET_AI_CALL_DEADLINE, F43 FR-D3), the consumer’s concurrency (default 2) and the attempts before a read ends not_read with UNAVAILABLE (default 3, F36-b FR-R6); where a profile’s per-task share is not above the background listeners’ summed concurrency, the task definition lowers the concurrency (F43 FR-D5). An environment that routes all three of lead triage, research and ranking to Haiku 4.5’s us. profile sets ET_CARDREAD_MAXCONCURRENT=1: card reading runs on that profile, and triage 2 + research 2 + ranking 2 + card read 2 = 8 background callers with 60 s waits would break F43 FR-D5’s rule; with application.yml’s routes (6 on us.) and with dev’s Global route for the three (card read alone on us.) the default of 2 fits (F36-b-WI-2.1 plan § Approach) |
proven (CardReadPropertiesBindingSpec: the visibility timeout by F36-b-WI-1.1; the concurrency, the attempts and a 0 refused naming its key by F36-b-WI-2.1) |
ET_MESSAGING_VISIBILITYTIMEOUT |
the shared visibility timeout (default 30s, EtProperties.Messaging): QueueProvisioner creates the local queues with it under local and test, and the mailbox jobs read it (the delta poll, renewal and backfill take half of it as their time budget, and MailboxIngestEnqueuer its redelivery backoff). Since F43 the lead-triage listener asks SQS on every receive for ET_LEADTRIAGE_VISIBILITYTIMEOUT instead, which overrides the queue’s own visibility in every environment, and the lead-research listener keeps its fixed 270 s. Since F43 no startup check reads this key: the whole-seconds checks and the cloud-shape check against the triage wait apply to ET_LEADTRIAGE_VISIBILITYTIMEOUT (F43 FR-D2, FR-D3; F43-WI-1.1 plan) |
derived from the relaxed-binding rule and not yet proven by a test — prove on dev before relying on it |
The concrete per-environment values for the CORS list, login-redirect-uri, the CSRF cookie
domain and name, and each frontend’s VITE_API_BASE_URL — with which are backend (task
definition) and which are frontend (GitHub Actions build variables) — are tabled once, in
.forge/specs/F5-j-cross-site-csrf-cookie-spec.md § “Where each value is set”.
Backend — must stay unset in cloud: et.files.storage.endpoint-override,
spring.cloud.aws.sqs.endpoint, spring.cloud.aws.credentials.* (the task role supplies
credentials), et.messaging.demo-queue, et.messaging.demo-dlq, et.messaging.mailbox-ingest-dlq
(the ingest DLQ is Terraform’s; the key only tells the local QueueProvisioner what to create),
spring.cloud.aws.sqs.listener.auto-startup (the cloud-shape test contexts set it false; in a task
definition that would leave every listener — the scheduler’s included — present and never polling;
SchedulerConfigGuard refuses it unless et.environment is test (BUG-070), and the scan-result
listener is not yet guarded (BUG-097)). Region is fixed to us-east-1
in application.yml. bootRun defaults the local profile; the deployed jar must not.
Outside app config: Entra reply URLs per host for /login/oauth2/code/microsoft and
/authorized/microsoft-mailbox (platform Web) and the certificate (prod’s own; dev and UAT
share the non-prod one) — both registered and uploaded on the CRM registration 2026-09-24; custom
domains under empire.lk; ALB health check on /actuator/health/liveness; the KMS key policy;
the scheduler.amazonaws.com role.
forge-et-fe (build-time, public): VITE_API_BASE_URL — the environment’s api.* address
(AD #33), agreeing with the backend CORS list and login-redirect-uri; and VITE_CSRF_COOKIE_NAME,
the CSRF cookie name for that environment (F5-j), equal to the backend’s
ET_SECURITY_CSRF_COOKIE_NAME. vite build refuses a build whose VITE_API_BASE_URL is a cloud
(https, non-localhost) address without it.
forge-et-pwa (build-time): VITE_API_BASE_URL is in .env.example but not yet declared in
src/env.d.ts or read; no other variables yet. When F36-a builds the capture app’s API client it
reads the same VITE_CSRF_COOKIE_NAME, with the same per-environment value as the CRM’s. The manifest and service worker assume the app is
served from / on its own HTTPS origin.
E. Stale statements in code to correct when the cloud halves land
Comments and examples that were true when written and are now behind the code. Fix them in the PR that validates the related row, so the code stops describing a state that no longer exists.
| Location | What it says | What is true now |
|---|---|---|
forge-et-be mailbox/MailboxSeamGuard.java 21–24 |
Already corrected by forge-et-be#191 (merged 2026-10-05): the Javadoc says the Graph adapter and the KMS key provider satisfy both seams in a cloud shape, so the guard is quiet | |
forge-et-fe src/features/email/startMailboxConnect.ts 17–25 |
404/401” |
Corrected by forge-et-fe#340 (merged 2026-10-26): the comment names the backend’s registration and callback (F20-a WI-2.1); the Wave 1 / Wave 2 sentence is gone |
forge-et-fe .env.example 10–11 vs vite.config.ts 38–39 |
Corrected by F5-j (2026-09-27, forge-et-fe#59, forge-et-be#78): .env.example now says the deployed setup is cross-site and documents VITE_CSRF_COOKIE_NAME; forge-et-be/docs/auth/README.md states AD #33’s choice in place of the two options |
|
forge-et-fe src/lib/api/apiClient.ts 11–12 (doc comment) |
Corrected by F5-j (2026-09-27, forge-et-fe#59): the comments name AD #33 and a configurable cookie name defaulting to XSRF-TOKEN |
|
forge-et-be pr-validation.yml 162–164 |
Corrected by forge-et-be#312: the comment says the wrapper runs code on the runner with the job’s read-only GitHub token in reach; neither pr-validation.yml nor build.yaml’s Gradle job holds AWS credentials |
|
forge-et-fe .github/workflows/pr-validation.yml 298–299, 303; docs/adding-a-feature.md 123 |
TRIVY_GATE_EXIT_CODE=0 “flips to 1 once be, fe and pwa all read 0 HIGH/CRITICAL” |
Corrected by forge-et-fe#111 (merged 2026-10-02): the comments, the guard-table row, .trivyignore.yaml and CLAUDE.md state AD #38 — the value stays 0 permanently, and the UAT and production release check (F38 FR-35, not built yet) will stop on an open finding. The forge-et-be copies are corrected in forge-et-be PR #145 |
forge-et-pwa .github/workflows/pr-validation.yml 290–291, 295 |
Corrected by forge-et-pwa#52 (merged 2026-10-02), as the forge-et-fe row |
|
forge-et-be mailbox/SubscriptionStatus.java 6–9 (class comment) |
Corrected by forge-et-be#312: the class comment says a FAILED subscription is repaired by the user re-consenting (F20-b WI-2.1 plan D14) |
|
forge-et-fe src/app/telemetry.ts 6–8, 16 (doc comment and body comment) |
Corrected by forge-et-fe#340 (merged 2026-10-26): V1 sends errors nowhere, by decision; reportError stays empty |
|
forge-et-pwa src/app/telemetry.ts 6–8, 16 (doc comment and body comment) |
Corrected by forge-et-pwa#93, a byte copy of forge-et-fe#340’s file, its hash recorded from e9def12 |
|
forge-et-be attachment/service/AttachmentMetrics.java 12–13, 47; docs/messaging/scan-result-cloud-contract.md 49, 125–131 |
Corrected by forge-et-be#312 | |
forge-et-be mailbox/service/MailboxMetrics.java 15–16, 47–48, 52, 61, 70; mailbox/service/MailboxRenewalService.java 69; src/test/groovy/com/empireteas/mailbox/MailboxObservabilityIT.groovy 32, 48; MailboxRenewalIT.groovy 41 |
MailboxRenewalService); “the meter name the FS-5 cloud half alarms on”; “the level the lapsed-subscription alarm reads”; the ITs’ “the FS-5 cloud half alarms on these”, “CloudWatch alarms on the gauge” and the CloudWatch alarm they cannot prove |
Corrected by forge-et-be#312: the comments say V1 exports no meter, and that a mailbox alarm would read the state-change log lines |
forge-et-fe CLAUDE.md 274; forge-et-pwa CLAUDE.md 231 (instruction text every session in those repos loads) |
Corrected by forge-et-fe#340 (merged 2026-10-26) and forge-et-pwa#93: “V1 sends errors nowhere, by decision” | |
forge-et-be ai/service/AiCircuitBreaker.java 27–28; docs/ai/README.md 179–181; docs/ai/bedrock-cloud-contract.md 36–39 |
MODEL_UNAVAILABLE “read” the meters; both alarms “deferred to dev” |
Corrected by forge-et-be#312: dev builds no alarm on the circuit; UAT and production decide, production must alarm on MODEL_UNAVAILABLE, reading the log lines |
forge-et-be attachment/service/ScanResultHandler.java 23 |
Corrected by forge-et-be#312 | |
forge-et-be platform/scan/GuardDutyFileScanProvider.java 25 |
Corrected by forge-et-be#312: an EICAR upload once scanning is switched on is the proof; the alarm was withdrawn with GuardDuty | |
forge-et-be docs/messaging/scan-result-cloud-contract.md 260; docs/messaging/README.md 87; docs/messaging/scheduler-cloud-contract.md 228 |
Corrected by forge-et-be#312, which also corrects the same documents’ claims that the scheduler, the queues and the DLQ are deferred (dev has them, forge-et-iaac base/scheduler, base/sqs) and that GuardDuty’s proof is F38-f’s on dev |
|
forge-et-be src/main/resources/application.yml 403–409 (the management.tracing comment) |
Corrected by forge-et-be#312 | |
forge-et-be attachment/service/SkippedScanSeries.java 10–11; attachment/service/ScanResultHandler.java 56–60 (found by F38-e WI-1.2’s re-run of the FR-9 searches, 2026-10-23, at 2efd29cd) |
AttachmentMetrics’ other series”; the INFECTED ERROR line is “a security signal for an alarm” that “alerts exactly once”, with ids in the MDC “so an alarm or a test reads them” |
Corrected by forge-et-be#312, with the two MDC-key comments (ScanResultHandler, ScanResultEventConsumer) that called their fields “for the alarm family” |
Log
-
2026-09-18 — Register created (Tharinda’s request during F5-a delivery). Seeded from the F5-a spec/plans (Entra handshake, cross-host cookies/CSRF/CORS, first-admin bootstrap on a fresh env, E2E
storageState) and the documented engagement cloud halves (FS-1, FS-5, Bedrock, CI/deploy, Trivy flip). Append per feature as more cloud-only work is deferred. -
2026-09-22 — F20-a (mailbox connection) appended six rows from
.forge/specs/F20-a-mailbox-connection-spec.mdFR-16, via WI-2.2: five to §1 (theauthorized/microsoft-mailboxreply URL per environment, the real consent round trip, a live refresh and a forcedinvalid_grant, Graph subscription creation against a reachable callback, and the KMS-wrapped token keys) and one to §3 (the customer-managed KMS key FS-1 must provision, which also carries spec OQ-C’sET_MAILBOX_PUBLIC_BASE_URLon the task definition). All six are cloud-gated for the same structural reason spec D-2 states: nothing in F20-a may require a live AWS account, a provisioned key or a real Entra exchange, so every one of them is faked at its boundary offline and none is proved. -
2026-09-23 — Rebuilt as the single reference (Dinil, at the team’s request). Three read-only sweeps — every foundation spec/plan/findings file, every feature spec/plan, and the three app repos’ code and CI — found the items the register was missing. Changes: (1) the file is now organised by owner (§B, one block per slice or feature, with a recorded “nothing cloud-gated” list) so a walk answers “what does FS-7 still owe?” in one place; the old §1–§6 capability sections become §C, which points at §B and keeps only engagement-level rows — the F5-a and F20-a rows moved from §1–§3 into their §B blocks verbatim, with the new detail folded in. (2) A new §A names the four things that block a first deploy outright — the FS-11
SeamGuardboot refusal, the absent Terraform, the F20-b Wave-1-alone hazard, and the unmade cross-host cookie decision. (3) New rows: FS-1 (9), FS-2 (6), FS-4 (5, incl. the Trivy flip re-described as an ordering constraint rather than cloud-gated, per FS-4 Rev 3), FS-5 (8), FS-6 (12), FS-7 (4), FS-9 (6), FS-11 (8), F5-a (7), F20-a (11), F20-b (10 pre-registered from the approved spec’s FR-13), and a backlog table of 14 features whose stub specs already name a cloud dependency. (4) §D is the task-definition checklist derived from the code; §E lists stale code comments to fix when the related row is validated. (5) The keep-current rule is now enforced, not just stated: CLAUDE.md § Boundaries, the spec/plan templates,spec-reviewer§7,plan-reviewer§6,/forge-pre-pr-reviewand/forge-reflectstep 5b all check that a shipping PR registered its cloud-gated items. -
2026-09-23 — Tharinda’s notes on the pending cloud and identity work. Added A-5 (the AWS accounts do not exist yet; they wait on a billing discussion between Empire Teas and Altrium management, and the Phase 1 UAT window depends on them). Added §C-7: everything built before the dev environment exists is retested end to end in one pass by F38, a new Phase 1 tracked item for the whole cloud bring-up; no per-feature rows are kept, because which features are affected is not known until that day. Added to §C-1 the proposed laptop sign-in, tracked as F5-i (laptop reply URLs, a laptop certificate, an opt-in backend mode), and to §B FS-6 the go-live step that removes those laptop entries. The F20-a mailbox reply-URL row now records that the ask is on the Empire IT agenda; the F21-b, F22 and F36-a backlog rows now say what each means for reply URLs (the capture app signs in like the CRM — lead decision — so it needs none). Detail and the meeting agenda:
.forge/design/environments-and-identity.md§2.2, §2.3, §2.5 and §5. Nothing listed here has been asked, registered, generated or provisioned yet. -
2026-09-23 — Completeness review of every feature spec, foundation slice, the issued proposal and SOW ET-002 against this register. Added: the “Start here” checklist (the order of work from now to go-live); §C-8, the proposal and SOW commitments only a real environment can meet (backups and the restore rehearsal, the security baseline and the pre-go-live security review, log retention and the no-prompt-bodies-in-production-logs rule, budgets, monthly reporting); §C-3 rows for the DNS records Empire controls and database-credential rotation; the §C-5 Bedrock model-access row; SES production sending access on the §B FS-5 row; A-5 now names Bedrock and SP-ARCH-004 stage 2; A-1 names F5-f and F38 as the owners of its two halves; a §B F5-c block (the live identity link and the deactivated-user refusal, which its plans call cloud-gated); “carried elsewhere” lines for FS-8, FS-14, F5-i and F38; and backlog rows for F5-f, F5-h, F20-f, F26-c, F26-d, F4-b/F4-d, F23-a/F23-b, F18-b and the AI features on the
AIProviderseam. -
2026-09-23 — Fixes from an independent review of the above. The checklist is reordered and renumbered: the Terraform state bootstrap moves to day one (everything after it is Terraform), SP-ARCH-004 stage 2 moves up to run as soon as Bedrock and the AgentCore Gateway exist, making the browser suites runnable against dev becomes its own step before the retest, and a UAT step adds the UAT certificate, UAT DNS and the
deploy-uatworkflow. F38’s done is now the first UAT round; production and go-live are steps 28–29, outside it. Cross-region backup copy moves from the binding backups row to the client decisions (proposal §A.2.3 prices it; SOW ET-002 §6 excludes it). The A-5 row’s extra cell is fixed and F13 added to its AI list; A-3 is named in the first-deploy step; the intro no longer says the Entra app registration is missing. -
2026-09-23 — Lead decision recorded in §C-5: AI features stay stub-only until F38 (no direct Anthropic API for local development). The client-facing
.forge/client/dependencies.md§1.2 now lists the mailbox return addresses and the proposed laptop testing; checklist step 3 and the Empire IT agenda item 7 say to check it again after the meeting. -
2026-09-23 — F20-b WI-1.1 (mail storage, seam reads, ingest and the inbox API) refined its two pre-registered §B F20-b rows — the real-notification row now names the tests that stand in for it and the counter that must stay still, and the dashboard row names the shipped meter, its
reasontag and the first-walk review the plan records — and added two: theET_MESSAGING_MAILBOX_INGEST_QUEUEtask-definition key with the Terraform ingest queue, DLQ and redrive (plan D18: without the key ingest ships and never runs), and the deployed-API absence ofstub-notify. §D gains the key under “must set” andet.messaging.mailbox-ingest-dlqunder “must stay unset”. Wave 2’s rows (the two schedules, the scheduler key, the lapsed alarm) are WI-2.1’s. (Written on the F20-b wave-1 branch; merged withmainon 2026-09-24.) -
2026-09-24 — The one-origin decision recorded the day before (AD #33) is withdrawn at the lead’s request; A-4 is open again with Dinil as its owner. The A-4 row now states both options and their consequences, and the checklist makes the cookie decision step 3, ahead of the Empire IT asks (step 4), because it decides which reply URLs to ask for. The Azure-access correction (only Kavindu Anthony can change the app registration) stands.
-
2026-09-24 — A-4 decided: AD #33, the backend keeps its
api.*address and the apps call it cross-site (lead sign-off). TheXSRF-TOKENcookie gets a sharedDomainand a per-environment name; the CORS list andVITE_API_BASE_URLare set per environment; the reply URLs stay onapi.*(three mailbox URLs to ask for). The cookie code change is owed. Rows updated: checklist steps 3–4, A-4, §B FS-6 (three rows), §B F5-a, the F36-a backlog row, §C-2, §D, §E. -
2026-09-24 — The AD #33 cookie change is tracked as F5-j (Phase 1). A-4, the §B FS-6 CSRF row, §C-2, §D, checklist steps 3 and 19 and the carried-elsewhere list point at it.
-
2026-09-24 — Certificates: all four (dev, UAT, prod, laptop) are to be generated together before AWS exists and uploaded in one meeting with Kavindu, with the private keys in a restricted vault until each environment’s Secrets Manager exists. Checklist steps 1, 4, 15, 27 and 28 updated; a §B FS-6 row added for certificate expiry.
-
2026-09-24 — Certificates: two pairs replace the four — prod, and one non-prod pair shared by dev, UAT and the developer laptops (Shyamin Ayesh’s proposal, adopted the same day). Shyamin generates both; the non-prod pair was generated and sent on 2026-09-24, the prod pair is to come. Laptop sign-in (F5-i) needs no certificate of its own, so the go-live step removes only the laptop reply URLs, with replacing the non-prod pair as an option. Checklist steps 1, 4, 15, 27 and 28, the §B FS-6 certificate, laptop-removal and expiry rows, the §C-1 laptop row, and the §D “Outside app config” line updated. Detail:
environments-and-identity.md§2.3. -
2026-09-24 — Entra side done. Shyamin Ayesh generated the prod pair; Kavindu Anthony uploaded both certificates (thumbprints and the 23 Sep 2028 expiry in
environments-and-identity.md§2.3, §5) and registered the three mailbox and two laptop reply URLs as Web, confirmed by his portal screenshots. Checklist steps 1 and 4 marked done with what remains (UAT key location, rotation, the permission re-read, the renewal routine, the production administrator’s email); the §B FS-6 certificate, laptop-removal and expiry rows, the §B F20-a reply-URL and certificate rows, the §C-1 laptop row and the §D “Outside app config” line updated. Every row staysopen: each still needs a live check on a deployed environment. -
2026-09-24 — The access-model decisions (AD #34, AD #35, AD #36) and the
info@decision. The first administrator now comes from one Flyway migration that seeds an IT Admin role and user, with the email a per-environment placeholder (crm01@empire.lkon laptops, dev and UAT; the address Empire confirms in production, from Secrets Manager). Added a §B F5-k block (the dev and UAT value, the production value waiting on Kavindu’s confirmation, the unset-placeholder refusal to verify on the first deploy, and the sample data staying out of cloud databases; the checksum question is the F5-k spec’s). The §B F5-a first-administrator row is withdrawn — a new status, defined under Columns — and the §DET_SECURITY_BOOTSTRAP_ADMIN_EMAILrow is replaced by the placeholder variable, whose name the F5-k spec owes. The F20-e backlog row is withdrawn (F20-e left V1;info@arrives by forwarding). “Nothing cloud-gated” rows added for F39 and F5-f; the F5-f and F5-h backlog rows renamed for their re-scoping; a note under §B F5-c records its re-scoping. The §B FS-6 roles row says IT Admin instead of the superadmin; the F5-a round-trip,?login=failedandstorageStaterows mention the pre-created user, the audited sign-in and the seeded admin. Checklist steps 4, 16, 20 and 28 and §C-1 updated. -
2026-09-24 — The F20-b wave-1 PR review (
/forge-review-pron forge-et-be#51) added two §B F20-b rows. The first is the immutable-id row: the fix (Prefer: IdType="ImmutableId") lands in wave 1 by Dinil’s decision, and only a live mailbox proves the ids agree. The second collects four ingest-under-load findings that Dinil deferred to WI-2.1: the fetch inside the claim transaction, the enqueuer’s transaction across SQS sends, limiter refusals spendingmaxReceiveCount, and the back-fill running past the visibility timeout. -
2026-09-24 — F20-b WI-2.1 (scheduling, renewal and lapse detection) refined its pre-registered §B F20-b wave-2 rows: the two schedule rows now name their
jobKind, expression and the FR-D1 register section inscheduler-cloud-contract.md; theET_MESSAGING_SCHEDULER_QUEUErow says the key is required in every cloud shape (SchedulerConfigGuard); the alarm row names the shipped gauge and counter; the renewal and delta rows name the tests that stand in for them. Added: the delta poll’s reach and first-round cost, and the deployed-API absence ofstub-renewal-failure. The §B FS-9 listener row now records the code fix, with the dev tick still owed. §D: the scheduler key’s enforcement, andspring.cloud.aws.sqs.listener.auto-startupunder “must stay unset”. The deferred ingest-under-load row now records all four findings as fixed in code (none accepted), the tests that prove each, and the dev burst still owed. -
2026-09-24 — F20-b WI-2.1 at PR-open: the deferred ingest row’s finding 3 gains the pre-PR review’s follow-up (a refused delivery whose delayed copy cannot be sent is rethrown, not acknowledged). The reason-code migration shipped as
V20__subscription_reason_code.sql(F1-a tookV19). The dev 200-message burst stays owed. -
2026-09-24 — F20-b wave 2: added the §B F20-b row for the deployed Reconnect path (a real re-consent clearing a lapsed subscription), which WI-2.2’s pre-PR review found unregistered. Only the stub path is proved.
-
2026-09-25 — F20-b Reflect: both waves merged, so §A A-3 (do not deploy Wave 1 alone) is closed and the §B F20-b heading reads done. Reflect confirmed every cloud-gated item the feature deferred has a §B row; no row was added or flipped — all stay open until the dev walk.
-
2026-09-25 — BUG-025 (forge-et-be#60): the §B F20-b renewal row now records that a
401or a non-HTTP failure on renewal is retried, not lapsed. On dev, also confirm a real Graph401on renewal leaves the subscriptionACTIVEand the next run succeeds with a fresh token. -
2026-09-26 — F39 WI-2.1 (audit trail, Wave 2): the §B F39 line now reads “Spec confirmed: nothing cloud-gated” (spec FR-14, AC-21) and names where its two cloud proofs live; the §B FS-11 scan-result row gains the check for one
SYSTEMattachment.scan_completedrow labelledattachment-scan, and notes that the future queue worker must pass that job name. No row added or flipped. -
2026-09-26 — F11-a WI-1.1 (lead substrate): added the §B “Nothing cloud-gated” line for F11-a (spec FR-D5, AC-29 (g)). Every requirement of the feature runs on the local Postgres; no row added to any other block.
-
2026-09-27 — F5-k WI-1.1 (catalog, grant migration and seeded admin): the §B F5-k block’s heading drops “stub spec” and its preamble and four rows now name the settled names — property
et.security.seed-admin-email, variableET_SECURITY_SEED_ADMIN_EMAIL, Flyway placeholderseed_admin_email— with the unset-value row reading “the backend refuses to start, naminget.security.seed-admin-email”; the “Not cloud-gated” paragraph records FR-9’s proved answer (a changed value validates and migrates; the seeded user keeps its first email). §D’s row namesET_SECURITY_SEED_ADMIN_EMAILand what enforces it; checklist steps 16 and 28 use the variable name. No row added or flipped — all stay open until the dev walk. -
2026-09-27 — F38-c wave 1 (WI-1.2, GuardDuty scan adapter; carries WI-1.1’s cloud-gated items per its plan Decision 18): §A-1 and “Start here” step 19 now say the scanner half is on
mainand a cloud boot names onlyRecordVisibilityCheckuntil F5-f ships; §B FS-11 row 1 says the scanner is implemented and names what remains (F5-f, F38-f’s dev proof); §B FS-11 row 2 drops “LocalScanDispatcherscans in-process today”, says the consumer (ScanResultQueueListener) is onmainunder job nameattachment-scan, points atforge-et-be/docs/messaging/scan-result-cloud-contract.md, and lists five items to prove on dev (EventBridge delivery with no input transformer, the pre-commit retry under real timing, the deferral’s ERROR line from Spring Cloud AWS’sAbstractMessageProcessingPipelineSinklogger being excluded from F38-e’s alarm filters, the FR-9 rescan procedure, and the protection plan’s service-role, KMS and queue-policy grants); the §B FS-11 task-role S3 row addss3:GetObjectands3:ListBucketfor the consumer’sHeadObjectcheck; the §B FS-11 cloud-shape-test row no longer quotesSeamGuardIT’s old “Cannot prove” line; §D addsET_MESSAGING_SCAN_RESULT_QUEUE. No §E row: nothing stale was left uncorrected. No row flipped — all stay open until the dev walk. -
2026-09-27 — F5-j (cross-site CSRF cookie, AD #33) built: code PRs forge-et-be#78 and forge-et-fe#59. A-4, checklist steps 3, 18 and 19, and §C-2 now say the code is built and only the per-environment values remain. §B FS-6: the CSRF row’s owner is F38 alone; the CORS row’s max-age is done, with the real-preflight check added (FR-12 ©); the forwarded-headers and
Secure-cookies rows name the CSRF cookie by the environment’s name. Added a §B F5-j block with the one new row — both sides set, with matching names (FR-12 (b)) — and removed F5-j from the “Nothing cloud-gated” table. §D namesET_SECURITY_CSRF_COOKIE_DOMAIN,ET_SECURITY_CSRF_COOKIE_NAMEandVITE_CSRF_COOKIE_NAME, and notes that F36-a reads the same variable. The two F5-j rows in §E are closed. -
2026-09-27 — F5-j Reflect: both code PRs merged to
main(forge-et-be#78, forge-et-fe#59); the §B F5-j block heading now says so. No new rows — the three FR-12 items registered at PR-open are the complete set, and all stay open for F38. -
2026-09-27 — F5-i Wave 1 (laptop sign-in) built: the §C-1 laptop row now reads “Built”, naming the how-to
forge-et-be/docs/auth/laptop-sign-in.md, theentra-laptopprofile (withlocal) and theVITE_ENTRA_LAPTOPflag, and the F5-i Wave 1 code PRs (forge-et-be#86, forge-et-fe#65); the live run asCRM01(F5-i-WI-2.1) is still owed, so the row staysopen. The §B “Nothing cloud-gated” F5-i line records what was checked (spec FR-11: no new cloud-gated item). The §B FS-6 go-live removal row is unchanged. -
2026-09-28 — F38-c wave 2 (WI-2.1, no-verdict logging and counters; spec AC-9c): the §B FS-11 “Alarm on
INFECTED/ scan failures” row now names its signals — theAttachment InfectedERROR line, theScan Result Without VerdictWARN line with its MDC fields, andet.attachment.scans.no_verdictwith its four tag values — its owner becomes F38-e, and it points at the export gap. Two §B FS-11 rows added: the no-verdict proof on real GuardDuty (a password-protected PDF, owner F38-f), and the proposed “uploads with no scan results” alarm onet.attachment.uploadsandet.attachment.scans.results(owner F38-e), which records that no meter registry exports the meters yet. No row flipped — all stay open until the dev walk. -
2026-09-28 — F26-a WI-1.1 (price and cost schema): added the §B “Nothing cloud-gated” line for F26-a (spec FR-D3, AC-29). Every Wave 1 requirement runs on the local Postgres; no row added to any other block.
-
2026-09-28 — Entra key material (forge-et-be#95 and forge-et-be#99, lead decisions on the forge-et-be#44 review): the §B FS-6 certificate-expiry row records that the backend now refuses an unreadable, mismatched, expired or not-yet-valid pair on every signing with
EntraKeyMaterialException, and names what is still owed on dev — an F38-e metric filter and alarm on that exception, and a warning about 30 days before the certificate’s end. The row stays open. -
2026-09-28 — F5-a AC-10b ticked once forge-et-be#99 and forge-et-fe#75 merged. §B F5-a gains the
?login=unavailablerow: each half is proved locally on its own, and the full path needs a real sign-in that signs a client assertion, so it is owed on dev (F38). The §B FS-6 certificate-expiry row no longer says nothing logs the fault at ERROR: the sign-in path does since forge-et-be#99. -
2026-09-29 — F40 WI-1.4 (the browser gate’s skill and doc changes): added the §B “Nothing cloud-gated” line for F40 (the spec’s Constraints and the Decomposition Plan). The gate, its CI description check, the hook and the skills run locally and in the existing CI; no row added to any other block.
-
2026-09-29 — F4-a WI-2.1 (the Pipedrive dry run, end to end) added the §B F4-a block with its one row, spec FR-G1: the dry run on dev behind the ALB (request size, 60-second idle timeout), then on production with the production import owner, validated by matching the local run’s summary counts.
-
2026-09-29 — F26-a Wave 2 (spec Rev 3, the developer’s decision at forge-et-be#110): the F26-a line leaves “Nothing cloud-gated” and becomes a §B block with one row — NFR-2’s response times, whose binding measurement moves to dev because two laptop readings each breached one FR-A1 shape by about 10 ms while the laptop’s own per-request cost was 43–50 ms. Owner F38, in the §C-7 retest.
-
2026-09-29 — F38-a spec drafted (
.forge/specs/F38-a-account-baseline-spec.md, draft): step 9 and the §C-5 model-access row reworded to the Bedrock first-time-use form, the Marketplace permissions and the quota check (F38 FR-4); step 11 reworded to the two-request DNS order (F38 FR-6). New §C-5 row: Claude Haiku 4.5 has no in-Region on-demand route onbedrock-runtimeinus-east-1, and its end of life is no sooner than October 2026 (AWS model card, read 2026-09-28) — the route is the lead’s decision (F38-a Q-3), with a pointer on F12’s spec. No row changes status; the accounts still do not exist (A-5). -
2026-09-29 — The AWS account exists: Empire Teas’ own, held by Kavindu Anthony, billed to Empire Teas directly; SOW ET-002 superseded. A-5 updated (open until dev is usable); the “Start here” intro records the Foundation target of Fri 16 Oct from the revised Statement of Work; step 6 struck as overtaken; step 9 (as reworded by the F38-a spec) gains a note that the form and the payment method are now Empire Teas’. On hold until Shyamin Ayesh finishes his setup: the Terraform location (step 2), the account structure and sign-in model (step 7) and the GuardDuty decisions (step 13, A-1).
-
2026-09-29 — BUG-037: a refused sign-in now redirects to
/login?login=failed(and/login?login=unavailable) instead of the SPA’s root, which dropped the marker. The two §B F5-a redirect rows name the/loginpath and depend on F38 AC-14’s deep-path rule; the?login=failedrow records the laptop evidence (a real refusal against the Empire tenant, laptop, not dev). No new row, and no row changes status. -
2026-09-29 — BUG-041: the §D
ET_SECURITY_LOGIN_REDIRECT_URIrow states the startup rule (an http or https origin only: no path, trailing slash, query or fragment). BUG-040 (laptop mode’s account chooser) adds nothing cloud-gated. No new row, and no row changes status. -
2026-09-29 — The Terraform location is decided:
altrium-io/forge-et-iaac, a separate repository for all DevOps work (AD #23 revised). “Start here” step 2 struck as decided; A-2 annotated (open until the dev estate’s Terraform is merged there); the §C-3 row becomesimplemented. How the delivery team reviews that repository stays on hold until Shyamin Ayesh’s setup. -
2026-09-29 — BUG-047 (no saved-request replay after a mailbox connect), BUG-048 (a mailbox Graph cannot read is marked
MAILBOX_NOT_ENABLED, V30) and BUG-049 (login_hinton the mailbox consent) add nothing cloud-gated: each is proved locally, and laptop mode exercised the real Entra and Graph paths (the CRM05 connect). The real consent round trip on dev stays the existing §B F20-a rows. No new row, and no row changes status. -
2026-09-29 — BUG-022 (forge-et-be#122): the Microsoft mailbox consent is gated on
email.manageat both ends. §B F5-k gains the BUG-022 real-consent row F5-k FR-22 asked for (the fix shipped on its own, not in F5-k WI-2.1), owner F38, open. The §B F20-a consent round-trip row notes thatCRM01’s role needs the key for that check. No row changes status. -
2026-09-30 —
forge-et-iaacread atdaf4f1c: the dev VPC and an empty ECS cluster are applied, inus-west-2rather thanus-east-1; thecrm.empire.lkRoute 53 zone exists outside Terraform, and its NS delegation is to be asked of Empire IT. “Start here” gains the 30 Sep update (the region, and the whole-subdomain delegation against F38’suat-only design, for the lead to decide); A-2 annotated and stays open. No new row, and no row changes status. -
2026-09-30 — F5-l (reach from the reporting line), WI-1.1: “nothing cloud-gated” row added for F5-l (spec FR-22). The grants-without-scope migration, the
allkeys and the reach rule are proved on the local Postgres. No new §B block, and no row changes status. -
2026-09-30 — F5-i-WI-2.1 (Nanndish): laptop evidence as CRM01 recorded in
.forge/plans/F5-i/F5-i-WI-2.1-record.md; notes marked “Laptop, not dev” added to the §B FS-6 certificate row, two §B F5-a rows and four §B F20-a rows. Every row staysopenuntil it passes on dev. -
2026-09-30 — Lead decisions:
crm.empire.lkis delegated whole to Route 53 (F38 Rev 2, F38-a FR-10), so step 11 and the §C-3 DNS row become one NS request plus Terraform-written records; and Claude Sonnet 5.5 replaces Sonnet 5 (AD #19 revised), called through Bedrock’s Global profile until aus.profile exists (ai-provider-strategy.md§4b). Step 9, the F13 row, the live-AIProviderrow, the Bedrock-usable row and the §C-5 Haiku 4.5 row now name Sonnet 5.5. No new row, and no row changes status; the region (us-east-1against the dev Terraform’sus-west-2) waits on the lead. -
2026-09-30 — The favicon PRs (forge-et-fe#89, forge-et-pwa#39) add the CRM’s first static files without a content hash. §B FS-1 gains one row: the S3 sync gives such files
max-age=0, must-revalidate, neverimmutable(architecture.mdRev 49). No other row changes. -
2026-09-30 — The post-merge
/forge-review-prbackfill of forge-et-fe#21 found that the §B F5-a “from-route preservation” row described app code as a cloud-only item: the route was kept only in router location state. The code fix is BUG-054 (forge-et-fe#101; unit tests and the stub-path browser suite, with the real-Entra laptop-mode check left to Tharinda after the merge); the row now keeps only the dev confirmation of the real Entra round trip as cloud-gated, owned by F38. No new row, and the row staysopen. -
2026-09-30 — F41 (dependency bumps that need no browser gate run): added the §B “Nothing cloud-gated” line for F41, from the spec’s Constraints and the plan. No other row changes.
-
2026-10-01 — F5-i Reflect. The §C-1 laptop sign-in row becomes
implemented: the live run as CRM01 (F5-i-WI-2.1) is recorded and merged (forge-et-harness#322). The seven §B rows it annotated “Laptop, not dev” stayopenuntil they pass on dev. F5-i adds no other row. The §B F20-a consent row’s check changes from an exact scope list to “includes” (F5-i spec Rev 4, L-104), and its laptop note records that R-3 did not capture the stored scopes, the connection row or themailbox.connectedevent, so all three are owed on dev. -
2026-10-01 — BUG-022 walked live in laptop mode against the real Empire tenant (backend at forge-et-be
1fde2a3): asCRM01withoutemail.managethe consent start answered403, with no redirect to Microsoft; with the key, the consent connected. That is laptop evidence, not dev: the §B F5-k BUG-022 row stays open for F38’s dev check, including the key removed mid-consent. -
2026-10-01 — §B FS-5, the error-reporter row: the PWA’s file path corrected to
forge-et-pwa/src/app/telemetry.ts, where the file is onmain. No row changes status. -
2026-10-01 — The
crm.empire.lkdelegation is live: Kavindu Anthony added the four NS records in Cloudflare, which servesempire.lkDNS (registered at NIC.lk).dig NS crm.empire.lkreturns the four Route 53 name servers through 1.1.1.1 and 8.8.8.8, andempire.lk’s website and mail records are unchanged. Step 11 and the §C-3 DNS row are annotated; the row staysopenuntil Terraform writes the certificate, host and SES records (environments-and-identity.md§1.1). -
2026-10-02 —
forge-et-iaacreview decided by the lead: every change is a PR to its trunkdev, reviewed by the others with write access (Shyamin Ayesh, Senuthmi Thimansa, Tharinda Rajapaksha, Dinil Gamage —git-conventions.md § forge-et-iaac). The §C row on the separate Terraform repository is annotated. The region was decided at the 2026-10-01 meeting with Shyamin Ayesh:us-east-1, as step 7 already says. Senuthmi Thimansa’s PRs #1 (dev tous-east-1, the state bucket left inus-west-2) and #2 (ECRet-be) are open, not merged; A-2 and A-5 do not change. -
2026-10-02 — §B F38-d block added for the backend container image, built ahead of the F38-d spec in
forge-et-bePR #136 (Dockerfile,.dockerignore, digest-pinned bases, a Dependabotdockerentry). Five rows: the image on FargateARM64from ECR, the task-definition values,trivy imageindeploy-dev, the jar built on the runner, and RDS certificate verification. The lead’s decisions behind them are recorded in the F38-d and F38-b specs’ Open Questions. -
2026-10-02 — F5-f wave 1 (WI-1.2, the real record visibility check; spec FR-C1, AC-18):
RoutingRecordVisibilityCheckreplaces thelocal/teststand-in in every profile. A-1 shows the visibility half built and awaiting F38-f’s dev boot; “Start here” step 19 and the 2026-09-30 update paragraph (by an appended update note) say the same; §B FS-11’s first row now names only F38-f’s proof, and its cloud-shape-tests row says those tests boot the real visibility check and fake only the mailbox and key seams. The F5-f “nothing cloud-gated” row and the F5-f backlog row are replaced by a new §B F5-f block holding the one cloud-gated item (the boot pastSeamGuard, owner F38-f). The rows read true once the F5-f wave 1 PR is onmain. -
2026-10-02 — BUG-050: §B F20-b gains one row, the excluded-folder cache across ECS tasks (cleared first in every back-fill job, one-hour expiry). Proved locally on one JVM; the cross-task behaviour needs two or more tasks on dev. No other row changes.
-
2026-10-02 — F38 Rev 4: CI reaches AWS through an IAM service user per workload account with its key in GitHub secrets, not GitHub OIDC (Shyamin Ayesh’s setup, confirmed 2 Oct). Step 18 and the §B FS-1 credentials row are reworded; the row gains the key-rotation record. Rev 4 is pending Tharinda’s approval.
-
2026-10-02 — F38 Rev 4 approved by Tharinda (forge-et-harness#383). No row changes.
-
2026-10-02 — §B F38-d:
forge-et-bePR #144 replaces the digest pin and the Dependabotdockerentry with tag-named bases and anapt-get upgradein the runtime stage (lead decision); the block’s introduction is updated. One row added: one image per commit (immutable ECR tags, anddeploy-devnot rebuilding a commit ECR already holds), from that PR’sbackend-reviewerpass. -
2026-10-02 — AD #38: vulnerability scans report on every PR and dev deploy and never block them. Start-here step 17 (the Trivy flip before the first deploy) is struck through; the §B FS-4 flip row becomes the release vulnerability check in
deploy-uatanddeploy-prod(F38 FR-35, Rev 5), owned by F38-h and F38-i; §C-6 and the F38-dtrivy imagerow are reworded. The backend’s Tomcat and Jackson findings are fixed by version overrides inforge-et-bePR #145. §E gains the frontend and capture-apppr-validation.ymlcomments that still promise the flip. -
2026-10-02 — GuardDuty decided on (threat detection and Malware Protection for S3); the “on hold” note under “Start here” now records it. §B FS-11 gains a row for F38-j (added the same day: a setting that runs the file pipeline without GuardDuty, proved on dev by F38-f); the uploads-with-no-scan-results alarm row is recommended to F38-e as required; the GuardDuty running-cost row is owed with F38-b’s resources.
-
2026-10-02 — stale-state reconciliation, no row added or closed: “Start here” step 3 and step 19 record F5-j as merged (forge-et-be#78, forge-et-fe#59, 2026-09-27); step 5 is struck through as done (F38’s spec approved and split; F5-i done); the §B F5-c heading cites spec Rev 4.
-
2026-10-02 — AD #23 revised by the lead: “Start here” step 8 and the §C Terraform row name S3’s own lock file (
use_lockfile) instead of a DynamoDB lock table (F38 spec Rev 6); no row added or closed. -
2026-10-02 — §B FS-2 engine-version row: the “recheck against 18” trigger fired and the lead chose PostgreSQL 18 for RDS (FS-2 spec Rev 8;
architecture.mdRev 51). Local, the Testcontainers suite and the pricing proofs moved to 18 in forge-et-be#146. The row stays open, owned by F38-b, untilforge-et-iaacpinsengine_version = "18"and the dev instance reports it. The RDS values sent to DevOps the same day leave rotation off at first; F38 FR-10 and AC-10 still require it, with the mechanism chosen by OQ-K, so the §C-3 rotation row does not change. -
2026-10-02 — F5-k WI-3.1 (plan Decision 13): §B F5-k gains the lost-access path across sites (the
/mere-read after a403, a CSRF refusal included, and on focus, when the SPA callsapi.*cross-site), and “Start here” step 24 adds the stub grant to the stub-only endpoints the browser suites call. -
2026-10-03 — §E: the
forge-et-feandforge-et-pwa“flips to 1” rows are struck through as corrected by forge-et-fe#111 and forge-et-pwa#52 (merged 2026-10-02), which state AD #38 in both repos. No other row added or closed. -
2026-10-03 — F5-c WI-1.1 (plan subtask 9): the §B F5-c identity-link row is refined to the re-planned WI-1.1 plan and spec Rev 5 FR-4 / AC-5 — the user is created on dev through
POST /api/v1/users(or the WI-1.2 screen) as the seeded IT Admin, then signs in with Microsoft, and theuser_identityrow and theuser.identity_linkedevent appear; the block heading no longer says the plans are back to draft. No row added or closed: nothing else in WI-1.1 is cloud-only. -
2026-10-03 — BUG-067 (fixed in forge-et-be#154) added a §B F20-a row: proving on dev with two or more backend tasks that a re-consent reaches every task’s token cache through the issued-scope check, and that Entra’s refresh responses name their scopes in a form the normaliser reads.
-
2026-10-04 — F38-j WI-2.3 updated five places for the switch that turns malware scanning off (F38-j spec FR-14): the §B FS-11 F38-j row now names the final keys, the acknowledgement, the startup sweep, the rescan of
NOT_SCANNEDfiles and the contract’s switching section as the procedure F38-f walks on dev; the “Proposed alarm on uploads with no scan results” row says uploads storedNOT_SCANNEDare not counted inet.attachment.uploads(F38-j decision 11); §D gains an optional sub-table withET_FILES_SCAN_PROVIDERandET_FILES_SCAN_OFF_ACKNOWLEDGED, and itsET_MESSAGING_SCAN_RESULT_QUEUErow is required unless scanning is switched off; the “Start here” GuardDuty sentence points at the switching procedure. No row added, closed or changed in status. -
2026-10-04 — F5-c WI-2.1 (plan subtask 9): §B F5-c gains the AC-17 row — after an email change the next real Microsoft sign-in by the new address links again, the old address is refused, and the mailbox connect is refused until that sign-in and succeeds after it. Locally the unlink, its
user.identity_unlinkedevents and a stub sign-in by the new address are proved. No row closed. -
2026-10-04 — Project audit (D8): “Start here” now says its 2026-09-24 status is dated; A-5’s UAT date follows the revised Statement of Work (Phase 1 UAT to Thu 10 Dec) and its mid-October trigger is replaced by the Foundation milestone (Fri 16 Oct).
-
2026-10-02 — F26-b-WI-1.1 (standards book backend): added the §B “F26-b — Standards book on the account” block with its one row, NFR-2’s response times measured on dev at the NFR-1 volume on the revised
pricing-volume.sql(spec FR-D3, AC-35), owner F38,open. No other row changes. (Merged ontomain’s log on 2026-10-04, so it sits after later-dated lines.) -
2026-10-05 — F5-d Reflect: a “Nothing cloud-gated” row added for F5-d (spec FR-19); no row opened or closed.
-
2026-10-05 — F5-c WI-3.1 (plan subtask 10): in §B F5-c (heading now cites spec Rev 7) the AC-17 row says the email change releases a connection made before it and the user reconnects after signing in by the new address; the “F5-c deactivation releases the mailbox” backlog row moves into §B F5-c as a full row covering deactivation and the email change (
open); the deactivated-user row names the in-transaction session delete (WI-3.1 D5) and adds the two-task session check. No row closed. -
2026-10-05 — F11-b-WI-1.1 (lead list and tag list API): a “Nothing cloud-gated” row added for F11-b (spec FR-H5, AC-29); no row opened or closed.
-
2026-10-05 — F11-b wave-1 review (
forge-et-bePR #202): the F11-b “nothing cloud-gated” row replaced by a §B F11-b block with one row, ALB access logs and the lead search text (spec NFR-6), owner F38,open. -
2026-10-05 — second project audit: A-1’s visibility half recorded as on
main(F5-f done 2026-10-04); the F5-c heading names spec Rev 7; the risk-homes sentence names where each risk family is defined. No row opened or closed. -
2026-10-05 — BUG-069 fixed: the §B F20-b row “The delta poll’s reach and its first-round cost” now says the first delta request is filtered to the back-fill window and every delta request asks for 50 messages a page, notes Graph’s 5,000-message cap on a filtered round, and asks dev to confirm Graph honours both and how the cap behaves. The review of forge-et-be#192 added that a round restarted after a refused link is covered by the cap alone (the restart’s bound is open for the lead), that a refused fresh round now fails the poll, and a dev check that ids stay immutable alongside the page-size preference. The row stays
open; no row added or closed. -
2026-10-05 — BUG-070 (forge-et-be#193): the §D “must stay unset” entry for
spring.cloud.aws.sqs.listener.auto-startupnow records thatSchedulerConfigGuardrefuses it outsideet.environment=test, and that the scan-result listener is BUG-097’s. Nothing new is cloud-gated; no row opened or closed. -
2026-10-06 — F5-c wave-3 review (forge-et-be#201): the §B F5-c mailbox-release row notes that a connect completing in the same instant as the deactivation can survive it (BUG-100, accepted for wave 3) and is rechecked after that fix. No row opened or closed.
-
2026-10-06 — F5-c Reflect: the §B F5-c heading reads done; wave 4 (WI-4.1, the end-to-end suite) ran entirely on the
localprofile and added no row. No row opened or closed. -
2026-10-06 — F26-c spec: the backlog row “F26-c Weekly workbook upload” is withdrawn — the workbook is read in memory and never stored or scanned (F26-c spec D-3), so F26-c has no cloud dependency. Its shipping harness PR adds the §B “nothing cloud-gated” line (FR-23). No row opened or closed.
-
2026-10-06 — F11-f WI-1.1: §B “Nothing cloud-gated” gains the F11-f row (spec FR-H3) — archive and restore, notes and tags, the account timeline entry and its backfill all run on the local stack. No row opened or closed.
-
2026-10-07 — F26-c-WI-1.1 (the backend of the weekly price import): §B “Nothing cloud-gated” gains the F26-c row — the sales, the check, the create, the V37 migration, the seed and the
lkrPerUsdmember run and are proved locally, and the workbook is never stored or scanned (spec FR-23, D-3, AC-28). No row opened or closed. -
2026-10-07 — F5-e Reflect: nothing cloud-gated; one line added to §B’s “Nothing cloud-gated” table. No row opened or closed.
-
2026-10-07 — F1-b Reflect: nothing cloud-gated as shipped; one line added to §B’s “Nothing cloud-gated” table beside the spec-time line. No row opened or closed.
-
2026-10-07 — F5-h decomposition: spec FR-C1 and the Decomposition Plan checked, nothing cloud-gated; one line added to §B’s “Nothing cloud-gated” table. No row opened or closed.
-
2026-10-08 — F26-c: the §B row names the migration as
V36__price_import.sql. It was renumbered from V37 by Dinil’s decision of 2026-10-07, and it becomes V37 again if F11-f’s V36 reachesmainfirst. No row opened or closed. -
2026-10-09 —
forge-et-iaacreconciled atb7e923aandforge-et-beat03404e8e: the backend is deployed to dev (livenessUP) without sign-in. “Start here” gains the 2026-10-09 update; A-2 and A-5 annotated; §C-5 records Haiku 4.5 subscribed and Sonnet 5.5 and Haiku 5.5 refused for the account, and the AgentCore Gateway asked for; §C-3’s rotation row records RDS-managed rotation with no restart yet. The differences between what is built and F38 are listed once, inenvironments-and-identity.md§4. No row opened, validated or closed. -
2026-10-09 — The migration numbering settled: F26-c’s
V36__price_import.sqlreachedforge-et-bemainfirst (forge-et-be#218), so it stays V36, and F11-f’s backfill shipped asV37__lead_timeline_backfill.sql(forge-et-be#219). The 2026-10-07 line’s “the V37 migration” for F26-c and the 2026-10-08 line’s condition are settled by this one. No row opened or closed. -
2026-10-09 — GuardDuty dropped by Altrium management (Thilina, with Malinda; relayed by Tharinda): neither Malware Protection for S3 nor threat detection, until needed or Empire Teas asks. §B FS-11 gains a block note; its GuardDuty proof, no-verdict,
INFECTEDalarm, no-scan-results alarm and cost-line rows arewithdrawn; the switching row is narrowed to the switched-off half; A-1, step 19, the 2026-09-29 update, §C-8’s security-baseline row and the F33 backlog row are annotated. -
2026-10-09 — F1-d (spec FR-H3, via F1-d-WI-1.1): nothing cloud-gated; one line added to §B’s “Nothing cloud-gated” table. No row opened or closed.
-
2026-10-09 — F12 wave 1 review (forge-et-be#229): §B F12 gains row 8, no AI text or signed headers in the logs when DEBUG is raised on dev (the security pass’s finding; the
application.ymlcaps are proved offline only). -
2026-10-10 — F12-WI-2.3 (the paste dialog): §B F12 gains row 9, the client timeout
EXTRACTION_TIMEOUT_MS(12 s) against dev’set.ai.call.deadlineand the measured end-to-end time; proved only under jsdom and against the local stub (F12-WI-2.3 plan Decision 3). No row closed. -
2026-10-10 — F26-f-WI-1.1 (the weekly round’s backend): added the §B “F26-f — Tea Room review, edit and publish” block with its one row, NFR-2’s response times measured on dev at the NFR-1 volume, with
pricing-volume.sqland the newpricing-round-volume.sqlloaded (spec FR-25, AC-28), owner F38,open. Everything else in F26-f, the price-withdrawal migration included, runs and is proved locally. No other row changes. (The migration merged asV39__price_withdrawal.sql, renumbered from V40 before merge; corrected at F26-f’s Reflect, 2026-10-11.) -
2026-10-11 — F5-g Reflect: nothing cloud-gated (spec FR-G1); one line added to §B’s “Nothing cloud-gated” table.
-
2026-10-11 — F7-e and F16 dropped (PRD Rev 61): removed from the §C “AI features on the
AIProviderseam” row; they leave nothing to prove on Bedrock. -
2026-10-11 — F18-b merged into F18-a (lead): the §C scheduled-jobs row names F18-a for both. F38-b and F38-d become “check and close” items: what the DevOps team built is checked against F38’s requirements, and the gaps in
environments-and-identity.md§4 are closed (no register row changes; the checklist steps they carry are the same). -
2026-10-11 — F36-f dropped (PRD Rev 63, OQ-60 closed): its §B row is marked withdrawn.
-
2026-10-11 — BUG-100 (forge-et-be#235): the §B F5-c mailbox-release row no longer asks for a recheck after the fix. The race between a connect and a deactivation or an email change is settled by a lock on the user’s row and proven locally by
MailboxConnectOwnerChangeRaceIT; nothing about it needs the cloud. Its review found that the lock is held across the KMS data-key call, whose client sets no timeout: §B F20-a gains the BUG-174 row,open. -
2026-10-12 — F20-h (Email attachments) split out of F20-c (PRD Rev 68): its row added to the backlog table (the attachments bucket and scan on dev, Graph’s upload session).
-
2026-10-12 — F20-c-WI-1.1 (the send API): added the §B “F20-c — Compose, reply and send” block — spec FR-G2’s items (a) the four sends on dev with the Sent Items match, (b) Graph’s
429and real error codes, © deliverability under SPF and DKIM; (d) the ALB idle timeout and the dialog’s handling of a load balancer’s own page, and (e) a later frontend CSP inherited by thesrcdocframes, both from WI-1.2’s and WI-1.3’s plans; and thestub-send-failureroute’s absence on a deployed API. The F20-c backlog row moves into that block. Allopen. -
2026-10-12 — BUG-174 (forge-et-be#244):
KmsConfignow sets the KMS client’s call timeouts (5 s per call, 2 s per attempt), and a unit spec asserts them. The §B F20-a row staysopenfor the one thing only the cloud shows: a stalled KMS call fails within the bound on dev. Its owner is now Tharinda. -
2026-10-13 — F1-e (Notes and tags on accounts and contacts), via WI-1.2: the “Nothing cloud-gated” table gains the F1-e row (spec FR-H3) — the notes and tags routes, the tag configuration, the account list’s tag filter, the
notemigration and the screens all run on the local stack with the stub login. -
2026-10-12 — F26-g-WI-1.1 (the price expiry roll and manual override’s backend): added the §B “F26-g — Price expiry roll and manual override” block with spec FR-24’s two rows — the EventBridge Scheduler schedule for
price-expiry-sweep(owner F38 /forge-et-iaac) and NFR-2’s response times measured on dev at the NFR-1 volume withpricing-carry-volume.sqladded to F26-a’s and F26-f’s fixtures (owner F38, the §C-7 retest), bothopen— and removed F26-g’s row from “Backlog features whose stub spec already names a cloud dependency”. Everything else in F26-g runs and is proved locally, the sweep through the LocalStack scheduler queue. -
2026-10-13 — A read-only review of the dev account with the CLI profile
et-dev(.forge/audits/2026-10-13-aws-dev-account-audit.md) confirmed what §A-5 and the “Start here” steps still owe — sign-in placeholders (step 15), no state bucket per account (step 8), SES sandbox and the Bedrock 5.5 models at a quota of 0 (steps 9–10), no CloudTrail or budgets (step 12), most of the estate (step 13) — and found the dev database with backups disabled and unencrypted (§C-8) and its secret rotating on 16 Oct with no restart wired (§C-3). No row changes state: each is owed by F38 and the DevOps team, and the dated list isenvironments-and-identity.md§4 “Open with the DevOps team”. Migration V41 applied on dev on 12 Oct, and the AgentCore Gateway (F38 FR-15) was built on 13 Oct. -
2026-10-13 — SP-ARCH-004 stage 2 ran on dev: the §C “SP-ARCH-004 stage 2” row goes to
validated(the register’s first), with the gateway called assvc-harnessthrough an inlinespike-invoke-gatewaypolicy and Claude Haiku 4.5 judging; AgentCore came last (lead-research-source-options.md§11). Start-here step 14 and §B’s F11-c backlog row carry dated notes: the provider choice is a decision, not a cloud item, and the Java call and the task role’sInvokeGatewaygrant are owed only if AgentCore is chosen. Sonnet 5.5 and Haiku 5.5 re-checked the same afternoon: still refused (§C-5 row unchanged). -
2026-10-13 — F1-f (spec FR-H3, via F1-f-WI-1.1): nothing cloud-gated; one line added to §B’s “Nothing cloud-gated” table. No row opened or closed.
-
2026-10-14 — Reconciled with
forge-et-iaac9eac094,forge-et-bedfc9be15,forge-et-febf4adc8and a read-only pass over the dev account: sign-in works on dev (13 Oct), so “Start here” gains the 2026-10-14 update, steps 15 and 20 are done, step 18 is done for the backend and the CRM frontend, and A-5 closes. §C-3’s rotation row and §C-5’s Bedrock row gain 14 Oct notes: rotation still due 16 Oct 23:59:59 UTC; the 5.5 models’ agreements accepted but their quotas 0 tokens a minute. Later the same day, after the lead’s decisions (F38 Rev 9): §B FS-1’s CI-credentials, deploy-workflow, static-file header and cloud-onboarding rows gain 14 Oct notes — the frontend’s S3 sync sets no cache headers. Then (F38 Rev 11): §C-8’s backups row notes that backups are optional in development, production’s 14 days unchanged. -
2026-10-14 — F20-h (Email attachments, WI-1.1): §B gains the
### F20-h — Email attachmentsblock — spec FR-G2’s items (a) to (d) (the versioned attachments bucket with no expiring lifecycle rule and FR-G1’s sends on dev, the load balancer’s idle timeout above 130 seconds, the scan-on path, theemail-upload-cleanupschedule) and the plan’s items (e) to (g) (Microsoft’s attachment throttling, the cross-site upload, the task role’ss3:GetObjectand versionlesss3:DeleteObject), plus the two stub routes’ absence on deployed APIs; F20-h’s row leaves the backlog table. -
2026-10-14 — F20-c-WI-2.1 (Dinil Gamage, with Claude at his direction): laptop evidence of four real sends from CRM01 recorded in
.forge/plans/F20-c/F20-c-WI-2.1-record.md; “Laptop, not dev” notes added to the §B F20-c item (a) row and the §B F20-b delta-query row. Both stayopenuntil they pass on dev; F20-c items (b) to (e) are untouched. -
2026-10-15 — F38-a spec rewritten for F38 Rev 12 and Rev 13 (Tharinda): F38-a covers the dev account only, as “check and close”, and closes when Sonnet 5.5 and Haiku 5.5 answer in dev and SES can send from
dev.crm.empire.lk; production’s half moved to F38-i. In dev there is no CloudTrail trail, root protection, password policy, alternate contacts or account-level S3 Block Public Access, the state stays inet-ops, and the budgets are the DevOps team’s and Empire Teas’. §C-8’s security-baseline and budget rows are annotated; §A-1’s scan-result-queue sentence now says the queue is needed only with scanning on. F38-c and F38-j were checked against the GuardDuty decision of 9 Oct: no code or document change needed beyond that sentence. -
2026-10-15 — F38-a-WI-1.1 (Claude, for Tharinda): the dev account’s checks recorded in
environments-and-identity.md§4 “F38-a — the development account, checked” (AC-1, AC-2, AC-3, AC-6, AC-7, the AC-4 preparatory records, the NFR-3 list), read assvc-harnesswithdescribe,listandgetcalls, plusiam generate-credential-reportandcloudtrail lookup-events, each run once with the lead’s permission (09:10 UTC); §C-8’s security-baseline and budget rows and §C-3’s DNS row annotated with the dev half; step 9 and §C-5’s model-access and Haiku 4.5 route rows reworded to the dev-only scope. No row changes state. The credential report shows keys only onsvc-githubandsvc-harnessand none on the root user, and the event history answers. No findings for the lead; Service Quotas has no requests-per-minute quota for Sonnet 5.5 or Haiku 5.5, so only their tokens per minute can be raised (§C-5). -
2026-10-16 — §C-5’s Bedrock row gains a 16 Oct note: the six quota requests Shyamin Ayesh filed in dev on 15 Oct were all closed by AWS the same day with no quota raised, so Sonnet 5.5 and Haiku 5.5 are still refused (
environments-and-identity.md§4 item 4). No row changes state. -
2026-10-16 — Database password rotation (F38 item 1): Shyamin Ayesh lengthened the dev RDS secret’s rotation to every 365 days on 15 Oct, without rotating it; read-only
secretsmanager describe-secretandcloudtrail lookup-eventsassvc-harnessconfirm the next rotation is 2027-10-15 23:59:59 UTC, the last is still 9 Oct, and the change was a CLIRotateSecret— not Terraform, whose matchingforge-et-iaaccommit76fce06failsterraform planon an unsupportedaws_db_instanceargument and comments out the apply step. “Start here”'s 14 Oct sentence and the §C-3 rotation row are annotated; the row staysopenuntil the backend restarts after each successful rotation, and production does not copy the yearly schedule. -
2026-10-16 — F20-h-WI-2.1 (Dinil Gamage, with Claude at his direction): laptop evidence of five real sends with files from CRM01 (FR-G1’s three and two plan-level ones) recorded in
.forge/plans/F20-h/F20-h-WI-2.1-record.md; a “Laptop, not dev” note added to the §B F20-h item (a) row, which staysopenuntil it passes on dev; items (b) onward are untouched. -
2026-10-16 — F20-d Reflect: a “nothing cloud-gated” row added for F20-d, email templates and merge fields.
-
2026-10-16 — F11-h-WI-1.1 (AI drafting and lead approval): an §B F11-h block with spec FR-H3’s nine items — the real Converse call for
email-draft, the evaluation run on dev, the manual paths, an end-to-end Approve & send asCRM01, Sonnet 5.5 once its quota is raised, log hygiene at DEBUG, the dev model override while Sonnet 5.5 is refused, the client timeout against dev’s deadline, and a matched send reaching a second draft; and §D rows forET_AI_FEATURES_EMAILDRAFT_MODEL(with the dev override),ET_AI_FEATURES_EMAILDRAFT_ENABLEDandET_LEADS_COLDAFTERDAYS. All open. -
2026-10-16 — The lead decided the database password is not rotated on a schedule (F38 Rev 14, answering OQ-K): §C-3’s rotation row now validates the long interval set in Terraform in every environment and a change by hand with a redeploy; planned, applied by the DevOps team. Read in dev the same day: the secret still rotates every 365 days (set by hand on 15 Oct), next 16 Oct 2027.
-
2026-10-17 — F26-d Reflect: a “nothing cloud-gated” row added for F26-d, send this week’s prices; its backlog row is withdrawn (the send is §B F20-c’s).
-
2026-10-17 — F20-f split (reslice,
.forge/project-prd-history.mdRev 74): the backlog row for F20-f now covers linking and storing; rows added for F20-i (live inbox, the 2-second target, automatic connect after sign-in) and F20-j (the 12-month scan on a real mailbox). Each feature’s shipping PR turns its row into §B rows. -
2026-10-17/18 — F38-b wave 1 (forge-et-iaac#3, open, not applied): the dev database stays unencrypted (F38 Rev 17; §C-8 row updated); the RDS-managed secret’s rotation is turned off in
base/rds(§C-3 row updated); the scheduler role’saws:SourceArnis the schedule group’s ARN (the FS-9, F20-b and F20-h/F26-g schedule rows note it); billing settings are not the delivery team’s (F38 Rev 18; §C-8 budget row and “Start here” step 12). The §B rows F38-b closes (FS-2, FS-7, FS-9, FS-11, F12, F20-a, F20-b, F20-h, F26-g) are updated by its wave 2, after the apply. -
2026-10-18 — F38-b spec Rev 7 (forge-et-iaac#3,
e35554d): a §B FS-1 row added for the CRM distribution’s security headers, including whether the 403/404 →index.htmlresponses carry them; checked by F38-b’s wave 2 after the apply. F38-d’s spec was drafted the same day; its cloud-gated items are the §B “F38-d — The backend container image” block (F38-d FR-11). -
2026-10-19 — F38-d-WI-1.1 (Claude, for Tharinda): the DevOps team’s four deploy files reviewed (
DR-1–DR-35, each with its home) and the first-deploy check, dev’s first startup log, AC-1’s settings half (forge-et-be,forge-et-fe) anddev.yamlrecorded inenvironments-and-identity.md§4 “F38-d — the dev deploy pipelines, checked”; §B F38-d block made F38-d’s (the task-definition row’s owner moved from F38-b); §B FS-1’s two rows annotated, the deploy-workflows row’s 14 Oct note corrected (DR-35); one row added to the §B F38-d block, a deploy that ECS rolls back failing the run (DR-9). Findings sent to the lead: four.forge/owed.mdrows (DR-19, DR-23, DR-27, DR-28). -
2026-10-19 — F11-d-WI-1.1 (Claude, for Dinil): an F11-d §B block with spec FR-H3’s seven items (the
lead-triagequeue and DLQ with the task role’s rights andET_MESSAGING_LEADTRIAGEQUEUE, the hourlylead-triage-sweepschedule, theET_AI_FEATURES_LEADTRIAGE_*§D rows, a real Converse call and its cost, the live evaluation run, the 60-second p95 on dev, the DLQ alarm); §D rows forET_MESSAGING_LEADTRIAGEQUEUE(with the DLQ key’s note),ET_AI_FEATURES_LEADTRIAGE_MODEL/_ENABLEDandET_LEADTRIAGE_*; the backlog row’s F11-d entry points at the block. The DevOps request for rows 1 and 2 is in.forge/owed.md(Dinil). -
2026-10-19 — F38-d WI-2.2 plan review (Claude, for Tharinda): a §B F38-d row added for the CRM deploy’s new build job on dev (the
VITE_*values underenvironment: dev, the guard, theweb-distdownload underpermissions: {}), built by WI-2.2 and proved by WI-4.2 or WI-6.1. -
2026-10-19 — F38-d wave 3–5 plan reviews (Claude, for Tharinda): a §B F38-d row added for the capture app’s deploy on dev, built by WI-5.1 and proved by WI-7.1, and a note on §B FS-1’s “Static files without a content hash” row for the capture half; WI-4.2’s plan now adds a §D “Backend — must set” row for
ET_DATASOURCE_SSL_MODE=verify-full(WI-4.1, the lead’s Q-1 answer). -
2026-10-19 — F38-g spec drafted (Claude, for Tharinda): a §B F38-g block added with five rows (the capture app’s target and smoke path; the retest checks needing the file bucket, the mailbox, timed jobs, and the Sonnet 5.5 / Haiku 5.5 models). The §B F5-a
storageStaterow and the §B FS-4 smoke-tier rows updated for the lead’s decision that the smoke tier is run by hand, with no CI job (F38 Rev 21,architecture.mdRev 68); “Start here” steps 25 and 26 annotated (the retest is a manual checklist). -
2026-10-20 — F11-h (Claude, for Dinil): forge-et-be#280 merged, adding
ET_AI_FEATURES_EMAILDRAFT_MODEL(the Haiku 4.5 profile) to dev’s.aws/td.dev.json; §B F11-h item 7 and its §D row now say so. Item 7 stays open until a draft succeeds on dev, which needs forge-et-iaac#3’s Bedrock grant. -
2026-10-20 — F11-d follow-up (Claude, for Dinil): §B F11-d item 5 now runs on prompt
lead-triage/v2(draft forge-et-be#281), whose merge waits on that run. -
2026-10-20 — F38-d spec Rev 3 (Claude, for Tharinda): the dev deploy neither waits for the rollout nor scans the image (forge-et-be#286). §B F38-d’s
trivy imagerow withdrawn; its rollback row revised to ECS’sFAILEDrecord; §B FS-1’s deploy-workflows row notes it. -
2026-10-20 — F11-d (Claude, for Dinil): §B F11-d items 1 and 2 written by the delivery team — the queue, DLQ, grants and
lead-triage-sweepschedule in forge-et-iaac#4 (stacked on #3), the task-definition key in draft forge-et-be#283; both rows stay open until applied and checked on dev. -
2026-10-20 — F11-d (Claude, for Tharinda): forge-et-iaac#3 and #4 applied to dev (run 39475850266) and forge-et-be#283 merged and deployed (
backend:44). §B F11-d row 2validated(the 20:34:58 UTC sweep loggedCompleted); row 1 stays open for a lead created on dev; the block’s opening paragraph and §D’sET_MESSAGING_LEADTRIAGEQUEUErow now say dev has the queue and key. -
2026-10-21 — forge-et-iaac#3 and #4 merged into
dev(d230b133, 2026-10-20 18:33 UTC; Shyamin Ayesh approved both, the lead merged them) and applied bydev.yamlrun 39475850266, and forge-et-be#283 merged (Claude, for Tharinda; read-only AWS reads assvc-harnesson 21 Oct): “Start here” gains the 2026-10-21 update and step 13 a done note; A-2 and A-5 annotated (A-2 stays open for F38-b wave 4’s no-drift check); notes added to the §B FS-1 security-headers row (both/and/leadscarry the four headers; the browser load is still open), the F20-a/F20-b renewal and delta-poll rows, the F20-h rows (a) and (d), the F26-gprice-expiry-sweeprow, the F11-h item 7 row, the three F38-g retest rows and the §C-3 rotation row (RotationEnabled: falsein dev); the F11-d rows keep the 2026-10-20 F11-d update above. No row’s status changes here: each still needs its behaviour seen on dev, and F38-b WI-2.1 records the applied resources against its rows. -
2026-10-21 — F1-c (Claude, for Biseni): Reflect added F1-c to “Nothing cloud-gated” (spec FR-H4; the four WI plans checked).
-
2026-10-21 — F11-c (Claude, for Dinil): spec approved; the backlog-features row is rewritten for Exa (no AgentCore) and the “Start here” step 14 note records the provider; the §B block comes with the first shipping PR (spec FR-H3).
-
2026-10-21 — F11-c-WI-1.3 (Claude, for Dinil): an F11-c §B block with spec FR-H3’s eight items (dev’s Exa key and its
secretsreference with an Exa budget, thelead-researchqueue and DLQ with the task role’s rights andET_MESSAGING_LEADRESEARCHQUEUE, the hourlylead-research-sweepschedule, a live research run with the429/402paths proved only on a mocked server, the evaluation on dev on Haiku 4.5, the dev model override, Sonnet 5.5 on dev, Empire’s own Exa account for production); §D rows forET_MESSAGING_LEADRESEARCHQUEUE(with the DLQ key’s note and its departure from FR-H3 item 2),ET_SEARCH_EXA_APIKEY,ET_AI_FEATURES_LEADRESEARCH_MODEL/_ENABLED,ET_LEADRESEARCH_*and theET_RESEARCH_*/ET_SEARCH_*overrides; the §B backlog row now points at the block. The DevOps request is in.forge/owed.md(Shyamin Ayesh), the task-definition lines a follow-upchore(F11-c)PR (Dinil). -
2026-10-22 — F11-c wave review (Claude, for Dinil): item 2’s visibility timeout is now 240 s, not “at least
et.messaging.visibility-timeout” (30 s) — the review of forge-et-be#297 found one run can take about 110 s with the person search on, so SQS could hand the message to a second consumer; the local queue matches (QueueProvisioner,e93fc895), and.forge/owed.md’s DevOps row says the same. -
2026-10-21 — F12 (Claude, for Dinil): the live evaluation runner is built — forge-et-be#289, merged 2026-10-21 (
./gradlew liveEvaluation, run from a laptop with the lead’s OK, never bybuildor CI). §B F12 row 4, F11-h item 2 and F11-d item 5 now name the command; all three stay open until their runs are recorded. -
2026-10-21 — F11-d / F12 (Claude, for Dinil): the first live run (lead-triage v2, Haiku 4.5) was throttled by the dev account’s 10-requests-a-minute Bedrock quota and gave no result; §B F11-d item 5 records it. forge-et-be#290 paces the run; the quota decision is in
owed.md. -
2026-10-21 — F38-a-WI-3.1 (Claude, for Tharinda; read-only reads as
svc-harness): dev’s SES recorded inenvironments-and-identity.md§4 “F38-a — the development account, checked”, AC-5 — identitydev.crm.empire.lkverified with 2048-bit Easy DKIM and MAIL FROMbounce.dev.crm.empire.lk, DMARCp=none, production access granted (case 179256302200929, 2026-10-21), declared inforge-et-iaacat0eee623and applied bydev.yamlrun 39553084687; §B FS-5’s SES row (itsOwnercell now names F38-a, F38-e and F38-i), step 10 and §C-3’s DNS row annotated, each stillopen; no finding for the lead; who sends the first test message is a newowed.mdrow. -
2026-10-21 — F38-a (Claude, for Tharinda): the first test message through dev’s SES, sent by the lead to the CRM01 mailbox through a temporary
svc-harnesspolicy (added 08:40:41, removed 08:45:37 UTC), reached the Inbox with SPF, DKIM and DMARCpass; recorded inenvironments-and-identity.md§4 AC-5, “The first test message, delivered”, and on §B FS-5’s SES row (stillopenfor the backend’s send, F38-e, and production, F38-i); theowed.mdrow closed; spec Rev 2 widens AC-8’s non-read exception to these calls. -
2026-10-21 — F11-d (Claude, for Dinil): §B F11-d item 5
validated— v2 failed on the migrated lead with no inquiry text; the final v3 (F11-d spec Rev 2, forge-et-be#281) met the bar in one clean run, missing one spam case, which Dinil accepted as a known limit (owed.md, the v4 row). Bedrock’sServiceUnavailableExceptionfailures on the earlier v3 runs are added to the quota row inowed.md. -
2026-10-21 — F13 (Claude, for Dinil; F13-WI-1.1): §B gains the F13 block, items 1–5 of F13 spec FR-H4 (the real call and its cost per call against 2.1 US cents, the evaluation run with the dropped re-engagement count, the manual path, dev’s Haiku 4.5 override, the log check), all
open; the F13 row of the backlog table is removed; §D gainsET_AI_FEATURES_ACCOUNTBRIEFING_MODEL(with dev’s override, set in.aws/td.dev.jsonon the F13 wave branch) andET_AI_FEATURES_ACCOUNTBRIEFING_ENABLED, both proven byAiPropertiesBindingSpec. -
2026-10-21 — F38-b-WI-2.1 (Claude, for Tharinda; read-only reads as
svc-harness, 21:11–21:17 UTC): forge-et-iaac#3 merged (d230b133) and applied (dev.yamlrun 39475850266); the applied resources recorded inenvironments-and-identity.md§4 “F38-b — the dev estate, checked” (AC-1’s merge run, AC-4’s and AC-5’s read halves, AC-6, AC-8, AC-9, AC-11, the rotation read, the ECR lifecycle and the security headers); §D carries the three dev values. Rows updated: §B FS-1 security headersopen — the browser load is pending (D-10); FS-7 main queue and DLQopen; FS-9 schedules and IAMvalidated — producer half, message templateopen, shared queue (OQ-5)validated; FS-11 bucket, task-role S3 andET_FILES_STORAGE_BUCKETopen; F12 row 1open(its task-role-policy half met); F20-a KMS-wrapped keys and customer-managed keyopen; F20-b ingest queueopen, renewal and delta-poll schedulesvalidated — producer half; F20-h rows (a) and (g)open, row (d)validated — producer half; F26-gprice-expiry-sweepvalidated — producer half; notes on A-2, “Start here” step 13 and the §C-3 estate row, which stayopen. forge-et-iaac#4’s lead-triage queue, its dead-letter queue andlead-triage-sweep, and0eee623’ssespolicy, were found and match their Terraform; the F11-d rows are left as the F11-d session wrote them. No findings for the lead. -
2026-10-22 — F38 reconcile (Claude, for Tharinda; read-only reads as
svc-harnesson 21 Oct, 21:13–21:15 UTC, and the DevOps Trello board as exported on 21 Oct): “Start here” gains the 2026-10-22 update (SES works in dev, the backend has no SES sender yet,backend:51still holds the step-16 placeholders, Bedrock refused again); step 9 and §C-5’s quota row record the three requests AWS refused on 21 Oct and the lead’s Support-case reply, both stillopen;environments-and-identity.md§4 AC-4 records all nine requests with their case numbers. -
2026-10-22 — F13 (Claude, for Dinil; Reflect): both F13 wave PRs merged (forge-et-be#296, forge-et-fe#327), so dev’s override for
account-briefingis onmain; §B F13 item 4 and the §D row now say so. Items 1–5 stayopenuntil a brief runs on dev. -
2026-10-22 — F38 reconcile, second pass (Claude, for Tharinda; read-only reads as
svc-harnessat 04:59–05:05 UTC): §C-5’s quota row notes that every quota is still unchanged and which quota each call uses — Haiku 4.5’s cross-Region and Sonnet 5.5’s Global for the backend, Haiku 5.5’s cross-Region for F38-a WI-2.1 — while the Service Quotas requests of 21 Oct for the two Haiku models were the Global ones; stillopen. -
2026-10-22 — F38-b (Claude, for Tharinda): §B FS-1’s security-headers row notes that the browser load F38-b WI-2.1 could not make (no browser tool) is now WI-4.1’s, by the lead’s decision; still
open. -
2026-10-22 — F38-g-WI-2.1 (Nanndish Satgunarajah; these register edits made by Claude at his request): the smoke role, CRM05’s role and the SMOKE records set up on dev on 2026-10-21;
pnpm e2e:smokerecorded green after a forge-et-fe deploy (run 39714930703) and a forge-et-be deploy (run 39727038992), interrupted cleanly, and failed on purpose in a discarded copy (.forge/plans/F38-g/F38-g-WI-2.1-record.md). §B FS-4’s smoke/full-tier row and §B F5-a’sstorageStaterow split into a smoke row (validated) and a full-tier row (open, F38-h); the smoke-set row validated; §B F5-b’s note re-pointed at the full-tier row; §B F5-k’s fresh-database row given dev’s role and user lists before this WI (R-1); “Start here” steps 24 and 26 annotated. -
2026-10-22 — F38-b wave 5 (Claude, for Tharinda; read-only reads as
svc-harness): forge-et-iaac#5 merged (5e078c0d) and applied bydev.yamlrun 39740047356 — the capture app’s certificate, bucketet-dev-capture, distributionEZ06T8FTKF7UZwith its own headers policy, and thecapture.dev.crm.empire.lkrecord, each read in the account. “Start here” step 13 annotated: the capture app’s hosting is done for dev. §B’s capture-deploy rows stayopenuntil F38-d’s capture deploy runs. -
2026-10-22 — F38 (Claude, for Tharinda; read-only reads as
svc-harnessat 11:52 UTC): §C-5’s quota row records AWS’s emailed refusal of the 21 Oct Sonnet 5.5 Global request (case 179257474300142) and that Shyamin Ayesh then sent AWS the lead’s reply; every quota unchanged; stillopen. -
2026-10-23 — F38-d-WI-4.2 (Claude, for Tharinda; read-only reads as
svc-harnessand of GitHub, 22 Oct 19:16–19:25 UTC): wave 2’s runs recorded inenvironments-and-identity.md§4 “F38-d — the dev deploy pipelines, checked” (AC-1, AC-2, AC-3, AC-4, AC-12); DR-1, DR-4, DR-6 – DR-8, DR-12 – DR-15, DR-17, DR-18, DR-20, DR-24, DR-25 marked fixed; §B F38-d’s ARM64 row madevalidated, and its jar-on-runner, one-image-per-commit and CRM-deploy rows annotated (stillopen); §B FS-1’s “CI’s AWS credentials” and deploy-workflows rows annotated; §D gainsET_DATASOURCE_SSL_MODE=verify-full(F38-d WI-4.1, not yet merged); one cancelled CRM deploy recorded as BUG-254. -
2026-10-23 — F38-e spec (Claude, for Tharinda; read-only reads as
svc-harness): the lead’s decisions recorded, with no status changed. (1) Dev gets the alert topic, one test alarm deleted after it is tripped, and an email when a backend deploy fails. Every feature-owned alarm leaves dev, and the owners of the §B FS-5 feature-alarm row, the FS-6 certificate-expiry alarm, the FS-7 and FS-9 alarm rows, the F11-d row 7, F12 row 6, F20-a and F20-b alarm rows become F38-h and F38-i. (2) §C-5’s AI-spend row moves to the billing owners. (3) The FS-5 error-reporter and exporter rows record “none in V1”, with two §E rows for thetelemetry.tscomments. (4) The SES row records that F38-e proves the send through the backend’s task role and that F36-c builds the sender. A §B F38-e block holds the failed-deploy row. F38 Rev 23. -
2026-10-23 — F38-e-WI-1.2 (Claude, for Tharinda; read-only reads as
svc-harness, 05:28–05:35 UTC): the F38-e record layout added below the §B F38-e table (the**Evidence.**paragraph and its subsections); AC-1’s first half (every log group in every Region at 30 days — the one group,/ecs/dev/backend— and the committed and running task definitions’awslogsdriver andecsformat), AC-6 (owners read as F38-h and F38-i, or the billing owners; no metric filter in any Region), AC-8 (the FS-5 error-reporter and exporter rowsvalidatedagainst the code; FR-9’s searches re-run atforge-et-be2efd29cd,forge-et-fe3935f926andforge-et-pwa578013f, one new §E row forSkippedScanSeries.javaandScanResultHandler.java56–60) and NFR-1’s prices (read 2026-10-23 from AWS’s price pages and the Price List API) recorded; §B FS-5 log-group, format and feature-alarm rows and §C-8’s retention row annotated, each stillopen; no findings for the lead. -
2026-10-23 — F11-c (Claude, for Dinil; read-only reads as
svc-harness): forge-et-iaac#8 applied (run 39936834361) and dev’s Exa key set; forge-et-be#307 merged (eccf2992) and deployed bybuildrun 39945256127 asbackend:62, which started at 11:22 UTC with no ERROR line. Lead 12 (Twinings), created by Dinil at 11:39:41 UTC, was researched in 6.6 s: one Exa call (SUCCESS, 5 results, HTTP 200, 2.7 s) andLead Research Completed … state=done, outcome=SUCCESS. §B F11-c item 2 isvalidated; items 1 (the Exa budget), 3 (the first sweep line), 4 (the invented-company lead and the source links) and 6 (theai_invocationrow) stayopenwith what is left; theET_MESSAGING_LEADRESEARCHQUEUE,ET_SEARCH_EXA_APIKEYandET_AI_FEATURES_LEADRESEARCH_MODELrows record the dev values. -
2026-10-23 — F11-c (Claude, for Dinil; read-only log reads as
svc-harness): thelead-research-sweepfiring at 12:08 UTC queued the nine open dev leads with no research and all nine completed by 12:08:23 UTC — sevennothing_found, twono_input. §B F11-c item 3 isvalidated; item 4 records the invented-company case as met by those runs and staysopenfor the source-link check. -
2026-10-25 — F38 reconcile (Claude, for Tharinda; read-only reads as
svc-harness, secret values never read): “Start here” gains the 2026-10-25 update — step 16 done for dev (forge-et-be#300’s values in the runningbackend:62), the schedules seen running (all 173 scheduler messages of 24 Oct consumed, none dead-lettered), forge-et-iaac#6, #7 and #8 merged and applied on 23 Oct,forge-et-pwa’sdevEnvironment complete, the Bedrock quotas unchanged. §B F38-g’sFR-6/bucket,FR-6/mailboxandFR-6/timed-jobsrows record that their capability is now on dev, so their checks can be walked; each staysopenuntil it is. No status changed. -
2026-10-25 — F38-e-WI-2.1 (Claude, for Tharinda; read-only reads as
svc-harness, 08:26–08:28 UTC): forge-et-iaac#7’s apply confirmed (dev.yamlrun 39940862972,1590009, “8 added”). AC-1’s second half recorded — #7 adds no log source, alarm or metric filter, every log group still at 30 days (the one,/ecs/dev/backend), the committed (forge-et-beeccf2992) and the running (backend:62) task definitions unchanged — so AC-1 is met as a whole and ticked in the spec. A new “F38-e evidence — AC-3” subsection records the topic’s policy (one statement: onlycloudwatch.amazonaws.comandevents.amazonaws.com, onaws:SourceAccount570417736903; not encrypted), the four email subscriptions as found (4 confirmed, 0 pending, 0 deleted), the ruleENABLEDwith the expected pattern and one target, the topic, through the input transformer, no CloudWatch alarm, and the no-change plan as pending for F38-e-WI-3.1 (nodev.yamlrun since the merge run). §B FS-5’s alarm-substrate row and §B F38-e’s failed-deploy row annotated, each stillopen; no finding for the lead. -
2026-10-25 — F38-f spec approved (Claude, for Tharinda; no AWS call): §B gains the F38-f block, three rows — the Sonnet 5.5 half of the live Bedrock proof, the capture app’s AC-40 check, and AC-40’s database checks (F38-f spec FR-6; F38 Rev 25).
-
2026-10-22 — F4-d WI-2.2 (Claude, for Biseni): added the §B F4-d block (spec FR-G1, WI-2.2 plan Decision 16) with its two rows — the write run on dev behind the ALB, with the RDS lock-table size to record beside the run’s tagged-record count, and the production first load and final run with Kavindu Anthony’s import owner — and narrowed the “Backlog features” row to F4-b, pointing at the new block.
-
2026-10-24 — F4-d wave 2 review (Claude, for Biseni; forge-et-be#310,
backend-reviewerfinding 2): §B F4-d row 1’s lock description corrected — since WI-2.1’s bulk tag attach (revision 2) the run takes an advisory lock only per tagged record it did not create, plus a row lock per account and contact it merges into or updates, so a first load takes almost none; the local first-load write run is 3.2–4.7 s (revision 3’s bulk creates). -
2026-10-25 — AD #38 (Claude, for Tharinda):
forge-et-bePR #311 upgrades Spring Boot to 4.1.2, whose BOM ships Tomcat 11.0.27 and Jackson 2.21.7 / 3.1.7, and removes PR #145’s three version overrides; Trivy 0.74.0 reports no HIGH/CRITICAL finding on the new lockfile. §B’s release-check row records it; its status is unchanged. -
2026-10-26 — F38-h-WI-2.2 (Claude, for Tharinda): the UAT data checklist written (
.forge/plans/F38-h/uat-data-checklist.md); §B FS-2’s UAT-data row reworded to point at it, still open until the rehearsal enters it (WI-5.2). -
2026-10-26 — §E (Claude, for Tharinda; no AWS call): every open §E row is corrected, at the lead’s request, by forge-et-be#312 (merged
ca27bfcd), forge-et-fe#340 (e9def12) and forge-et-pwa#93 (c4b5db1e), and struck with the PR that corrected it; theMailboxSeamGuard.javarow had already been corrected by forge-et-be#191. The reviews also corrected stale text of the same kind in the touched files (the scheduler and queues called deferred, GuardDuty’s proof placed on dev, two MDC-key comments naming “the alarm family”). §E has no open row. F38 Rev 27 and F38-f spec Rev 1 record it; the two.forge/owed.mdrows for these comments are removed. -
2026-10-26 — Capture-app rows brought up to date (Claude, for Tharinda; no AWS call): “Start here” step 18’s note on the frontends’
VITE_*variables no longer saysforge-et-pwahas no deploy, and §B F38-f’s capture-app AC-40 row says the capture app has deployed to dev since 25 Oct (F38-d wave 5), so the check can be run now. Both rows stay open. -
2026-10-26 — F38-f-WI-1.1 (Claude, for Tharinda; read-only reads as
svc-harness, 04:49–05:03 UTC; no change to the account): the F38-f record layout added below the §B F38-f table (the**Evidence.**paragraph and its subsection table); AC-8 (a) (1 task, no direct path; the load balancer answers), (d) (six schedules, each with its completion line), (e)'s startup WARN (one per task), (f)'s expected outcome (read atforge-et-be30fd601c), (h) (not applicable) and AC-9 (met; ticked in the spec) recorded; §B FS-9’s scheduler row and F20-b’s renewal and delta-poll rows set to “validated — producer half (F38-b WI-2.1, 2026-10-21); the worker’s log line (F38-f WI-1.1, 2026-10-26)”, F20-h’semail-upload-cleanupand F26-g’sprice-expiry-sweeprows to “… and the worker’s log line (F38-f WI-1.1, 2026-10-26); the database half open — read through the F38-f tunnel (WI-4.1)”, FS-11’s switched-off row annotated, stillopen; for the lead: a successful token refresh writes no log line on dev, so FR-8 (f)'s refresh cannot be timed from the logs as spec Q-2 assumed (.forge/owed.md). -
2026-10-26 — F38-f spec Rev 4 and F38 Rev 28 (Claude, for Tharinda; no AWS call): no database tunnel is built — Shyamin Ayesh agreed to it for dev only, and the lead chose the log reads instead. §B F38-f’s database-checks row, FS-11’s switched-off row, and the F20-h
email-upload-cleanupand F26-gprice-expiry-sweeprows now point at the log reads (Flyway’s startup lines for the schema, F38-f WI-3.1; the audit service’s DEBUG line in a short window, F38-f WI-2.6). The F38-f evidence table is renumbered (AC-7 withdrawn; the work items renumbered, with a note mapping the old numbers in earlier entries). -
2026-10-26 — F38-b WI-6.1 (Claude, for Tharinda; reads as
svc-harness, plus one manualforge-et-bebuildrun onmainat the lead’s request): §B F38-d’s one-image-per-commit row annotated. Run 40274424861 redeployed the image ECR already held for30fd601cwithout rebuilding it, healthy asbackend:67. The row staysopenfor the refused re-push and the UAT digest. -
2026-10-26 — F38-b-WI-4.1 (Claude, for Tharinda; read-only reads as
svc-harness, 05:34–05:40 UTC; no change to the account, no workflow started): the database, network and hosting recorded as found and the no-drift check inenvironments-and-identity.md§4 “F38-b — the dev estate, checked”, AC-1, AC-2, AC-3, AC-7 — AC-1 met except the no-changedev.yamlplan,not yet(no run afterc728e06bprintsNo changes.); AC-2 met; AC-3 met; AC-7 met, the capture app’s half included (its bucket holds a build), with the browser load WI-2.1 left open; §B FS-2 engine andbtree_gistrows validated; §B FS-1’s security-headers row validated; A-2 open — the no-change plan; §C-3 rotation, estate and DNS rows and “Start here” step 13 annotated, each stillopen; closing sweep not run — wave 3 not merged; no findings for the lead. -
2026-10-26 — F38-b-WI-4.1 follow-up (Claude, for Tharinda; one
dev.yamlrun onforge-et-iaacdevstarted at the lead’s request, no other change): run 40278225094 atc728e06bplannedNo changes.and applied nothing; recorded inenvironments-and-identity.md§4 “F38-b — the dev estate, checked”, AC-1; F38-b AC-1 ticked; A-2 closed; §C-3’s estate row and “Start here” step 13 annotated (the estate row staysopenfor production anduat). -
2026-10-26 — Harness review (Claude, for Tharinda; no AWS call): the header’s list of what is missing now says what dev has (the account, the Entra certificates and reply URLs, SES, the KMS keys, the alert topic and deployed hosts; Bedrock serving Haiku 4.5 only) and that UAT and production do not exist yet; “Start here” step 24 says the Playwright base URL is configurable through
ET_E2E_TARGETsince F38-g FR-1. -
2026-10-23 — F38-b-WI-3.1 (Nanndish Satgunarajah): the write checks on dev — a malformed message dead-lettered on et-dev-scheduler and et-dev-mailbox-ingest after three receives by the running backend (
backend:60, which now consumes the ingest queue), both test messages deleted; an upload to et-dev-attachments refused unsigned (403) and served by a five-minute signed URL (200), its version deleted; the temporary svc-harness policy f38-b-write-checks created 08:39 UTC and removed 09:58 UTC, each write approved by Tharinda Rajapaksha. Rows updated: §B FS-7 “Real SQS main queue + DLQ with redrive …” →validated; §B FS-11 “S3 bucket with public-access block and versioning …” →validated; §B FS-7 “Spring Cloud AWS 4.1.1 on Boot 4.1 against real SQS …”, §B F20-b “ET_MESSAGING_MAILBOX_INGEST_QUEUE… plus the ingest SQS queue and its DLQ …” and §B F20-b “ET_MESSAGING_SCHEDULER_QUEUEon the backend task definition” → dated updates, stillopen. Evidence: environments-and-identity.md §4 “F38-b — the dev estate, checked”, AC-4, AC-5, AC-12, and .forge/plans/F38-b/F38-b-WI-3.1-record.md. (Merged into this log on 2026-10-26, when wave 3 mergedmainin to close.) -
2026-10-26 — F38-b wave 3 close (Claude, for Tharinda; no AWS call): Nanndish’s WI-3.1 record confirmed at
/forge-deliverM2; the closing sweep (F38-b AC-10) read every FR-10 row on the merged tree — §B FS-2 engine andbtree_gist, FS-7 redrive, FS-9 schedules and shared queue, FS-11 bucket, F20-b renewal and poll schedules, F20-h (d) and F26-g’s schedulevalidated; FS-11 task-role S3, F20-a KMS key, F20-b ingest queue and F12 row 1open, each with its stated reason (F38-f’s real-call or end-to-end proof); recorded inenvironments-and-identity.md§4 “F38-b — the dev estate, checked”, AC-10. Every F38-b acceptance criterion is ticked. -
2026-10-26 — AWS’s answer on the Bedrock quotas (Claude, for the lead; read-only Service Quotas reads as
svc-harness, no other AWS call): AWS keeps the dev account at new-account limits until it has a few billing cycles of Bedrock usage and spend, with no date; two more requests that day closed unchanged. §C-5’s Bedrock-usable row annotated, stillopen; recorded inenvironments-and-identity.md§4 item 4 and AC-4’s 26 Oct entry; the decisions it leaves are a newowed.mdrow. Later the same day the lead decided them: Haiku 4.5 runs every AI feature in every environment until AWS lifts the limits, and the backend paces its Bedrock calls (ai-provider-strategy.md§5 row 10); §D’s three model-override rows now name every environment. -
2026-10-26 — F38-a spec Rev 4 and F38 Rev 29 (Claude, for the lead; no AWS call): new §C-5 row “Sonnet 5.5 and Haiku 5.5 answer one call each in dev”,
open, carrying F38-a’s former AC-4 calls so F38-a can close; the A.3.1 comparison row notes the lead declined running it on Anthropic’s API. Every row that waits on the 5.5 models is listed inai-provider-strategy.md§4d. -
2026-10-26 — F38-a closed (Claude, for Tharinda;
/forge-deliverStage 15; no AWS call): AC-1 to AC-8 ticked againstenvironments-and-identity.md§4 “F38-a — the development account, checked”; “Start here” step 9 gains a note that F38-a’s part ends with the quotas recorded and the two 5.5 calls are the §C-5 row “Sonnet 5.5 and Haiku 5.5 answer one call each in dev”. No row changes status. -
2026-10-26 — F38-d-WI-6.1 (Claude, for Tharinda; each AWS and GitHub write approved by the lead at the time of the step): the deliberate checks on dev (
environments-and-identity.md§4 “F38-d — the dev deploy pipelines, checked”, AC-1, AC-2, AC-5, AC-7, AC-8, AC-9, AC-11, AC-12); DR-2, DR-3, DR-5, DR-11 fixed; §B F38-d rows madevalidated: the task-definition values, the jar built on the runner, the RDS certificate, the rolled-back deploy leaving the previous task serving, the CRM deploy’s build job; leftopen: one image per commit (the refused re-push —svc-harnessmay not push,.forge/owed.md— and the digest UAT promotes, F38-h’s) and the capture app’s deploy (F38-d WI-7.1); §B F38-e’s failed-deploy email row noted (one subscriber received the email for the first deliberate failure; still F38-e WI-3.1’s). -
2026-10-26 — F38-d-WI-7.1 (Claude, for Tharinda; reads as
svc-harnessand of GitHub, and each GitHub write approved by the lead at the time of the step): the capture app’s deploy recorded (AC-5, AC-9, AC-1, AC-7); AC-6 re-read; the closing sweep (AC-10). Madevalidated: §B F38-d “The capture app’s deploy on dev” and §B FS-1 “Static files without a content hash” (both frontends). Noted and stillopen: §B F38-d “One image per commit” (AC-11’s refused re-push, which the lead left open, and the digest UAT promotes, F38-h); §B FS-1 “CI’s AWS credentials” (twosvc-githubkeys in use; UAT and production follow FR-19) and the deploy-workflows row (forge-et-pwahas one;deploy-uatanddeploy-prodare F38-h’s and F38-i’s). “Start here” step 18 noted done for the capture app. -
2026-10-26 — BUG-248 fixed in forge-et-be#316 (Claude, for Tharinda; no AWS call): every Bedrock request uses
toolChoiceauto. New §B F12 row 10,open: one Sonnet 5.5 call on dev and the Haiku 4.5 re-runs of four live evaluations underauto, each with the lead’s approval. F12 row 1 now namesautoinstead of the forced tool. -
2026-10-26 — AD #34 refined (Nanndish; signed off by Tharinda on 2026-10-27; no AWS call):
access.manageis folded intosettings.manage. §B F5-k’s seed-admin row now expects/meon dev to returnsettings.manageafterV54, and BUG-174’s row names the renamed advisory lock. No row is added or closed. -
2026-10-27 — Frontend Content-Security-Policy drafted (Claude, for Tharinda; no AWS call): §B F20-c item (e) notes the CRM’s policy in forge-et-iaac#14 (open), and stays
openuntil the check runs on dev after it is applied. What each app loads is in.forge/owed.md, “A Content-Security-Policy on both frontends”. -
2026-10-27 — Haiku 5.5’s inference profile chosen: Global (no AWS write; reads as
svc-harness): §C-5’s row “Sonnet 5.5 and Haiku 5.5 answer one call each in dev” now callsglobal.anthropic.claude-haiku-5-5and waits onL-F26CA8D1rather thanus.andL-47520EB7. The task role’s grant for that profile is forge-et-iaac#13, open. -
2026-10-27 — F38-i’s spec approved (Tharinda; no AWS call): F38 OQ-J closed (F38 Rev 31), so §C-8’s two client-decision rows (Multi-AZ and cross-region copy; WAF) say the proposal stands and SOW ET-002 §2.2 no longer binds. F38-i is cloud work by construction; its implementation PRs register what each leaves open (F38-i FR-17).
-
2026-10-27 — F42-WI-1.3 (Claude, for Tharinda; no AWS call): §B gains the F42 block (Bedrock call pacing) with FR-H4 item 5,
open: a burst while a deploy replaces dev’s backend task is throttled toRATE_LIMITED, neverCIRCUIT_OPEN. Items 1–4 are added with F42’s wave 2 (F42-WI-2.4). -
2026-10-28 — F20-i decomposition (Claude, for Tharinda; no AWS call): the F20-i backlog row is widened to every item of the spec’s FR-25 (the 2-second target, notifications over the public URL, the limiter under real load, the automatic connect on dev’s reply URL, reply and forward delivered), each tagged with the wave that defers it; F20-i-WI-5.1 turns them into §B rows.
-
2026-10-29 — F20-i-WI-1.1 plan review (Claude, for Tharinda; no AWS call): F5-c’s row “Deactivation and an email change release the user’s live mailbox connection” gains a third dev check — a role edit that removes
email.managereleases the connection asPERMISSION_REMOVEDand deletes its Graph subscription, which locally is proved only against the stub. -
2026-11-02 — forge-et-be#349 review (Claude, for Tharinda; no AWS call): the role-edit release no longer deletes the Graph subscription (the lead’s decision, WI-1.1 plan
## Notes): F5-c’s row’s role-edit check now expects the subscription left to expire and no notification processed after the release. -
2026-10-28 — F20-l Decomposition Plan (Claude, for Tharinda; no AWS call): §B’s “Nothing cloud-gated” table gains F20-l, per its spec FR-17.
-
2026-10-28 — F42-WI-2.4 (Claude, for Tharinda; no AWS call): §B F42 gains items 1–4, each
open— a burst of lead triage on dev stays under the quota, an interactive call during it, the effective quota, and the per-task pacing values in each task definition. §D gains theET_AI_PACING_*rows (proven byAiPropertiesBindingSpec, F42 AC-19) and a row forET_MESSAGING_VISIBILITYTIMEOUT, which the lead-triage listener now asks SQS for on every receive. -
2026-10-28 — F38-i’s spec revised (Rev 1; Tharinda; AWS reads only —
describe-db-instanceson dev, dev’s CloudWatch logs, the Price List API): checklist step 28 and §D’s administrator-email row say the email is a plain task-definition value (AD #40); §D’s Entra row says the tenant ID, client ID and certificate are plain values in every environment (F38-i FR-19); §C-8’s two client-decision rows say no Multi-AZ, cross-region copy or WAF is built. Nothing is created in AWS for production or UAT until the lead says production is needed (F38-i decision 12). -
2026-10-29 — forge-et-iaac#13 and #14 merged and applied on dev on 28 Oct (Claude, for Tharinda; reads as
svc-harnessand of GitHub, and unauthenticated page loads; no AWS write): §B F20-c item (e) staysopen— the CSP is served on both dev hosts and the pages that need no sign-in load with no violation, and the signed-in HTML-email check is left (.forge/owed.md, “A Content-Security-Policy on both frontends”). The backend’s task role now allows Haiku 5.5’s Global profile, so on dev a feature moved to Haiku 5.5 is no longer refused by IAM; it still waits on the quota, which is 0 (§C-5, unchanged). -
2026-10-29 — the dev account read against
forge-et-iaacdevat876d634(Claude, for Tharinda; read-only AWS CLI calls assvc-harness, no write), with the review of forge-et-iaac#11. Everything Terraform declares is there:- ECS
backendruns 1 task onbackend:85, with its rolloutCOMPLETED, and the revision’s environment, secrets, size, platform and roles equalforge-et-be.aws/td.dev.jsononmain. - The 8 queues, the 6
ENABLEDschedules, the 3 buckets, both KMS aliases, both CloudFront distributions with their header policies, the 5 app secrets,crm/backend(IMMUTABLE),et-dev-alertsand the failed-deploy rule are there. - The task role has its 6 inline policies, and SES has production access with
dev.crm.empire.lkverified. - Nothing is tagged
Environment = UAT. - The database reads
BackupRetentionPeriod0, no deletion protection and no encryption, as the lead decided for dev (the proposal’s post-issue note of 14 Oct;.forge/owed.md, the F38-i revision row, item 11).
No status changes.
- ECS
-
2026-10-29 — F42 dev proof (Claude, for Tharinda; reads as
svc-harness, the Bedrock calls of a 30-lead burst approved by the lead): §B F42 items 1, 2 and 3 →validatedon dev (backend:85, one task, shipped rows) — the burst and the next two hourly sweeps triaged every lead with noPROVIDER_ERROR, no open circuit, at most 9 Bedrock invocations and no throttle in any minute, and no redelivered message; an email draft during the burst answered in about 1.4 s. Item 4’s dev read is recorded and the item staysopenfor UAT and production. Item 5 staysopenuntil a burst runs while a deploy shows two tasks. -
2026-10-30 — F42 item 5 attempt (Claude, for Tharinda; ECS and CloudWatch reads as
svc-harness, no Bedrock call): during forge-et-be#336’s dev deploy (run 40929354828) the backend service ran two tasks for about 80 s, 18:19:54 – 18:21:12 UTC on 29 Oct, about 2.5 minutes after the run started, and no burst reached dev in that window. §B F42 item 5 staysopen; its steps now start the burst about a minute after the deploy run starts instead of when the second task appears. -
2026-11-02 — F42 dev proof, item 5 (Claude, for Tharinda; ECS, CloudWatch and SQS reads as
svc-harness; a manual dev deploy ofmainand the Bedrock calls of a 30-lead burst approved by the lead): §B F42 item 5 →validated. The burst started 46 s before a deploy’s two-task overlap; Bedrock throttled 4 attempts in the 05:31 minute, as the overlap ended and while it still counted both tasks’ calls, which ended as 2RATE_LIMITEDresearch rows logged at WARN, with noPROVIDER_ERRORor open circuit, and every lead was triaged and researched by the run and the next sweeps with no message redelivered. The run also showed again that about two in five of a burst’s leads wait for the hourly triage sweep, which F43 addresses (.forge/specs/F43-ai-quota-backlog-and-busy-answer-spec.md). Every §B F42 item is now validated on dev except item 4’s UAT and production reads. -
2026-11-02 — F43-WI-1.2 (Claude, for Tharinda; no AWS call): §B gains the F43 block (AI load under the Bedrock quota) with FR-H3 item 1,
open: a burst of about 30 leads is triaged within about 10 minutes with no first attempt refused for want of a slot, now that a triage call waits up to 60 s for its slot. Items 2–5 are added with F43’s waves 2, 3 and 4 (F43-WI-2.4, F43-WI-3.1, F43-WI-4.2); all five are proved by F43-WI-5.1’s one dev run. -
2026-11-02 — F43-WI-2.4 (Claude, for Tharinda; no AWS call): §B F43 gains FR-H3 item 3,
open: a person’s AI call refused for no slot shows “The AI is busy right now. Try again in N seconds.” with the pacer’s wait on dev, seen with dev’sus.Haiku 4.5 pacing row at 3 for the run and the override then removed. Proved by F43-WI-5.1’s one dev run. -
2026-11-03 — F43-WI-3.1 (Claude, for Tharinda; Bedrock and IAM reads as
svc-harness, no AWS write): §B F43 gains FR-H3 item 4,open: the backend’s task role may call all six inference profiles of Haiku 4.5, Sonnet 5.5 and Haiku 5.5. Dev’s three missing routes (global.Haiku 4.5,us.Sonnet 5.5,us.Haiku 5.5) are in forge-et-iaac#16 todev, applied when it merges; UAT’s three are on forge-et-iaac#11 (6596205); production’s six wait for F38-i’s overlay. -
2026-11-03 — F38-g (Claude, for Tharinda; no AWS call): §B F38-g’s first row updated — the capture app’s configurable target is built (F38-g-WI-1.2) and its smoke check, buildable since the capture app’s dev hosting (25 Oct), is in no work item; F38-g’s decomposition plan and tracker note now carry it as a work item to add before F38-g closes. The capture app’s UAT hosting and its UAT and production deploys are already F38-i work items (WI-7.1, WI-9.3, WI-13.3, WI-16.1).
-
2026-11-03 — F20-i wave 2 (Claude, for Tharinda; no AWS call): F20-i’s backlog row gains the cross-site attachment download from the live message view (FR-15, AD #33; WI-2.5). F20-a’s row on the per-connection Graph rate-limit default names the new defaults: capacity 60, refill 240 a minute, interactive reserve 20 (F20-i NFR-2, WI-2.1, the lead’s decision of 2026-11-03). Later, from the forge-et-be#356 review (2026-11-03 and 11-04): F20-i’s row gains two checks for WI-5.1’s laptop walk — that a message’s
parentFolderIdand the folder list’s ids share one format, and whetherGET /me/mailFolders/deltacan replace the per-folder tree reads. -
2026-11-04 — BUG-342 (Claude, for Tharinda; no AWS call): §B F12 gains item 11,
open: which stop reason Haiku 5.5 sends for a refusal on Bedrock, read from one evaluation run once it answers on dev. -
2026-11-04 — F38-i WI-1.1 (Claude, for Tharinda; GitHub reads and AWS reads only): forge-et-iaac#15 merged (
37b7210) and applied by dev.yaml run 41881575399, with no change to dev (plan: No changes.). The options are proved in production only when WI-4.1’s overlay sets them, WI-12.1 applies it and WI-14.1 records AC-4; no status changes. -
2026-11-04 — F43 wave 3 (Claude, for Tharinda; IAM and Actions reads only): forge-et-iaac#16 merged and applied on dev (run 41882320196, “0 added, 1 changed, 0 destroyed”), and the task role’s
bedrockpolicy was read back with all six profiles. §B F43 item 4 records dev’s part; it staysopenfor UAT and production. -
2026-11-04 — F43-WI-4.2 (Claude, for Tharinda; no AWS call): §B F43 gains FR-H3 items 2 and 5,
open: the two Haiku 4.5 request quotas proved separate under a burst, and each environment’s task definition carrying the background route. §D namesapplication.yml’s six pacing rows, the background model keys for dev, UAT and production (Haiku 4.5’s Global profile, production’s falling back tous.if its Global quota is 0),ET_LEADTRIAGE_VISIBILITYTIMEOUTin a row of its own, and F43 FR-R3’s lowermax-concurrentwhere a profile’s per-task share is not above the waiting background callers. §B F11-c item 6, F11-d item 3, F12 item 1, F42 item 4 and §C-5’s Haiku 4.5 route row gain dated updates. All five F43 items are proved by F43-WI-5.1’s one dev run. -
2026-11-05 — F11-g-WI-1.1 (Claude, for Dinil; no AWS call): an F11-g §B block with spec FR-H3’s seven items (the
lead-rankingqueue and DLQ with the task role’s rights andET_MESSAGING_LEADRANKINGQUEUE, the hourlylead-ranking-sweepschedule, theET_AI_FEATURES_LEADRANKING_*§D rows with the Global profile on dev, a real Converse call and its cost, the live evaluation run, the 60-second p95 on dev, the DLQ alarm) and an eighth for the SQL fingerprint’s locale on dev’s database; §D rows forET_MESSAGING_LEADRANKINGQUEUE(with the DLQ key’s note),ET_AI_FEATURES_LEADRANKING_MODEL/_ENABLED,ET_AI_PACING_BACKGROUNDMAXWAIT_LEADRANKINGandET_LEADRANKING_*; the backlog row’s F11-g entry points at the block. The DevOps request for rows 1, 2, 3 and 7 is in.forge/owed.md(Dinil). -
2026-11-05 — F20-i WI-3.1 plan review (Claude, for Tharinda; no AWS call): F20-i’s backlog row gains wave 3’s Graph checks for the laptop-mode walk (WI-5.1): the search folder’s real fill time, which replaces the five-minute window of spec Rev 6; what Outlook on the web shows of the folder;
filterQuery; the category match’s case rule; theinferenceClassificationliteral; a filterednextLink;parentFolderIdand$searchthrough the search folder (WI-3.1 E-13 to E-21). -
2026-11-05 — Correction (Claude, for Tharinda; no AWS call): §D’s
ET_SECURITY_SEED_ADMIN_EMAILrow still said production’s value comes from Secrets Manager by asecretsreference, which AD #40 (2026-10-28) replaced with a plain task-definition value; the row now says so, as §B F5-k’s production row already did. Found while reviewing forge-et-iaac#17. No status changes. -
2026-11-05 — F20-i wave 4 (Claude, for Tharinda; no AWS call): F20-i’s backlog row gains the folder-tree reads F20-b’s notification path makes to decide whether a new message is in a shown folder (WI-4.1 Decision 10, the lead’s decision of 2026-11-05: no re-read for an unknown folder id), and the New mail notice raised by a real change notification on dev.
-
2026-11-06 — F38-i wave 3 (Claude, for Tharinda; no AWS call): the wave PR forge-et-be#374 logs
MODEL_UNAVAILABLEat ERROR and every other kind at WARN (F38-i decision 27), and moves dev’s Entra tenant ID, client ID and certificate fromsecretsreferences to plainenvironmentvalues in.aws/td.dev.json(F38-i FR-19, AD #40). §B F12 row 5’s check now reads the ERROR line; §D’s Entra row names the PR. Not yet proved on dev: the deploy of the merge and aCRM01sign-in bypnpm e2e:smoke(F38-i-WI-3.1 plan subtask 9, a person’s run). No status changes. -
2026-11-06 — F38-i wave 3 deployed (Claude, for Tharinda; ECS reads as
svc-harness, no secret read): forge-et-be#374 merged (5a14884b) andbuildrun 42157845766 deployedbackend:111, healthy, with the three Entra values as plain values;pnpm e2e:smokepassed all nine checks asCRM05, the sign-in made by hand (environments-and-identity.md§4, “Dev’s Entra values as plain values”). §B F12 row 5’s ERROR line is still proved only on the fake client. No status changes. -
2026-11-06 — Gate 3 Run 4 reslice (Claude, for Tharinda; no AWS call; PRD Rev 84): the backlog rows follow the split and the move to Phase 1. F33’s row becomes F33-a’s and F33-b’s (F33-a builds the attachment store and proves S3 first); the export row names F23-d, which builds the export, beside F23-a and F23-b; F22’s and F20-g’s rows note they are tracked in Phase 1 (started early). F37-a has no row: like F26-d’s, its sends are F20-c’s Graph
Mail.Send, and its spec decides whether its batch adds anything of its own. No status changes. -
2026-11-06 — F4-b-WI-1.4 (Claude, for Dinil; no AWS call): §B gains an F4-b block with spec FR-21’s three items,
open: (a) the write run with leads on dev behind the ALB, timed against its 60-second idle timeout, with dev’s lock-table size beside the run’s counts; (b) the triage backlog on dev, read from (a)'s first load and logged here beside NFR-2’s expected figures (about 1.5 hours with F43’s wait, about 4 hours without); © the production first load and final run with an import owner holdinglead.manageandlead.all, out of working hours, with no deploy and no live evaluation run during the triage backlog. The backlog-features table’s F4-b row now points at the block. -
2026-11-06 — F33-a-WI-1.3 (Claude, for Dinil; no AWS call): a §B F33-a block with spec FR-H1’s five items, each
open: (a) upload, list, download and delete on dev on an account, a contact and a lead, with the404out of reach; (b) the download saved under the file’s own name from the real presigned URL; © a deleted file’s version kept behind a delete marker inet-dev-attachments; (d) A-1’s boot check with thecontactowner type; (e) NFR-3’s list timing at 200 files. The backlog row now names F33-b only. No status changes. -
2026-11-07 — F3-a-WI-4.1 (Claude, for Nanndish; no AWS call): §B “Nothing cloud-gated” gains the F3-a row (spec FR-H3) — the follow-up tables and routes, the timeline row of a done account follow-up, the Activities page, the badge and the record pages’ panel and action all run and are proved on the local stack, checked against the eight WI plans. No status changes.
-
2026-11-08 — the review of forge-et-iaac#18 (Claude, for Tharinda; GitHub reads only, no AWS call): §B F11-d row 7 and §B F11-g row 7 (the dead-letter alarms on the
lead-triageandlead-rankingDLQs) closed as superseded by F38-i spec decision 26 — no dead-letter alarm in any environment; production alarms on the age of the oldest message on each main queue (F38-i WI-5.1), whose list must name both queues. F11-g rows 1 and 2 stay open until #18 is merged and applied. -
2026-11-09 — F32-c Reflect (Claude, for Tharinda; no AWS call): §B “Nothing cloud-gated” gains the F32-c row (spec FR-11) — the hand-out by Pipedrive owner on both lists runs entirely on the local stack.
-
2026-11-09 — F11-g on dev (Claude, for Dinil; read-only AWS calls:
ecs describe-services,ecs describe-task-definition,logs filter-log-events): forge-et-iaac#18 merged and applied on dev (theet-dev-lead-rankingqueue and DLQ, the task role’s rights, the hourlylead-ranking-sweepschedule), and forge-et-be#375 deployed asbackend:120. The first sweep at 09:15 UTC ranked 77 leads on Bedrock. §B F11-g rows 1–4 closed: the queue, the schedule, the Global Haiku 4.5 profile, and the real calls and their cost (USD 0.0028 a call). Row 8 gained the first sweep’scurrentInJava=0. Row 4 notes 8 no-tool replies in 81 attempts; that is in.forge/owed.md, “Watch lead ranking’s no-tool replies on dev”. -
2026-11-09 — F20-g-WI-2.1 (Claude, for Dinil; no AWS call): an F20-g §B block with spec FR-G2’s four items — (a) the minute
scheduled-email-sendschedule, with its 43,200 invocations a month against the free allowances (AC-32), (b) FR-G1’s run on dev, © the tracking endpoints through the load balancer and production’sET_MAILBOX_PUBLIC_BASE_URL, (d) the scanners’ effect on the counts; the backlog row no longer names F20-g. The schedule is forge-et-iaac#19, a draft until F20-g wave 1 runs on dev; the UAT and production applies are in.forge/owed.md(Tharinda). -
2026-11-11 — F20-g-WI-1.3 plan review (Claude, for Dinil; no AWS call): F20-g row (b) also covers a tracked send’s Sent Items row and message view on
CRM01, which proves Graph returnsinternetMessageIdon the live reads; locally only the stub stands in for it. -
2026-11-11 — F34-c Reflect (Claude, for Tharinda; no AWS call): §B “Nothing cloud-gated” gains the F34-c row — custom field definitions and values on accounts, contacts and leads run entirely on the local stack.
-
2026-11-11 — F38-h spec Rev 7 (Claude, for Tharinda; one plan-only
dev.yamlrun with the lead’s approval, and a read-onlyresourcegroupstaggingapi get-resources): “Start here” step 27 now says each UAT round comes up in three steps (uat.yamlapply without the service,deploy-uatregisterstd.uat.jsonand stops,uat.yamlapply with the service), because forge-et-iaac#11 no longer registers a bootstrap revision. No row added or closed: AC-8, AC-9 and AC-5 are still proved only at F38-h’s rehearsal. -
2026-11-12 — F26-g on dev (Claude, for Tharinda; read-only AWS calls:
scheduler list-schedule-groups,scheduler list-schedules,scheduler get-schedule,logs describe-log-groups,logs filter-log-events): theet-dev/price-expiry-sweepschedule isENABLEDwith the contract’s expression (cron(5 0 * * ? *),Asia/Colombo),Inputand target (et-dev-scheduler), and/ecs/dev/backendshows onePrice Expiry Sweep Completedline every night from 2026-11-06 to 2026-11-12 at 00:05 Colombo time. The run of 2026-11-12 loggedcurrentSaleId=7, closeDate=2026-11-12, closed=0, held=1. No row changes: §B F26-g row 1 stays as it is, its database half still open, because no night in the window closed a price. -
2026-11-12 — F20-g wave 2 on dev (Claude, for Dinil; read-only AWS calls:
scheduler get-schedule,sqs get-queue-attributes,cloudwatch get-metric-statistics,logs filter-log-events; one plan-onlydev.yamlrun on the branch with the lead’s approval, apply skipped): forge-et-iaac#19 merged (4b3d883e) anddev.yamlrun 43176389232 added theet-dev/scheduled-email-sendschedule and nothing else. It readsENABLED,rate(1 minute),UTC, targetet-dev-scheduler. The scheduler queue is empty and its dead-letter queue holds 0. No sweep completion line is in/ecs/dev/backendyet, because nothing is scheduled; row (a) stays open for that line, to be seen in wave 3’s run. -
2026-11-13 — F32-d Reflect (Claude, for Tharinda; no AWS call): recorded as nothing cloud-gated in the § Nothing cloud-gated table.
-
2026-11-12 — AD #42 (Claude, for Nanndish; no AWS call): the capture app sends no email. The §B FS-5 SES row loses its capture half (its update of this date), and the §B backlog table’s F36-c row is withdrawn; the backend’s own SES sender now has no V1 consumer, recorded in
.forge/owed.mdfor the lead. No status changes. -
2026-11-13 — F36-b-WI-1.4 (Claude, for Nanndish; no AWS call): §B gains the F36-b block (AI card reading) with spec FR-CG1’s items (a)–(h), all
open— thecard-readqueue andET_MESSAGING_CARDREADQUEUEper environment, a live vision call and its cost, the evaluation run on dev, the call’s time against the 9-second deadline, the startup check ofCARD_READ’s visibility, the end-to-end case from a real phone, the time from upload toreadwith the photo preparation’s time, and a burst of 20 cards; (b) to (e) can run once F36-b’s wave 1 is onmain, (b) to (d) after the lead approves the Bedrock calls. The backlog table’s F36-b row now points at the block; §D gainsET_MESSAGING_CARDREADQUEUEand the card-reading model, switch, wait, visibility, concurrency and attempts rows. -
2026-11-13 — F36-a-WI-1.4 (Claude, for Nanndish; no AWS call): §B gains the F36-a block (capture and upload) with spec FR-CG1’s items (a)–(f), all
open— the capture redirect address per environment, the sign-in round trip on dev, the UAT and production CORS lists, the card photo on dev’s S3, the camera on real phones, andblob:in the capture app’s CSP; the backlog table’s F36-a row now points at it; §D gainsET_SECURITY_CAPTURE_REDIRECT_URI. -
2026-11-13 — F36-a wave 1, Stage 12 hand-over (Claude, for Nanndish; no AWS call): the §B F36-a block gains item (g),
open— the install prompt, standalone mode and safe-area insets on a real iPhone and Android phone over HTTPS, raised by F36-a-WI-1.2’s plan (FR-SC1, AC-7) and appended as the Decomposition Plan’s § Notes, “Stage 12 hand-over, wave 1”, says. -
2026-11-13 — F38-h WI-1.1 and WI-3.1 (Claude, for Tharinda; GitHub reads and read-only AWS calls as
svc-harness:resourcegroupstaggingapi get-resources,acm describe-certificate,scheduler list-schedules,rds describe-db-instances,ecs describe-clusters,sqs get-queue-attributes,service-quotas get-service-quota, Cost Explorer): forge-et-iaac#11 merged (b175dae), adding the threebase/options (withcreate_servicein place of the bootstrap revision, spec Rev 7),overlays/uatanduat.yaml;dev.yamlrun 43376614685 applied no change to dev. Senuthmi Thimansa then applied theuatinstance withserviceunticked (run 43378306661, 75 resources; spec Rev 8): both certificatesISSUEDthroughdns-uat, no service, no task definition, no bundle. Step 27 and §C-3’s estate row annotated. Bedrock quotas re-read: unchanged (Haiku 4.5 10 requests a minute on both routes; Sonnet 5.5 and Haiku 5.5 0). Nothing is proved on a running backend until F38-h wave 4; no status changes. -
2026-11-13 — F36-a-WI-2.2 (Claude, for Nanndish; no AWS call): the §B F36-a block gains item (h),
open— Choose a Photo with a real iPhone’s and Android phone’s camera (an HEIC photo included), decoded upright and sent as a JPEG no longer than 2,048 px, raised by the WI-2.2 plan’s## Risks(subtask 11). -
2026-11-14 — F36-a-WI-2.2 pre-PR review (Claude, for Nanndish; no AWS call): the §B F36-a item (e) gains two real-phone checks, still
open— the camera after the phone is locked or the app is backgrounded, and the live picture in iPhone Low Power Mode (review finding N-4). -
2026-11-14 — F36-a-WI-2.3 (Claude, for Nanndish; no AWS call): the §B F36-a block gains item (i),
open— a capture kept on a real iPhone and Android phone survives closing the browser and the installed app for a day and is sent on reconnect, raised by the WI-2.3 plan’s## Risksand Decision 8 (subtask 13). -
2026-11-15 — F36-a-WI-3.5 (Claude, for Nanndish; no AWS call): the §B F36-a block gains item (j),
open— choosing an exhibitor list file (.csv,.xlsx) through a real iPhone’s and Android phone’s own file picker, from the Add Exhibitors sheet and the nothing-exists panel, raised by the WI-3.5 plan’s## Risksand Decision 14 (subtask 10). -
2026-11-15 — F23-d spec (Claude, for Tharinda; no AWS call): the §C backlog row for list export and import narrows to F23-b. The lead decided that a list export is built and returned in its request, up to 10,000 rows, with no background job or stored file (F23-d spec FR-E7), so F23-d and F23-a have nothing cloud-gated; F23-b’s import upload still needs real S3.
-
2026-11-15 — F37-a-WI-2.1 (Claude, for Tharinda; no AWS call): an F37-a §B block with spec FR-C1’s three items — (a) the minute
price-broadcast-sendschedule, with its 43,200 invocations a month against the free allowances (NFR-7), (b) a batch sent from a real Microsoft 365 mailbox on dev at 20 a minute without a429, © a batch’s emails received with the right To, Cc, standards and tracking. The schedule is forge-et-iaac#20, a draft until F37-a wave 1 runs on dev; the UAT and production applies are in.forge/owed.md(Tharinda). This replaces the 2026-11-06 line’s “F37-a has no row”. -
2026-11-15 — F37-a §B row (b) (Claude, for Tharinda; no AWS call; asked by the F37-a wave 1 session after F37-a-WI-1.3’s plan review, pass 2, N3): a dated Update says a pace below 20 a minute caused by the CRM’s own per-connection rate limiter (
price_broadcast_item.deferredevents with no Microsoft429behind them) is expected, and that the check confirms no Microsoft429arrives and no batch sends more than 20 emails in 50 seconds. The row staysopen. -
2026-11-16 — F36-c-WI-1.6 (Claude, for Dinil; no AWS call): §B gains the F36-c block (capture research, scoring and in-app notification) with spec FR-CG1’s items (a)–(i), all
open, the captured-lead parts marked as shipping with F36-c’s wave 2 — the two research queues and their keys, thecapture-research-sweepschedule, the research models’ overrides, the visibility check in the cloud shape, a live capture run, a 100-exhibitor import, the live evaluations, the notifications on two phones and a conversion’s copy; the backlog table’s F36-c row now points at it; §D gains the two queue keys, the three model overrides, and the switch, wait and consumer keys. -
2026-11-16 — F36-b early checks on dev (Claude, for Dinil; the one live evaluation run the lead approved the same day: 20 Bedrock
Conversecalls on Haiku 4.5 through theus.profile, US$0.089; read-only AWS callssts get-caller-identity,ecs list-services,ecs describe-services): §B F36-b item (b) gains its drawn-card result (20 of 20, 0.45 US cents a call; stays open for (f)); item © is run and misses the bar on 2 invented countries (97.6% matched); item (d) is done (p95 3.1 s). -
2026-11-16 — F36-b second evaluation run (Claude, for Dinil; the lead approved it: 20 Bedrock
Conversecalls on Haiku 4.5, US$0.091): §B F36-b item © gains the run after the prompt fix — 98.4% matched, the phone label fixed, 3 invented countries; still missed. -
2026-11-16 — F36-b third evaluation run (Claude, for Dinil; the lead approved it: 20 Bedrock
Conversecalls on Haiku 4.5, US$0.096): §B F36-b item © gains the run with worked examples in the prompt — 98.4% matched, 2 invented countries; still missed. -
2026-11-16 — F36-b spec Rev 4 (Dinil): a country filled where the card prints no address is a suggested country, not an invented value; §B F36-b item © is met on the third run (0 invented, 2 suggested countries, 98.4% matched). No AWS call.
-
2026-11-16 — F20-g-WI-3.1 (Dinil Gamage, with Claude for the browser and AWS-read steps; AWS reads only): FR-G1’s run on dev — row (b) validated (R-9 failed once on 2026-11-12, BUG-418 open, and passed on 2026-11-16; R-10 missed NFR-3’s literal measure, follow-up in
owed.md); dev notes on rows (a) and ©; a note on row (d). -
2026-11-16 — F36-b-WI-2.1 (Claude, for Dinil; no AWS call): §B F36-b gains item (i), the backend container’s memory on dev while card reads run, against WI-2.1’s local measurement at the dev task’s limits; §D’s
ET_CARDREAD_*row gains the note that an environment routing triage, research and ranking all to Haiku 4.5’sus.profile setsET_CARDREAD_MAXCONCURRENT=1(F43 FR-D5), and its concurrency and attempts are now proven byCardReadPropertiesBindingSpec. -
2026-11-16 — F20-g row (b) update (Dinil; no AWS call): BUG-418 closed
wontfix, NFR-3 measured as the backend’s own time (spec Rev 3), and the Outlook-on-the-web retest dropped by Dinil. Status unchanged (validated). -
2026-11-16 — §B F36-b item (a) (Dinil, with Claude; no AWS call): forge-et-iaac#22 opened to
dev— thecard-readqueue and its DLQ inbase/sqs, and dev’s and UAT’s task-role rights and output; reviewed (approveon94ecf78), waiting on Shyamin Ayesh’s approval. Theforge-et-bekey follows the dev apply; production asked for on forge-et-iaac#17. Status unchanged (open). -
2026-11-16 — F38-g WI-4.2, the group 4.2 retest on dev (Dinil Gamage; Claude ran F20-h.5 and the clean-up in the built-in browser at his request; AWS reads only): §B F20-c “
POST /api/v1/mailbox/stub-send-failureis really absent on deployed APIs” (F20-c.3), §B F20-h “POST …/stub-attachments-failureandGET …/stub-last-sendare really absent on deployed APIs” (F20-h.2) and §B F20-h “(f) The cross-site upload” (F20-h.5; the server’s own413not seen, because the dialog refuses an oversize file before sending) setvalidated; evidence:.forge/plans/F38-g/F38-g-retest-report.mdgroup 4.2 andF38-g-WI-4.2-record.md. F20-h ©, the scan-on path, staysopen(F20-h.8 not on dev). -
2026-11-17 — F38-g WI-4.4, the group 4.4 retest on dev (Biseni Perera; Claude ran the deploy reads and wrote the console scripts at her request; no AWS call): §B F13 “4. The
account-briefingmodel override while Sonnet 5.5 is refused” given a dated Update note — F13.4 passed, a brief succeeded on dev; the row staysopen(itsmodel_idhalf is a database read). F11-d rows 1 and 6 and F11-h items 4, 7 and 9 unchanged: F11-d.1, F11-d.11 and F11-h.2 were left out of this walk (WI-4.4 record, R-0), and F11-h.8 and F11-h.9 wait onFR-6/mailbox. -
2026-11-17 — §B F11-c item 2 and §D’s
ET_AI_PACING_BACKGROUNDMAXWAIT_LEADRESEARCHandET_MESSAGING_VISIBILITYTIMEOUTrows (Claude, for Dinil; one read-only AWS call assvc-harness): lead research’s visibility timeout is 270 s, not 240 s (F36-c WI-1.2 plan Rev 1). forge-et-iaac#21 was merged intodevon 2026-11-17 (41322061) and applied bydev.yamlrun 44022112490 (“Apply complete! Resources: 0 added, 1 changed, 0 destroyed”);sqs get-queue-attributesonet-dev-lead-researchreadsVisibilityTimeout270 the same day. Status unchanged (validated). -
2026-11-17 — §B F36-b item (a) (Dinil, with Claude; read-only AWS calls
ecs list-services,ecs describe-services,ecs describe-task-definition): dev done — forge-et-iaac#22 merged and applied (dev.yamlrun 44021940709: thecard-readqueue pair created, the task role’s SQS policies changed), forge-et-be#458 setET_MESSAGING_CARDREADQUEUE=et-dev-card-read, and dev’sbackend:150task definition shows it. A card read seen on dev, and UAT and production, stay open. -
2026-11-17 — §B F37-a row (a) (Tharinda, with Claude; read-only AWS calls
ecs describe-services,ecr describe-images,scheduler get-schedule,logs filter-log-events,sqs get-queue-attributes): forge-et-iaac#20 merged intodev(fa20b5e8) and applied bydev.yamlrun 44085299528 (“Apply complete! Resources: 1 added, 0 changed, 0 destroyed”);get-schedulereadsENABLED,rate(1 minute),UTC, targetet-dev-scheduler; five minutes on, no firing refused as an unknown job kind andet-dev-scheduler-dlqat 0. Status unchanged (open): a batch leaving Waiting on dev (row (b)), and UAT and production, still owed. -
2026-11-18 — §B F11-i block added (F11-i-WI-1.1; Claude, for Dinil; no AWS call): F11-i spec FR-H3’s three items — the stored switch on dev and the person search following it, a task definition that still sets
ET_RESEARCH_PERSONSEARCH_ENABLED, and the key granted and the switch set before Empire’s real leads enter an environment — allopen. §D’s search-settings row no longer namesET_RESEARCH_PERSONSEARCH_ENABLEDor says the person search staysfalse: the variable is retired (F11-i FR-P2), and the row points at the F11-i block. -
2026-11-18 — F11-i §B item 3 removed: Dinil decided the switch is on everywhere with no precondition (F11-i spec Rev 3).
-
2026-11-17 — §B FS-6, the bad-grant row (Tharinda, with Claude, F38-h-WI-5.1). This was the UAT rehearsal’s first half:
- the 13 Nov apply, recorded as FR-9 step 2;
- the step-3 failure cases in both repositories;
v0.1.0-rc1deployed through the three-step bring-up (backendet-uat-backend:2, its digest equal to dev’s);- the step-5 failure cases: version, rollback and tag on another image;
- AC-6, with its dev half deferred.
The evidence is in
.forge/plans/F38-h/F38-h-WI-5.1-record.md. R-2’s request and R-10’s state search wait on people outside the delivery team. -
2026-11-18 — §B F36-b item (f) (Dinil, with Claude, F36-b Reflect; no AWS call): wave 3 is on
main, and the local half of the end-to-end convert passed in F36-b-WI-3.5. The item staysopenfor item (a) on dev and the real phones. F36-b adds no new cloud-gated item in wave 3. -
2026-11-18 — F36-c-WI-3.7 (Claude, for Dinil; no AWS call): §B F36-c gains spec Rev 6’s items (j), a focus edit on dev, and (k), the
capture-ranking-v2andlead-ranking-v2evaluation sets run on dev on Haiku 4.5 — a precondition of the first UAT or production release that carries F36-c’s wave 3, since no setting chooses a prompt version; bothopen. Item (i) gains the carried-ranking check: a converted lead’s ranking is the capture’s, with nolead-rankingcall for it. Item (g) and §B F11-g item 5 each gain a (Rev 6) clause: from wave 3 their commands run thev2sets, so the Haiku 4.5 run is recorded once, under item (k); item (g) keeps the Haiku 5.5 and Sonnet 5.5 runs. The backlog table’s F36-c row no longer counts the block’s items. No §D row: wave 3 adds no configuration key. -
2026-11-19 — §B F11-d row 1
validated(Dinil): a lead created on dev got its AI summary, the create check the row left; itsowed.mdrow is removed. -
2026-11-19 — §B F11-d row 1 (Dinil, with Claude): forge-et-iaac#24 set the
lead-triagequeue’s own visibility timeout to 150 s, applied on dev by run 44488259851; UAT gets it at its next apply. -
2026-11-19 — §B F36-c items (a)–© (Dinil, with Claude; no AWS call): forge-et-iaac#25 opened with both research queues, their dead-letter queues, the
capture-research-sweepschedule and the task role’s rights (dev and UAT overlays); forge-et-be#485 opened as a draft with dev’s two queue keys and three model overrides. Item (a)'s text no longer asks for the*DLQkeys in a task definition (local/test only). All three stayopenuntil #25 is applied and #485 deployed and read. -
2026-11-19 — §B F36-a rows (a) and © (Claude, for Tharinda; read-only AWS calls): UAT’s half is written — forge-et-be#474 put the capture redirect and the capture CORS origin in
td.uat.json, and forge-et-iaac#23 put the capture host on UAT (applied byuat.yamlrun 44464377454). Both stayopenuntil F38-h-WI-6.1 checks them at the first-rcdeploy that carries them (F38-h spec Rev 12, AC-18). -
2026-11-19 — F38-h-WI-5.2 (Tharinda; Claude drove the browser at Tharinda’s instruction): the UAT rehearsal’s signed-in half — CRM01’s first sign-in (IT Admin with the six Settings page keys), the UAT data checklist entered through the app,
td.uat.jsonatv0.1.0-rc1read against §D and the running revision,/v3/api-docs404, the smoke suite green against dev (againstuatBLOCKED: the target was never built,owed.md), the mailbox check (harmless); §B F5-k’s two rows (onevalidated) and FS-2’s UAT-data row updated; evidence in.forge/plans/F38-h/F38-h-WI-5.2-record.md. -
2026-11-19 — F20-i-WI-5.1 (Tharinda Rajapaksha, with Claude driving the browsers at his instruction): laptop-mode walk recorded in
.forge/plans/F20-i/F20-i-WI-5.1-record.md; F20-i’s backlog row becomes the §B block### F20-i — Live inbox(FR-25), its dev rowsopenwith laptop notes; the Graph behaviours the walk checked are in R-20, and none left open needs dev, so the block has no row 8 (the open ones are.forge/owed.mdrows). -
2026-11-19 — F36-d added to “Nothing cloud-gated” at its Reflect (Nanndish, with Claude): spec FR-CG1, no §B block.
-
2026-11-19 — “Start here” steps 13 and 14 and A-2 (Claude, for Tharinda; no AWS write): the AgentCore Gateway they call built is unused since Exa was chosen, and forge-et-iaac#28 (open,
approveon8f77e99) removes it fromdev; its merge’s apply destroys the gateway, its target, its role and the role’s policy.svc-harness’sspike-invoke-gatewaypolicy is removed by hand afterwards (.forge/owed.md). Nothing cloud-gated depends on the gateway. -
2026-11-19 — F37-b-WI-2.1 (Claude, for Tharinda; no AWS call): an F37-b §B block with spec FR-C1’s six items — (a) the minute
bulk-email-sendschedule, with its 43,200 invocations a month against the free allowances (NFR-7), (b) a send from a real Microsoft 365 mailbox on dev without a429, © a real non-delivery report matched, (d) the unsubscribe page from a received email and Safe Links not unsubscribing, (e) the unsubscribe link unrewritten in a tracked email, (f) the postal address in each task definition and a received footer. The schedule is forge-et-iaac#29, a draft until F37-b wave 1 runs on dev; the UAT and production applies and production’s postal address are in.forge/owed.md(Tharinda). -
2026-11-20 — §B F36-c and §D (Claude at F36-c’s Reflect, under Dinil’s overnight delegation of 2026-11-19, for his review; no AWS call): the Wave 2 marks on items (a), ©, (d), (e), (g), (h), (i) and on the §D rows
ET_MESSAGING_CAPTURERESEARCHQUEUE,ET_AI_FEATURES_CAPTURERESEARCH_MODELandET_AI_FEATURES_CAPTURERANKING_MODELare removed: wave 2 is onforge-et-bemain(#475; the keys inapplication.yml, thecapture-researchandcapture-rankingarms ofLiveEvaluation.groovy, both features inAiPropertiesCloudShapeIT). Wave 3’s items (j) and (k) and the Rev 6 parts of (g) and (i) were added by F36-c-WI-3.7. Every F36-c row staysopen. -
2026-11-20 — F22’s row under Backlog features whose stub spec already names a cloud dependency (Claude, for Tharinda; no AWS call): the lead took F22 and F20-k back into Phase 1 to start early; the row’s phase note now says so and that F22 moves to Phase 2 if not done by the end of Empire’s Phase 1 UAT round. Its cloud-gated item is unchanged. F20-k has no row yet; its spec adds any it needs.
-
2026-11-20 — F43-WI-5.1 (Dinil, with Claude in his session; reads as
svc-harness; the burst’s Bedrock calls and the AI presses made by Claude in the browser pane while Dinil watched, the pacing override on the unmerged branchchore/F43-busy-answer-check(3313e922, run 44706650441,backend:170) andmaindeployed again (run 44707873719,backend:171), under the lead’s approval of 2026-11-02): §B F43 items 1–3 validated on dev; items 4 and 5 validated for dev, UAT and production open; the run cost USD 0.42 (AC-18). Record: .forge/plans/F43/F43-WI-5.1-record.md. -
2026-11-20 — §B F11-g, the week’s
lead-rankingcount (Claude, for Dinil; read-onlylogs filter-log-eventson/ecs/dev/backend, 12–19 Nov): 15 calls — 7SUCCESS, 7INVALID_OUTPUT(every attemptNO_TOOL_USE), 1TIMEOUT. Lead 83 alone accounts for 6INVALID_OUTPUTand theTIMEOUT, failing every day on the same inputs; the other leads missed about 1 call in 9. Filed as BUG-512;owed.md’s watch row is closed. -
2026-11-20 — §B F36-c items (a)–© (Dinil, with Claude; read-only AWS calls
ecs describe-services,ecs describe-task-definition): forge-et-iaac#25 merged intodev(6fca402) and applied bydev.yamlrun 44673993352 (“5 added, 3 changed, 0 destroyed”): both research queues, their dead-letter queues and thecapture-research-sweepschedule (ENABLED) exist on dev. forge-et-be#485 merged (d240a830) and deployed by build run 44719709516; dev’sbackend:172carries the two queue keys and the three model overrides, and the service is steady on it. Item (a) is done for dev; (b) and © keep their dev checks of a live run (the sweep’s log line, theai_invocationrows); items (d) onwards and UAT and production stayopen. The.forge/owed.mdrow “Put F36-c’s queues, schedule and dev settings in place” is removed. -
2026-11-20 — §B F36-c items (j) and (k) (Dinil, with Claude in his session; the lead’s approval of the Bedrock and search calls as Dinil passed it on in the session; no AWS call that changes a resource): item (k)'s two
v2live runs on Haiku 4.5 both failed their bar (figures in the row; BUG-518 filed for the twoNO_TOOL_USEanswers), so wave 3 is held from UAT and production until a run passes, and the lead’s decision is anowed.mdrow. Item (j) partly done in the browser pane on a 5-exhibitor campaign: research, ranking and the focus edit’s re-ranking all worked; the counts, the cost, the 20-exhibitor size, captured leads and a converted lead are still to do. Both rows stayopen. -
2026-11-21 — F45-WI-2.1 (Tharinda, with Claude in his session under the lead’s waiver recorded in the run sheet’s
## Notes; no AWS call): the Graph split (forge-et-be#418, wave 1, merged at37bdb571) checked in laptop mode againstCRM01’s real mailbox, recorded in.forge/plans/F45/F45-WI-2.1-record.md. The Graph calls laptop mode cannot make stay owed on dev by these rows, which now cover the classes split fromGraphMailboxProvider(GraphConnectionCallsmakes the subscription calls) as well: F20-a “Graph subscription creation against the reachable callback”; F20-b “Real Graph change notifications arriving at/api/v1/mailbox/notifications” and “Renewal against Graph’s real expiry clock”; and F5-c “Deactivation and an email change release the user’s live mailbox connection”, for the subscription delete. -
2026-11-21 — F22-WI-2.1 (Claude, for Tharinda; no AWS call): F22’s backlog row becomes the §B block
### F22 — Contact sync(spec FR-17, AC-35), with twelve items — (a) the real Graph contacts delta read, (b) the hourlycontact-sync-sweepschedule and its first firing on dev, © the queue Check now uses: none added, it rides the mailbox ingest queue, available on dev and UAT and in production once F38-i sets its key, (d) a large first read’s run time on dev, (e)–(h) four Graph behaviours F22-WI-1.1 proved only against a mock server, (i)–(l) the delta page size Graph honours, a new folder’s contact, a deletion and a dismissed contact’s move, which F22-WI-1.2 proved only locally. The schedule is forge-et-iaac#30, a draft until F22 wave 1 runs on dev; the UAT and production applies are in.forge/owed.md(Tharinda). BUG-067’s row notes that F22 makes its scope half checkable. -
2026-11-21 — F45 Reflect (Claude, for Tharinda; no AWS call): recorded as nothing cloud-gated in the § Nothing cloud-gated table (spec § Constraints). No row opened or closed.
-
2026-11-21 — F38-e wave 3 (Claude, in Tharinda’s session at his request; every AWS write approved by the lead): WI-3.1 — the four subscriptions confirmed; the manual
dev.yamlrun 44945483585 planned one change, the SES configuration set’s repeatingmax_delivery_secondsdiff, so AC-3’s no-change plan is not met (BUG-532); the test alarm et-dev-f38e-test-alarm created 20:23, set to ALARM, delivered by SNS to all four subscriptions, deleted 20:24 (describe-alarms 0) with the temporary policy removed — AC-5; one deliberate failed deploy (ecs-svc/7652176713461752993, source B) rolled back with the previous task serving, its email delivered by SNS to all four subscriptions — AC-4; the four inbox reports are owed. WI-3.2 — the SES probe et-dev-ses-probe:1 run under et-dev-backend-task, message delivered to the CRM01 Inbox with spf, dkim and dmarc pass, the probe and its policy removed — AC-7. WI-3.3 — a Logs Insights query on one request’s traceId returned its two lines, field names workable — AC-2; the closing sweep waits. §B FS-5’s log-group, format and field-name rows and the alarm-substrate row, and §B F38-e’s failed-deploy row, setvalidated; the SES row annotated, still open. Evidence in §B F38-e and the three record files. -
2026-11-21 — F38-f-WI-2.1 (Claude, in Tharinda Rajapaksha’s session at his request; AWS reads only): the AI features exercised on dev through the app on Haiku 4.5 — one call from the task role (AC-2, CloudTrail found), 41 calls across the five features and lead ranking with each call’s latency and cost and a quality note per feature (AC-3), 16 Exa searches; the capture app’s five AI features not exercised; evidence in §B F38-f “F38-f evidence — AC-2 and AC-3” and .forge/plans/F38-f/F38-f-WI-2.1-record.md. Updates on §B F12 items 1 and 3, §B F11-d item 4, §B F11-h item 1, §B F13 item 1, §B F11-c item 4 and §C-5’s live-calls row; F12 items 1 and 3 change their status text.
-
2026-11-21 — F38-f-WI-2.2 (Claude, in Tharinda Rajapaksha’s session at his request; the writes approved by him at 20:16 UTC, AWS reads only otherwise): the six CRM AI features (the spec’s five and lead-ranking) switched off on dev by one manual build of the unmerged branch chore/F38-f-manual-path-check (b679b6e6, run 44955151808, deployed 22:13:40 UTC on backend:179), each showing its manual path with no search made; main deployed again (run 44956014556, 22:22:22 UTC on backend:180) and each answering through Bedrock; the branch deleted unmerged — AC-4; evidence in §B F38-f “F38-f evidence — AC-4” and .forge/plans/F38-f/F38-f-WI-2.2-record.md. Updates on §B F12 item 5, §B F11-h item 3 and §B F13 item 3, each recording the
enabled: falsehalf in its status text. -
2026-11-21 — F38-f wave 2, the rest (Claude, in Tharinda’s session at his request; every AWS write approved by him and put back): WI-2.3 (the hand-made password change) withdrawn by spec Rev 7 — no environment rotates the password on a schedule; WI-2.4 —
DirectKeysremoved 21:40:12–21:51:09 UTC with a forced deployment, each connection’s refresh failed with a WARN and stayed Connected, refresh worked again at 22:05 with the document read back identical — AC-8 (f) met; WI-2.6 — the audit logger at DEBUG 22:02:05–22:04:25 UTC, one upload storedNOT_SCANNED, downloaded and discarded, exactly oneattachment.scan_skippedline asUSER— AC-8 (e) met, F20-h and F26-g database halves still open; WI-2.5 — the capture app’s worker, manifest,/api/requests and cache checked in the browser pane, the install not run (owed.md), its row stays open. Evidence in §B F38-f and the records.forge/plans/F38-f/F38-f-WI-2.3-record.mdto-2.6-record.md. -
2026-11-22 — Row 11 of the capture app’s shadcn move (Claude, in Nanndish’s session; no AWS call): §B F36-a row (k) opened — the toasts under the deployed CSP on dev, since Sonner’s injected
<style>is blocked bystyle-src 'self'and altrium-io/forge-et-pwa#213 bundlessonner/dist/styles.cssinstead (Nanndish’s decision).