Empire Teas CRM — Test Strategy
Status: toolchain decided 2026-08-18 (Gradle = AD #18; Spock, Vitest, Playwright per team convention); the T1 half is wired and proven as of FS-1 (2026-08-21). Ratified as a team at Foundation slice 4. Last updated: 2026-11-18 (Rev 42 — see
## Revisions) Reviewed by: Biseni — 2026-08-24 (Rev 2–4). The two items Rev 4 raised as proposals — the Microsoft Graph CI posture (§ Non-Deterministic & External Surfaces) and the flaky-test policy (§ Flaky Tests) — were accepted by the team on 2026-08-25 (Rev 5). FS-6 / the F20 spec and FS-4 respectively carry them into effect rather than re-decide them. Related:.forge/design/coding-standards.md§7 (owns the per-repo tool list, the enforcement ladder, and coverage policy — this file never restates it),.forge/design/architecture.md(§ CI/CD Pipelines, AD-O-11),.forge/project-prd.md(§ Non-Functional Requirements — the targets tests have to prove),.forge/checklists/quality-checklist.md
Purpose
This document defines how code is tested across the project — what is tested, at which tier, with which tools, and when each tier runs. It is the spine of the Forge delivery workflow: decomposition assigns a test tier to every work item (## Test Strategy Map), plans declare their tier in ## Test Approach, the test specialists author tests for their tier, and /forge-test-verify guards that the prescribed tests exist and are green.
The tier model and the workflow wiring below are framework doctrine — do not vary them per feature. The concrete toolchain below is this project’s chosen stack (decided 2026-08-18; ratified as a team at Foundation slice 4, thereafter binding). Every feature plan must reference this document and specify which tiers apply.
Repos: forge-et-be (backend) / forge-et-fe (frontend) / forge-et-pwa (trade-show capture PWA) — see CLAUDE.md § Reference Map.
The Test Tier Model (framework-level — do not rename)
Forge delivery is organized around four tiers. The tier names (T1/T2/T3/T-E2E) are referenced by the decomposition ## Test Strategy Map, plan ## Test Approach, forge-plan-review (TC-2 tier-match, TC-3 AC-coverage), forge-test-verify, and the test-specialist agents — they are stable across projects. What changes per project is the toolchain that implements each tier.
| Tier | What it proves | Scope | Who authors | Who runs it |
|---|---|---|---|---|
| T1 — Unit | Individual functions/classes/components in isolation; dependencies mocked. | Within one file/module. | the implementer (backend/frontend) | the implementer (sub-agent can run it) |
| T2 — Integration | Units work together with real in-repo dependencies (DB, in-process API, framework context). | Within one repo. | the implementer | the implementer, if the infra is available in-session (e.g. a container) |
| T3 — Seam + WI-scope E2E | (a) the BE↔FE seam: the API contract holds across repos, no browser (seam-test-implementer, verify Phase 1); (b) WI-scope browser E2E: the slice this wave shipped works in a real browser (e2e-test-implementer, verify Phase 2). |
Cross-repo / through a running stack. | the test specialists (author + static-check only) | the orchestrator (/forge-deliver boots the servers and runs the live suite — a torn-down sub-agent cannot) |
| T-E2E — Full suite | All user workflows across all roles, end to end. | The whole product. | e2e-test-implementer (final-wave type: e2e WI) |
the orchestrator |
Execution-decoupling contract (load-bearing). A dispatched sub-agent is torn down when it returns, so it cannot keep two dev servers booted or drive a live browser. Therefore the T3/T-E2E specialists author + static-check tests and commit “Ready to run”; the persistent main orchestrator runs the live suites and owns the run → classify → dispatch-fix → re-run-until-green loop (test bug → the test specialist in
test-fixmode; code bug → the owning implementer). T1/T2 are sub-agent-runnable, so the implementers both write and run them./forge-test-verifyis the green-status guard that runs after the orchestrator’s run is green.
The orchestrator’s live runs use their own stack. Each run that needs a database in a known state — the baseline run on
mainand each wave run (lesson L-117) — starts the backend onforge-et-be’scompose.verify.yamlas its own compose project, namedet-<ticket>-<role>(-p et-f5-f-baseline— Compose accepts only a lower-case name, so lower-case the ticket id and write a work item’s dot as a dash; the session hookguard-docker-resources.shrefuses any other name,.forge/design/harness/docker-resource-labels.md; Postgres 55432, LocalStack 45666, named volumes), never on the developer’s stack fromcompose.yaml. The procedure, with the three environment variables the backend needs, isforge-et-be/README.md § Running an isolated stack for a verify run(lesson L-118).
Most tests belong at the bottom. Each tier up proves less and costs more — a T1 spec runs in milliseconds against a mock, a T-E2E run needs two servers, a database and a browser. So push every assertion to the cheapest tier that can actually prove it, and treat a rule that only has E2E coverage as under-tested rather than covered. The corollary is the useful one when a suite goes red: a green T1/T2 suite with a failing T3 means the pieces work alone and not together — look at the seam, not at the units.
Tier toolchain
The concepts (mock isolation, real-dependency integration, browser E2E + accessibility) are framework-universal; the tools below are this project’s choices.
T1 — Unit
| Aspect | forge-et-be |
forge-et-fe |
|---|---|---|
| Framework | Spock 2.5-groovy-5.0 — spock-core + spock-spring on the JUnit Platform, the team’s standard backend testing framework. The Groovy suffix tracks the Boot line and is a hard gate, not a label: Boot 4.2’s BOM manages Groovy 5 and Spock fails the compile against a Groovy line it was not built for (AD #25). Spring test utilities come from bare spring-test, deliberately not spring-boot-starter-test — Spock brings its own mocking and assertions, so the starter’s Mockito/AssertJ/Hamcrest would be unused weight inviting two idioms into one codebase. Spock’s built-in mocking replaces Mockito; its given/when/then blocks and data tables replace AssertJ-style chains |
Vitest + React Testing Library (+ @testing-library/user-event) |
| What to mock | external services (Graph, AIProvider, S3/SQS clients), data-access layer, security context — via Spock Mock()/Stub() |
API calls, router, auth context |
| What NOT to mock | the unit under test, simple value objects | the component under test |
| Naming / location | *Spec.groovy, mirrored package under src/test/groovy (one module since FS-14); a shared base spec in …/support carries the container lifecycle |
co-located *.test.ts(x) next to the source (one package since FS-14) |
| Run command | ./gradlew test |
pnpm test |
Unit-test per feature: every method with business logic, every state-transition rule, every validation rule, every component with conditional rendering or interaction, every custom hook/util. Don’t unit-test trivial accessors, framework-generated code, or styling.
T2 — Integration
| Aspect | forge-et-be |
forge-et-fe |
|---|---|---|
| Framework | Spock 2 + @SpringBootTest / @DataJpaTest (via spock-spring) + Testcontainers (PostgreSQL), with MockMvc for the web layer. A spec that needs a real port uses RestTestClient bound to the injected @LocalServerPort — Boot 4 removed TestRestTemplate and @SpringBootTest contributes no client bean (AD #25) |
Vitest + React Testing Library, with the API client stubbed at the module boundary — the same tools as T1, at page/flow scope. No HTTP-level mocking tool is adopted (an earlier draft named MSW; it was never added to coding-standards.md §7.3 and never installed at FS-1, so the row asserted a dependency the project does not have). Revisit if a generated API client is built (none exists — coding-standards §3) and there is a contract worth intercepting at the transport layer — until then a stubbed client is the honest description and the boundary the seam tests (T3) actually prove |
| Scope | data-access against a real DB, controller→service→data flows, auth/security chain, role-based data scoping, Flyway migrations (one public schema, one sequential version line since FS-14) |
page-level rendering with mocked API, form-submission flows, protected-route behavior |
| Real dependency | Testcontainers Postgres — a real PostgreSQL, not H2: the pricing subsystem’s Postgres-only temporal constraints (range/exclusion types — the settled scheme is <backend-repo>/docs/data-model.md §2, SP-ARCH-003 resolved 2026-08-21; no spike is open) and JSONB custom fields don’t exist on H2, so an in-memory DB would test a different database |
none — the stub is in-process (see Framework row) |
| Run command | ./gradlew check |
pnpm test |
Container-backed specs — the operating rules
Settled at FS-2 (.forge/plans/foundation/002-data-layer-plan.md, Decisions 12 / 20 / 21). They live here because the second person to add a container-backed spec will read this file, not a closed slice’s plan.
- One container per test JVM, shared through a base spec — not one per spec, which would put the CI budget out of reach on its own. Since FS-14 collapsed the build to one module, the base spec is simply
SharedPostgresSpecin the one test source set (src/test/groovy/com/empireteas/support/), which every container-backed spec extends; nojava-test-fixturescross-module publication is needed any more (there is one module, so the base class is visible everywhere). - “One per test JVM” means one container. Gradle runs a single test JVM for the one module unless
-Pet.testForksasks for more, so a shared static container starts once per test JVM: once in a local build, twice in the nightly full run, which passes-Pet.testForks=2(.forge/design/harness/test-run-cost.mdB-7). Budget wall-clock against that one start (FS-14 collapsed the former three-module, three-container arithmetic). - No
@DirtiesContext. The cost of sharing is shared state — see § Test Data & Isolation, whose no-absolute-COUNT(*)rule is what makes sharing safe. - Raw-JDBC specs carry no Spring context — a spec whose every assertion is a database assertion (constraints, triggers, exclusion behaviour, identity) extends the raw Testcontainers base and skips Boot, which adds startup cost for none of it; specs that need the context use
@SpringBootTest. The distinction is per spec, not per module (FS-14). One consequence worth knowing before you write an exception test: a raw-JDBC spec can only produceSQLState 23505off aSQLException, never Spring’sDataIntegrityViolationException, so a translator tested at this layer takes the former. - Specs supply
et.datasource.*from the container, not@ServiceConnection—et.datasource.*is the authoritative config source and produces theDataSourcebean. - A container-backed spec that drives HTTP authenticates through
LoginSupport(FS-6), a trait in the one test source set: it seeds a role with exactly the grants the spec’s paths need, signs in through the real filter chain via the profile-scoped stub login, and returns a client carrying the session cookie and a currently-valid CSRF token. Deny-by-default means every/api/v1call needs a session, so this is not optional — and it is one fixture rather than a copy of the cookie dance per spec, because two things about it are easy to get wrong in ways that read as unrelated failures (the CSRF token is fetched from an anonymouspermitAllpath, and it is re-read after login because authentication replaces it). Grants are a parameter, never a default: a fixture that seeded a role without its grants would 403 every migrated spec and look like the annotation’s fault. - Tests run under the
testprofile, neverlocal(FS-2 NFR-4). Since FS-6 the base spec activates it (DatasourceContextSpeccarries@ActiveProfiles('test')), because the stub login is profile-scoped and every container-backed spec now needs it.localpoints at the Docker Compose database a developer is using; a test that reaches it can wipe someone’s working data.
The wall-clock budget is 10 minutes for each CI job that runs tests, with a lead-approved exception recorded in coding-standards.md §6.8: 30 minutes for forge-et-pwa’s Slow lane, 45 for forge-et-fe’s and 90 for forge-et-be’s; each runs its suite only on demand (the backend’s suite took 13 to 19 minutes on the nightly runs of 2026-10-04 to 2026-10-11, passed 30 from 2026-10-16, and took 47 in CI on 2026-11-11); every Fast lane keeps 15 (coding-standards.md §6.8; every workflow job declares timeout-minutes with that cap, because GitHub’s default is 360 and one hung step otherwise burns six hours of a shared monthly quota). FS-2 makes it a measured obligation, not an assumption (NFR-1, AC-23). If container-backed tests stop fitting, the honest lever is fewer container-backed specs — not a raised cap (each raise is a lead’s recorded exception, not a precedent): push the assertion down to T1 where it can go, and keep the database for assertions that genuinely need a database.
T3 — Seam + WI-scope E2E
| Aspect | Details |
|---|---|
| Seam (Phase 1) | API/contract tests proving the BE↔FE seam against the frozen wave contract (<TICKET>-Wave-<N>-contract.md) — no browser. Toolchain: Playwright request fixture (APIRequestContext), TypeScript, in forge-et-fe/e2e/seam/ — session-cookie auth handshake, CSRF token dance, multi-step flows, response-shape assertions against the contract. Authored by seam-test-implementer; named in the verify-WI plan’s ## Test Approach. |
| WI-scope browser E2E (Phase 2) | Browser tests for the slice this wave shipped + an accessibility scan on each visited page. Toolchain: Playwright + @axe-core/playwright. Authored by e2e-test-implementer. |
| Config / location | forge-et-fe/playwright.config.ts, tests under forge-et-fe/e2e/ (e2e/seam/ browserless, e2e/browser/ by feature) |
| Run command | pnpm exec playwright test e2e/<scope> (scoped to this WI’s paths) |
T-E2E — Full suite
| Aspect | Details |
|---|---|
| Scope | all user workflows, across every fixture role (§ The roles the fixtures must cover — incl. the price-publish flow and a read-only role), end to end; runs as the merge gate on the final-wave integration PR. |
| Toolchain | Playwright + @axe-core/playwright, full coverage; pre-authenticated storage state per role. |
| Run command | pnpm exec playwright test (full suite) |
Selector & fixture conventions — the
e2e-test-implementerandseam-test-implementerread these here:
getByRole/getByLabelfirst — they assert accessibility for free (WCAG 2.1 AA is a PRD NFR).data-testidwhere semantics can’t uniquely address an element (Kanban cards, virtualized rows, icon-only buttons) — conventiondata-testid="<feature>-<element>[-<qualifier>]", added when the component is built, stable across refactors, never derived from user data. Full rule:.forge/design/coding-standards.md §4.- Never select by CSS class / XPath / styling; assert behavior over pixels.
- Reuse pre-authenticated storage state per fixture role — the sample roles in the laptop dataset (
R__local_seed.sql), plus the seeded IT Admin or the sample all-permissions test role (F5-k; § The roles the fixtures must cover); reset + reseed between test files. The default session is the seeded admin or the all-permissions test role — F5-k chooses which — and never a superadmin, which AD #34 removes.- Run the axe accessibility scan on every visited page — not optional.
- Wait for the page to settle, not for an element to appear, before a scan or an action that depends on it: every finite animation in the element finished (
getAnimations({ subtree: true }), skipping infinite ones), the previous interaction’s effect visible (a closed dialog’s content gone), and background work the test started settled, or the assertion accepts both outcomes. Assert the wait’s own target exists first, so a renamed element fails the test rather than removing the wait. Why: three browser-test bugs had this one cause, two failing on every run and one intermittently (.forge/lessons.mdL-101).
Capture app (forge-et-pwa)
The capture app follows the same tier model — same tools, same tiers, same conventions. Two obligations are settled and testable today:
- T3/E2E — Playwright against the capture app’s own URL for the capture → lead flow.
- T1/T3 — the service worker is scoped to shell assets by an allow-list, asserted rather than assumed: a worker that caches an API response survives a redeploy, which is the one form of the stale-data failure a redeploy cannot clear (
architecture.md § Trade-Show Capture App;coding-standards.md §0.1).
The backend’s capture API slice is covered by the backend’s seam tests like any other endpoint.
The rest of the capture test surface is deliberately not enumerated here yet. An earlier version of this row named QR parsing and an offline queue as known T1 surfaces; both were verified absent from the POC (L-005), and how much of F36 we build is the open question OQ-55. Filling this in against an unresolved scope would produce test rows for capabilities that may not ship. Revisit when OQ-55 closes.
Test Naming & Layout
The test specialists are told to read the layout from this section rather than invent one (.claude/agents/e2e-test-implementer.md), so it is a contract, not a suggestion.
forge-et-be
All tests live in src/test/groovy, mirroring the src/main/java package structure of the one module (FS-14 collapsed the former multi-module tree). A build check asserts src/test/java contains no test classes, so the Spock convention cannot erode one JUnit file at a time (FS-1 FR-B9, AC-6).
forge-et-be/
├── src/main/java/com/empireteas/{platform,pricing,account,lead,…}/ # feature-first packages (`ls` lists them)
└── src/test/groovy/com/empireteas/
├── platform/… · pricing/… · account/… # specs mirror src/main/java, package for package
│ ├── FooSpec.groovy # T1 — mocked, no container
│ └── FooIT.groovy # T2 — container-backed
├── support/SharedPostgresSpec.groovy # the shared Testcontainers base spec (T2 rules)
└── arch/ # ArchUnit hygiene guards (layering, feature slices, …)
*Spec.groovy= unit (T1) — mocked dependencies, no container, no Spring context.*IT.groovy= integration (T2) — needs the database, the Spring context, or both. Same source set and same./gradlew test/checktask; the suffix is what tells a reader (and a bisecting developer) which specs cost a container.- The rule binds new specs; existing ones are accepted legacy. About two dozen context- or container-backed backend specs predate it and are named
*Spec(e.g.DatasourceContextSpec,MailboxIngestContextSpec—grep -rlE "SharedPostgresSpec|@SpringBootTest|@ContextConfiguration" src/test/groovy \| grep 'Spec.groovy$'lists them). They are not renamed (lead decision, 2026-09-27); a new spec that needs the database or a Spring context is named*IT. - One spec class per production class; split when the production class is large.
- A feature-package spec stays within its capability —
FeatureSlicesSpeckeeps feature packages acyclic andplatform/uireaching no feature, in test code as in main (FS-14 replaced the former module-boundary rule with this cycle rule).
forge-et-fe and forge-et-pwa
Unit and component tests are co-located with their source; browser and seam tests live in a separate top-level e2e/ tree and are never mixed with unit tests.
forge-et-fe/
├── playwright.config.ts
├── src/ui/Button.tsx
├── src/ui/Button.test.tsx # T1 — co-located
├── src/features/quotes/…
│ └── QuotePage.test.tsx # T1/T2 — co-located, client stubbed
└── e2e/
├── fixtures/ # pre-authenticated storage state per role
├── seam/ # T3 Phase 1 — APIRequestContext, NO browser
└── browser/<feature>/ # T3 Phase 2 + T-E2E — by feature
*.test.ts/*.test.tsx= Vitest, beside the source file.e2e/seam/is browserless by construction — a spec there that launches a browser is in the wrong directory.e2e/browser/is organised by feature, so a wave’s WI-scope suite and the eventual full suite are the same files rather than two parallel sets.
Capture-app specific: pnpm dev runs with no service worker, by design; pnpm preview is where the worker is active. Any assertion about the service worker, offline behaviour, or the install manifest must therefore run against the preview build — a worker test pointed at the dev server passes for the wrong reason (see forge-et-pwa/CLAUDE.md § Common commands).
Naming the tests themselves
Name a test after the behaviour, not the method — shouldRejectSubFloorQuoteWithoutMdApproval, not testApprove3. The name is what a failing CI run shows first, and it should say what broke without opening the file.
When Tests Run
Commands are the project’s own — each app repo’s CLAUDE.md § Common commands is the source of truth; they are restated here so the most-consulted table in this file is usable without a jump.
| Trigger | What runs | Commands | Failure blocks |
|---|---|---|---|
| Every code change (dev loop) | the T1 and T2 tests the change needs + lint (.forge/design/harness/test-run-cost.md S-1, S-3) |
BE python3 scripts/select-tests.py · FE pnpm test:selected + pnpm check — each runs everything when shared code changed |
nothing — fast feedback |
| Per-subtask (implementer) | the static checks, the T1 and T2 tests the change needs, and the build where runnable (W-1) | BE ./gradlew check -x test then python3 scripts/select-tests.py · FE pnpm check && pnpm typecheck && pnpm test:selected && pnpm build |
the subtask commit |
Verify gate (/forge-deliver Stage 7, per wave) |
the orchestrator runs the live T3 seam gate then the T3 browser suite + the run→classify→fix→re-run loop; /forge-test-verify then guards green status (files exist, ACs covered, run green) |
pnpm exec playwright test e2e/seam/… then pnpm exec playwright test e2e/browser/… |
PR open — guard returns Re-run required / Fail until green |
| Once per PR, after its last push (all three app repos) | the T1 + T2 tests the change needs on the final head — every test when the change touches shared code (scripts/shared-code.json) or with --full (.forge/design/harness/test-run-cost.md S-4) — on the developer’s machine, recorded as a PR comment naming the commit and as the required Test record commit status on it; a later push leaves the new head without it, and a change to documentation no test or build step reads carries it forward without a new run, as does, on a PR into main, a merge of main whose own changes are such documentation (test-run-cost.md W-5; .claude/rules/git-conventions.md § The test suites run locally, and the PR records them) |
BE python3 scripts/local-build-evidence.py · FE pnpm test:record |
the merge — a PR whose head has no Test record check cannot merge, and review blocks it |
Before a PR into release/* is ready (forge-et-fe, forge-et-pwa) |
the full-profile browser gate on the committed head, on the developer’s machine (§ Browsers & Viewports), because the nightly run tests only main; the push that marks it ready, and the last later push that changes a gate input, goes through pnpm e2e:gate --full --push, followed by pnpm test:record on the same commit. A PR into main runs no gate |
pnpm e2e:gate --full --push |
the ready PR into release/* — the last step of the Fast lane’s full run fails it without a covering Browser gate: line |
Before a PR into main is ready (forge-et-fe, forge-et-pwa; wave PRs included) |
the browser specs and size-matrix states the change selects, on the final head after the test record, on the developer’s machine, with the line the run prints in the PR’s test evidence as Selected browser run: … (none selected when it selects nothing); again when main gains a merge in the same feature area before the PR merges (.forge/design/harness/fewer-nightly-reds.md R-1, R-2; PRD Rev 93). No full gate and no Browser gate: line |
pnpm e2e:gate --selected |
the PR being marked ready, checked by the review record (frontend-reviewer dimension 32) |
| Wave PR (CI) | lint/typecheck and the other static checks and the build (workflow pr-validation, the two lanes coding-standards.md §7.1 describes in every app repo). T1 + T2 run in the local record, whose Test record check the merge needs, not in PR CI. /forge-deliver Stage 12 reviews the draft wave PR, then the local record runs once (the live T3/T-E2E run of the wave’s own specs already happened pre-open, above), then, in the frontends, pnpm e2e:gate --selected on the same head with its line in the PR’s test evidence (.forge/design/harness/fewer-nightly-reds.md R-1); no full browser gate runs |
the static checks in CI; the local record locally | the wave PR merge |
Nightly on main (the lead’s Mac, 02:00 Sri Lanka time) |
In order: gitleaks over main’s whole history and Trivy over the lockfiles in every repo, every night (a secret fails; a Trivy finding is reported, AD #38); the backend build with every test, the timing tests and coverage; each frontend’s static checks and build (Biome, typecheck, folder boundaries, test-hook coverage, production build, bundle budget, the work-marker, skipped-test, fixture-leak, stub-sign-in, record-script self-test and Knip (advisory) checks, and the capture app’s live comparison of its copies of forge-et-fe’s files), the backend’s Fast-lane guard scripts, build record self-test, DDL drift and jar fixture check after its build, each frontend’s Vitest suite with coverage, and its browser gate in the full profile (.claude/lib/nightly/run-nightly.py, .forge/design/harness/test-run-cost.md N-3, S-4). A suite that passed and has not moved since is skipped (the scans never are), a red one runs again. It is the only run of every test and of every browser spec at every size for main, and it catches two PRs that merged cleanly but break each other’s tests or checks |
a nightly-red issue in the repo, and one comment on the harness nightly-log issue |
new work on main — a failed nightly run is fixed first |
| On demand (CI) | T1 + T2 in every app repo’s Slow lane, on a manual run that asks for them (a push to release/* runs the Fast lane only: its PR’s record ran every test; a Dependabot PR gets a local record like any other PR, test-run-cost.md N-5) |
as per-subtask, in CI | nothing — a run someone asked for |
Merge to main |
deploy-dev, then the E2E smoke tier against the dev environment, run by hand by the delivery-team member whose merge or manual dispatch started the deploy, from their own machine, once it finishes — no CI job (read-only — see architecture.md § CI/CD Pipelines, AD-O-11 as revised by Rev 68; F38-g) |
pnpm e2e:smoke in forge-et-fe (F38-g; its CLAUDE.md § Common commands) |
new work on main — a failing smoke run is raised as a bug and dev is fixed first |
| Release tag | Before tagging on main: CI’s on-demand Fast lane run on the commit (-f tests=false), and a nightly full run that passed every suite of each released repo on it (.claude/rules/release-and-branch-protection.md § Release Process, the lead’s rule of 2026-11-02). After the deploys, UAT and production: the E2E smoke tier, run by hand against the environment after the deploys (production’s with a role that reaches no records), with the release track’s regression cycle as the full check; no browser tier runs inside deploy-uat or deploy-prod (F38 OQ-D; AD-O-11 as revised by architecture.md Rev 72 and Rev 81; F38-i decision 3) |
the smoke suite against the deployed host (F38-g; production only with its explicit flag, F38-g Rev 2) | the release |
The smoke tier is read-mostly, and that is a design constraint on the specs — not an implementation note. It runs against the shared development environment while people are working in it, so it creates only uniquely-named records and cleans them up, and it never reseeds — the reset-and-reseed rule below is incompatible with a shared environment mid-workday. Write smoke specs to that constraint; do not reuse a wave-scoped spec unchanged (
architecture.md § CI/CD Pipelines, AD-O-11).
Coverage Targets
Coverage is a guardrail, not a goal — high coverage of trivial code is worthless; low coverage of critical paths is dangerous. Targets: business-logic ≥ 80%, interactive UI ≥ 70%, critical-path acceptance criteria 100% mapped to a test. What must always be tested regardless of any percentage is the next section.
How the numbers are enforced — read coding-standards.md §7.4, which owns this policy. Three points matter enough to restate, because a target quoted without them reads as a live gate:
- Measured per package, never on the aggregate. A repo-wide percentage is dominated by scaffolding and says nothing about whether the quote-approval rules are tested. JaCoCo
violationRulesscope to business-logic packages; Vitest thresholds scope tosrc/features,src/**/hooksandsrc/ui. - Reported through Phase 1, gated from Phase 2. The denominator is still moving during Foundation and Phase 1, so a threshold set now is either meaningless or obstructive. The report is a CI artifact read at wave review;
violationRulesswitch on at the Phase 1 → Phase 2 boundary, against the measured baseline. - Coverage never substitutes for the always-test list above. 100% coverage of a component with no assertion about role scoping is worth nothing.
Coverage tooling is wired at FS-4, not before (coding-standards.md §7.2/§7.3): JaCoCo (backend) and Vitest --coverage (v8) (frontend). FS-1’s ./gradlew check is Spotless + -Werror + doclint + ArchUnit + the Spock specs — since FS-4 it also produces the JaCoCo report (build/reports/jacoco), report-only until the Phase 1 → 2 boundary, so a green check still carries no coverage guarantee until the thresholds are set.
The Always-Test List (no exceptions)
Every row is anchored to a locked decision or a PRD clause — none of them is a preference, and none is waived by a green coverage number. A plan whose ## Test Approach touches one of these areas and has no row for it is incomplete, and that is a /forge-plan-review finding.
| Must be tested | Anchor | The obligation, concretely |
|---|---|---|
| AI never acts on its own | PRD § Governance (human-in-the-loop) | No path sets or approves a price, approves compliance, clears export, or sends to a buyer or the ERP without a recorded human approval — including sequence steps and drafted emails. Assert the absence of the autonomous path, not just the presence of the review queue. This is the highest-consequence rule in the product; it gets tests at whichever tier can prove it, and at least one at T2. |
| A permission check names the permission, never the role | AD #26 | The sub-floor gate asks user.can(APPROVE_SUB_FLOOR_QUOTE) — catalog key quote.approve_sub_floor (AD #26’s own example). The test that matters is the negative one: a role holding every Settings page’s key (AD #41) — the seeded IT Admin — but not quote.approve_sub_floor must not be able to approve a below-floor quote. An administrator who wants to approve must first grant themselves the key, which is a recorded permission change (AD #34 allows it; AD #36 records it). A role-name check would skip that grant, so the approval would look legitimate and the audit trail would show no grant to explain it — the silent widening AD #26 exists to prevent. |
| The sub-floor pricing gate | PRD § Security; F7 | A quote below the gross-profit-% floor cannot proceed without the approval permission. Decided server-side, never client-side (coding-standards.md §3.3) — a gate evaluated in the browser is an approval-bypass risk, not a display bug. |
| AuthN / AuthZ and data scoping | AD #15, AD #34, AD #35, AD #37 | Every owned-entity key resolved for a user with nobody below them, for a manager with a multi-level subtree, and with the entity’s all key — each with and without a share; <entity>.all refused without the entity’s view or manage; manage implies view; an out-of-reach single-record read is 404 (F1-a FR-O4), a write on a record in reach without manage is 403, and lists filter — never 403 for scope. Against a real database, not mocks. |
| The list rule and the per-record rule agree | AD #35, AD #37 | One case table drives both the list query and the single-record check — at minimum the lead’s worked example (users u1–u7, leads l1–l4, accounts a1–a3, with its expected results, recorded in the F5-f spec’s Acceptance Criteria), which also drives the sharing row below — and each case must give the same answer on both paths: owned by me; shared with me; owned by someone below me; shared with someone below me (the roll-up); owned by someone outside my tree; unowned (visible only with the entity’s all key); shared with someone who has since been deactivated (inert — reaches nobody, and counts again on reactivation); out of reach. The worked example gives the same results under AD #37’s scope-free grants as under the scopes it was written for. Children (a contact, note or attachment) answer as their parent does. Two paths tested from one table is what stops them drifting — the F1-a interim (list treats team as own) is the failure this row exists to catch. |
| Sharing: eligibility, transitivity, authority | AD #35 | The picker offers only active users whose role holds the entity’s view key and who do not already reach the record; people who already reach it are not returned; the share operation refuses an ineligible user server-side, not only in the picker. A user who reached the record through a share and holds manage can share it on, can remove any share and can reassign its owner (confirmed by the lead 2026-09-30); a view-only user can do none of the three. A share never grants authority: a sharee whose role lacks manage still cannot edit. Reassigning the owner keeps existing shares, and the previous owner loses reach unless shared back. |
| The last-admin invariant, at all four points | AD #34 | Each of the four actions is refused when it would leave no active user holding both settings.users.manage and settings.roles.manage (AD #41; one settings.manage from 2026-10-26, access.manage before): deactivating a holder, changing a holder’s role, editing a role to drop either key, and deleting a role. Test each one at T2, plus the allowed case: the last admin moves to another role that also holds both keys. A role held by any user — active or deactivated — cannot be deleted. |
| Nobody deactivates themselves | AD #34 | A deactivate request on one’s own user is refused server-side, whatever keys the role holds. |
| Deactivation ends sessions, and waits for reassignment | AD #34, AD #35 | After a user is deactivated, their next request is 401 — including a second open session in another browser — and a fresh sign-in is refused. Deactivation is refused, with the list, while the user owns any lead or account (deals, quotes and samples once they exist), and while they have direct reports. A reactivated user can sign in again. |
| Privilege changes apply on the next request | AD #34 | Remove a key from a signed-in user’s role: their very next request needing it is refused, with no sign-out (no grant cache). Add a key: the next request succeeds. Frontend: after a 403, and when the window regains focus, the app re-reads /me and the navigation and guards reflect the new permissions (design/ui/design-system.md §9). |
| The audit event is in the same transaction | AD #36 | Every write records its audit_event row. A change that fails after the audit call leaves no audit row, and an audit write that fails rolls the change back. Security events — sign-in success and refusal, sign-out, deactivation and reactivation, role and permission changes, share, share removal and reassignment, exports — are written; reads write nothing. Background jobs write as SYSTEM. |
| Every state transition — valid and invalid | PRD; F6/F7/F26 | Sample-round lifecycle, quote readiness gate, price publish/expiry. Every legal move, and every illegal move rejected. |
| Queue consumers are replay-safe | AD #13 | Deliver the same message twice with the same idempotency key → assert one side effect. AD #13’s own words: an outreach step sent to a buyer twice is unacceptable. SQS is at-least-once, so this is a certainty to test, not a risk to consider. |
| Effective-dated rows are never updated in place | AD #14 | A price row is inserted, never UPDATEd in its figures or validity (AD #14’s two exceptions: a cost row starting today or later may be deleted when a new cost replaces it, F26-b FR-W4; a published price starting today or later may have its status set to withdrawn, F26-f FR-10); an as-of query returns the price that was in force then, not the current one. Price history and as-of quoting are PRD requirements and cannot be retrofitted. |
| No in-process scheduling | AD #12 | All timed work goes EventBridge Scheduler → SQS → worker. A build/arch check that @Scheduled appears nowhere — with more than one ECS task it fires on every task, which means duplicate buyer-facing email. |
| The architecture guards hold | AD #31 / FS-14 | The ArchUnit suite is this test — web → service → repository layering, feature-package acyclicity (platform/ui reach no feature), @NullMarked presence, no schema qualifier on an entity, and no production class reaching a fixture package. It runs in Slow lane, and weakening a rule to make a build pass is an architecture change → ASK FIRST. (FS-14 retired the split’s boundary + banned-vocabulary rules; these hygiene guards are what remain.) |
| Never show the user something untrue — all four shapes | coding-standards.md §0.1 |
① Runtime freshness — change a value through the API, assert the UI reflects it with no rebuild or redeploy (mandatory on every pricing and quoting surface). ② No fabricated fallback — fixtures structurally unreachable outside local/test; the guards are ArchUnit (§7.2), dependency-cruiser (§7.3) and the built-artifact FIXTURE_ grep (§7.6). ③ Empty ≠ error — three distinct states with three distinct messages, and empty-because-filtered distinct from empty. ④ The server owns every number that matters — the client displays, never computes. Only ① was previously listed; all four are obligations. |
| Form validation, client and server | PRD | The client check is UX; the server check is the boundary. Both tested; the server one is the one that must exist. |
| Pagination, filtering, sorting | FS-10 conventions | Including the boundaries — first page, last page, empty result, and a filter that legitimately matches nothing. |
| Every new API endpoint | — | At minimum: happy path, 401 unauthenticated, 403 out-of-scope, 400 validation failure. |
Critical Flows & the Smoke Set
There is deliberately no hand-written list of critical flows in this file. The flow inventory is derived from feature acceptance criteria, not authored separately — /forge-release derives candidate critical flows from ACs, and the agreed list for a given cycle lives in that cycle’s plan/report under .forge/release/<cycle-id>/ as a flows[] checklist (.claude/rules/tracker.md § Release & UAT Lifecycle).
That is a decision, not an omission (made before Gate 3, when no feature had acceptance criteria). Features now carry their acceptance criteria in their specs, and a hand-written list here would compete with the list /forge-release derives from them — with no way for a reader to tell which one is current. Same failure mode as L-005.
What belongs here instead is the criteria — those do not go stale:
Choosing the smoke set (post-deploy-dev, run by hand)
AD-O-11 commits the smoke tier to “login plus one critical path per pipeline, budget under 5 minutes” against the shared development environment. Which paths is settled by F38-g’s plan (it was left to FS-4, which had no deployed environment). Choose against these rules:
- Login first, always. An auth regression makes every other result meaningless, so it fails fast and first.
- One path per pipeline, and it must be the one whose breakage is silent. Prefer a flow where failure is invisible in the UI — a price that reads stale, a permission that silently widens — over a flow that fails loudly. A loud failure is reported in minutes by a person; a silent one is not.
- Read-mostly, uniquely named, self-cleaning, never reseeds — the constraint under § When Tests Run. This is what makes a smoke spec different from a wave spec, so do not reuse a wave spec unchanged.
- Under 5 minutes total. If the set outgrows the budget, drop the loudest-failing path, not the newest one.
- Cross-wave breakage is the target. Two waves that pass independently can break each other once both land on
main— that, and deployed-environment failures (migrations, CORS, OAuth redirect URIs, CloudFront SPA routing), are the only things this tier exists to catch. Anything a local run already proves does not belong here.
Choosing the full T-E2E set
Every AC describing user-visible behaviour, across every fixture role — that mapping is enforced at plan time by /forge-plan-review TC-3 and at run time by /forge-test-verify, so it is a derived obligation too, not a curated list.
Test Data & Isolation
A deterministic seed dataset shared across T2/T3/T-E2E keeps tests reproducible.
The roles the fixtures must cover
Named, because “all roles” is not something a fixture author can act on. No role is predefined (AD #34) — the only seeded role in any environment is IT Admin, holding every Settings page’s key (AD #41) — so the fixture roles are sample data in the laptop dataset (R__local_seed.sql, the local profile only — tests seed their own fixtures — and never in a cloud database), written by F5-k on the AD #34 keys and modelled on the recommended day-one set. Pre-authenticated storage state exists per fixture role:
Sales Rep · Sales Manager · R&D / Sample Owner (Taster) · Administrator · Tea Room · Executive / Management — plus the seeded IT Admin (crm01@empire.lk locally) and a sample all-permissions test role, which every E2E suite can sign in as instead of the retired superadmin.
A fixture role is named after the recommended role it imitates, but a test asserts on the keys the role holds, never on its name — a test that passes because a role is called “Tea Room” proves nothing about a configuration Empire can rename.
The Managing Director is not a seventh role — sub-floor approval is a permission that someone holds, and Empire may operationally attach it to an Administrator (the recommended configuration gives it to a Managing Director role only; nothing in code prevents granting it elsewhere — AD #34). Build the fixture as a user holding quote.approve_sub_floor, never as a distinct MD role, or the fixture bakes in exactly the role-equals-permission conflation AD #26 exists to prevent — and then the always-test-list row above cannot be written.
Seed dataset — mechanism settled; UAT’s data entered by hand
- Mechanism (settled at FS-2): a
local-only Flyway migration location, structurally unreachable from a cloud profile — so seed data cannot ship to an environment a client touches (coding-standards.md §5; FS-2 FR-E5). AD-O-8is answered for the temporary UAT environment (architecture.mdRev 72; F38-h decision 5): the IT Admin enters UAT’s data by hand through the app after each apply, from.forge/plans/F38-h/uat-data-checklist.md; no seed or script writes to a cloud database. The local seed stays the laptop dataset.- For the local seed (and UAT’s data from Phase 2’s round): whatever is written must exercise the role permutations above, the price-history dimension (a quote whose snapshotted price differs from the current published one) and both sides of every gate — a seed set where no quote is below the floor cannot test the floor.
Isolation
| Tier | Strategy |
|---|---|
| T1 | Fully mocked; no shared state, no container, no context. |
| T2 | Rolls back per test, or authors and scopes its own keys against the shared container (see the T2 operating rules — the container is shared by design, so isolation is the spec’s job, not the container’s). |
| T3 / T-E2E | Reset + reseed between test files; parallel files use disjoint data (different users, accounts, quotes) so they cannot contend. |
| Smoke tier (dev env, run by hand) | The exception — never reseeds. Read-mostly, uniquely-named records, cleaned up after; F38-g’s smoke account holds only view keys (and all keys that widen them), so it writes nothing. See the note under § When Tests Run. |
⚠️ Shared-fixture hazard (from real engagements). In a shared test-DB/container singleton, never assert an absolute
COUNT(*)on a table other tests mutate — it is an order-dependent failure that detonates in a sibling work item’s wave. Assert>=, scope the count to your authored keys, or have mutating tests clean up. When sibling WIs in the same wave touch shared test infrastructure (e.g. a reused DB container), disable container reuse for the parallel run.
Accessibility & Security
Accessibility
WCAG 2.1 AA is the PRD’s stated target for core screens. Tested at two levels:
- Component (T1) — semantic HTML, ARIA, and keyboard navigation of lists, boards, forms and the calendar (the PRD names these specifically). Status is never conveyed by colour alone: age, priority, severity and rotting indicators carry text labels too, and the warm palette must clear contrast.
- Page (T3/T-E2E) — an automated
@axe-core/playwrightscan on every page visited during E2E. Not optional.
An axe pass is necessary, not sufficient — do not read it as “AA achieved”. Automated tooling catches a minority of WCAG failures, and it is weakest exactly where the PRD is most specific: axe can confirm that a control is focusable, but not that tab order is sensible, that a Kanban board can be operated without a mouse, or that focus lands somewhere useful after a dialog closes. The PRD names keyboard navigation of lists, boards, forms and the calendar as the requirement, and that is a manual walkthrough, not a scan. So: axe on every page (mechanical, blocking), plus a keyboard-only walkthrough of each core screen recorded as evidence when that screen ships. A screen that passes axe and cannot be driven from the keyboard has failed the actual requirement.
Know the gap in the lint layer. Biome’s whole a11y group runs as build errors, but it is markup-shape linting, not an accessibility audit — it cannot see an input with no label at all, because its rule works from the label’s side (finding F-004, FS-1). The assigned closure is the automated axe pass landing with the FS-4 Playwright toolchain, which sees rendered output where an unlabelled control actually manifests. Until FS-4, this specific defect class is a reviewer obligation, and both frontend CLAUDE.md files say so. Do not read a green pnpm check as an accessibility pass.
Test hooks. Every interactive element is addressable: by role and accessible name first, and by a data-testid where semantics cannot address it uniquely (Kanban cards, virtualized rows, icon-only buttons — coding-standards.md §4). A testid is named <feature>-<element>[-<qualifier>], added when the component is built, stable across refactors, and never derived from user data. check-testid-coverage.mjs (from FS-3) mechanically checks addressability — each interactive element has a testid, an ARIA label, a static accessible name, or a written exemption; testid naming is a review check, which the script does not attempt; the full rule is coding-standards.md §4. Prefer getByRole/getByLabel in the test anyway (see the selector conventions above) — data-testid is the fallback for elements semantics cannot uniquely address, not the default.
Security
The security home is .forge/security/ (its README plus the Trivy suppression-file reference template — AD #9 as refined by AD #27). No threat-model document exists yet — an earlier version of this section pointed at .forge/design/threat-model.md, which has never been written. Until one is, the standing inputs are the PRD’s § Security and Compliance and coding-standards.md §7.7.
Test each concern at the tier that actually proves it:
| Concern | Tier | What to verify |
|---|---|---|
| SQL injection | T2 | Bound parameters / typed repository methods; hostile input in a search or filter field cannot alter query structure. |
| XSS | T1 (FE) + T3 | Buyer-supplied text (account names, inquiry bodies, notes) renders escaped; no raw innerHTML / dangerouslySetInnerHTML. |
| Auth bypass | T2 | An unauthenticated request to any protected endpoint returns 401 — asserted per endpoint, not assumed from the filter chain. |
| Privilege escalation | T2 | A role lacking a Settings page’s key cannot reach that page — settings.users.manage for user and hierarchy administration, settings.roles.manage for roles (AD #41); a role holding only view keys changes nothing (read-only, as the recommended Executive / Management role is); a role lacking quote.approve_sub_floor cannot approve a below-floor quote. Every key tested with and without the entity’s all key, for a user with and without people below them, and with and without a share, not the happy path only. Self-escalation is allowed by design (AD #34) — a settings.roles.manage holder may grant any key to any role, their own included — so the test there is that the grant is recorded in the audit trail, not that it is refused. |
| Role ≠ permission | T2 | The AD #26 negative test — holding a role does not confer a permission that role was never granted. |
| Session & CSRF | T2 seam | Server-side session (Spring Session JDBC); a mutating request without a valid CSRF token is rejected; cookie flags as configured at FS-6; deactivating a user deletes their sessions (the always-test row). |
| Sign-in matching | T2 | A verified email that matches no user, or matches a deactivated user, is refused; nothing is auto-provisioned; after an email change the old identity link no longer signs in, and the next sign-in re-links by the new address (AD #34). |
| Secrets in the bundle | T1 / CI | The frontend bundle is public — assert no secret reaches it. gitleaks in the fast lane covers the repo; the bundle needs its own check (AD #22). |
| File upload | T2 | Type and size validated; virus scan gates visibility; the signed download URL is scoped to the record’s visibility (FS-11). |
| AI data handling | T2 | No PII beyond what the feature needs leaves the system, and AI request/response bodies never appear in production logs (proposal A.6.3; coding-standards.md §8 deny-list). |
| Dependency CVEs | CI (Fast lane, on a ready PR) |
Trivy, HIGH/CRITICAL ≡ CVSS ≥ 7.0, report-only on every ready PR, release/* push, manual and weekly CI run, and the nightly full run on the lead’s Mac (the dev deploy runs none; ECR scans each dev image on push — AD #38 as restated 2026-11-03); a UAT or production release stops on an open finding without a .trivyignore.yaml exception (AD #9 / AD #27 / AD #38; F38 FR-35). A standing gate — confirm green, don’t re-decide per feature. |
Browsers & Viewports
The matrix (lead decisions, 2026-09-27; built by F40 — each size is a Playwright project named size-<w>x<h>-<engine>, and pnpm e2e:gate runs them all):
| App | Sizes a browser suite runs at | Engines |
|---|---|---|
CRM (forge-et-fe) |
tablet portrait 768, tablet landscape 1024, desktop 1280 and 1440 — plus 320px reflow (WCAG 2.1 SC 1.4.10; zoom support, not a phone design) | Chromium; WebKit for the tablet sizes (iPad) |
Capture app (forge-et-pwa) |
phone 360, 390 and 430 (touch), phone landscape 844 × 390 (the orientation check), tablet 768 portrait and 1024 landscape, desktop 1280 — plus 320px reflow | Chromium and WebKit (Safari on iPhone and iPad) |
At every size, every page gets: the axe scan (WCAG 2.1 AA, plus the 2.2 AA target-size rule) — except the CRM’s 1440 × 900 Chromium size, which repeats 1280’s desktop band in the same engine and runs the checks below without it (.forge/design/harness/test-run-cost.md P-2); no horizontal page scroll; no interactive control clipped outside the viewport; targets at least 24 × 24px (CRM) or 44 × 44px (capture app); and no focusable element inside an aria-hidden region (use inert on anything hidden but still in the DOM). These browser suites run in full every night on main, in the nightly full run on the lead’s Mac, and on a PR into release/*; a PR into main runs none of them (Rev 12, Rev 35; PRD Rev 81; .forge/design/harness/test-run-cost.md S-4, N-3). pnpm e2e:gate runs the matrix locally (both need the local backend running: the CRM’s for its whole suite, and the capture app’s, since F36-a, for its seam project — the specs that run against the real backend through the app’s Vite /api proxy, in Chromium, once — while its chromium and size projects mock the API) and records a result for the commit it ran on; a developer runs pnpm e2e:gate --selected or one spec while developing. A ready PR into release/* is blocked without a passing full-profile result, by the last step of the Fast lane’s full run, which reads the Browser gate: line in the PR description. No CI job runs a browser — the post-deploy smoke and tag tiers (AD-O-11) are unchanged. Built by F40.
How the gate works (the one statement of these rules; .claude/CLAUDE.md § Quality Gates and each app repo’s CLAUDE.md restate them briefly):
-
The gate before ready — a PR into
release/*only. A PR intorelease/*inforge-et-feorforge-et-pwais marked ready — or opened ready — only with a passing full-profilepnpm e2e:gateresult that covers its head; a PR intomainneeds none (Rev 35). The rest of this list describes the gate wherever it runs: on such a PR, in the nightly full run, and when a developer runs it. The gate refuses to start on uncommitted changes to a gate input, runs the suite at its scope (every spec, theseamproject and the size projects — the CRM’s PR profile runs four of the seven sizes, and on Windows no WebKit; the full profile runs every project,forge-et-fee2e/README.md § The browser gate) with one retry, writes the result file.browser-gate/result.json(gitignored) stamped with the commit, and prints one line for the PR description:Browser gate: passed · <short sha> · <n> passed · <k> skipped · <m> known failures, followed by· 4 sizesor· 4 sizes, chromium onlyfor the PR profile and· <n> flakywhen a test passed only on its retry. A PR intorelease/*needs a full-profile line, which theFast lane’s check enforces. A skipped test fails the gate unless it is onscripts/browser-gate/expected-skips.json. -
What a result covers. A result covers the head when it passed on the head itself, or on an ancestor of the head whose diff to the head touches no gate input. The gate inputs are listed once, in
scripts/browser-gate/gate-inputs.json, which the gate and the CI step read throughscripts/browser-gate.mjs. That file andscripts/browser-gate/expected-skips.jsonare always gate inputs, whatever any version of the list says, so a change that narrows either one needs a gate run. Markdown files are not gate inputs, so a docs edit needs no new run, and neither are Vitest test files (*.test.ts,*.test.tsx;.forge/design/harness/test-run-cost.mdP-8) or the local test record’s scripts (local-test-evidence.*,fast-lane-scope.*,test-selected.*,vitest-run-facts.*; P-10): no browser test loads one, whichpnpm check:boundariesenforces. (In the capture app a record-script change is a re-copy that also changes the shared-file hash list, a gate input, so it still runs the gate there.) A dependency-only change (onlypackage.json,pnpm-lock.yamlor both) touches no gate input when the part of the lockfile the gate can run is unchanged: the packages reached from the direct dependencies that are on neither ofgate-inputs.json’s two package lists (typesOnlyPackages,toolsNotRunByGate), their resolutions and the links between them (F41; design record C-10). So a bump of a test runner, a linter, a git-hook tool or a type package needs no gate run, and a bump of anything the gate loads still does.pnpm check:boundariesfails if a file the gate loads reaches a listed package. -
On a PR into
release/*, the push that marks a draft ready goes throughpnpm e2e:gate --full --push, and so does the last later push that changes a gate input. A push to a draft is a plaingit push, and so is a push to a ready PR while more pushes will follow. Commit, thenpnpm e2e:gate --full --push: it runs the gate on that commit, then pushes and writes the newBrowser gate:line into the open PR’s description in one step; then runpnpm test:record(theTest recordcheck the head needs before merging). The push’s CI run is cancelled within seconds by the description edit’s run, so each stamped push costs about one full CI run. Running the gate after a plain push costs two (the push’s run, which fails at the description check, and the edit’s) — avoid it. -
Draft, then ready. A PR still changing opens as a draft: on a draft the
Fast lanestops after the static checks (steps.scope.outputs.full == 'false'), so the description check does not run. On a PR intorelease/*, oncepnpm e2e:gate --full --pushhas passed on the final head and the local test record (pnpm test:record) is on it, mark it ready (gh pr ready), which runs the fullFast laneonce; a PR intomainis marked ready aftergit push, the test record and, in these two repos, the selected browser run (pnpm e2e:gate --selected, its line in the test evidence;fewer-nightly-reds.mdR-1). -
The skip line. On a PR into
release/*the one exception isBrowser gate: skipped — <reason>(an em dash and a reason holding at least one letter or digit, BUG-046) in the PR description, used only when the developer decides it for that PR — for example a hotfix on arelease/*branch cut before F40, which has noscripts/browser-gate.mjs. It is treated as a review skip record is (.claude/CLAUDE.md § Quality Gates): a stated reason, visible on the PR. -
The shared check and the page-state list. Every axe scan goes through
e2e/support/pageCheck.ts(expectPageClean, which also holdsexpectNoAxeViolations), run on every entry ofe2e/pageStates.ts(PAGE_STATES) under every size project. A new route, or a new query-string value or data state of an existing route, needs aPAGE_STATESentry in the PR that adds it: a unit test fails on a router path with no entry, and/forge-pre-pr-reviewasks for a new state of an existing path. An accepted axe finding is anAXE_EXEMPTIONSentry ine2e/support/axeExemptions.tswith its reason. -
Known failures. A defect the gate finds in an existing screen is filed in
.forge/bugs.mdand marked withmarkKnownFailure('BUG-NNN', '<reason>'), the onlytest.fail()call site, so the gate stays usable; the result file and theBrowser gate:line count known failures, and one that starts passing fails the run, so the fixing PR removes the mark. -
Chromium is the primary target — every T3 and T-E2E run. WebKit joins as in the table above (decided 2026-09-27; it replaces the earlier “undecided” note). Firefox stays out unless a defect points at it.
-
A functional browser test can also run in WebKit without a new project when its acceptance criterion names WebKit: the spec file sets
test.use({ ...devices['Desktop Safari'], browserName: 'webkit' })at its top level and runs under thechromiumproject, so the gate runs it unchanged. The first such file is the CRM’se2e/browser/users/offboarding-keyboard-webkit.spec.ts(F5-h AC-17). In WebKitTabmay skip links, as Safari does by default, so a keyboard walk there checks whetherTabreaches a link and otherwise usesAlt+Tab. -
(Superseded 2026-09-27 by the matrix above:) whether Firefox and WebKit join the T-E2E suite was undecided at FS-4; it is a real cost against the 10-minute budget, so it is a decision, not a default. If it wants formal tracking, it belongs as an
AD-Orow inarchitecture.md § Open Decisions— it is recorded here so it is not silently skipped. -
CRM viewports: desktop and tablet, plus 320px reflow. The PRD commits to “usable on desktop and tablet”; a phone design is explicitly deferred, but every page must reflow at 320px (above). Core screens get a tablet-viewport assertion, not just a desktop one.
-
The capture PWA is phone-first and carries its own matrix — plus the preview-not-dev rule for anything service-worker related (see § Test Naming & Layout). It is also supported on tablet and desktop (design-system Rev 28), so its matrix runs phone, tablet and desktop sizes (table above). The capture app’s browser layout (F40):
forge-et-pwa/playwright.config.tscarries the phone, tablet and desktopsize-<w>x<h>-<engine>projects, Chromium and WebKit, and the capture suites run under them.- The shell’s axe scan is
e2e/browser/shell/a11y.spec.ts, mirroringforge-et-fe’s layout, and the shared checke2e/support/pageCheck.tsis a byte copy offorge-et-fe’s (listed inscripts/shared-with-fe.json). - Each new capture screen area adds its own
<area>-a11y.spec.ts(axe, WCAG 2.1 AA) with the 44 × 44px target size asserted, exactly as the CRM does per feature, and its page states joine2e/pageStates.ts. Accessibility is held to the same bar in both frontends.
-
Dark mode is not tested, because it is not in V1 (
design/ui/design-system.md). The token layer exists to make it cheap later; adding a dark-mode assertion now tests something we do not ship.
Performance Baselines
From the PRD’s § Performance. These are targets, not CI gates — performance tests are not part of the mandatory gate and run on demand when a performance-sensitive area changes (pricing reads, reports, list views at volume, imports, AI batch). They are written down because an unwritten target is unverifiable at UAT, which is the first place a client measures it.
| Metric | Target | How to verify |
|---|---|---|
| Interactive screens — lists, boards, detail views, calendar | < 2 s load under normal data volumes | Browser timing assertion at T3 against the seed volume |
| Kanban drag-and-drop, filters, calendar interaction | < 200 ms UI response (“feels instant”) | Interaction timing at T3; a regression here is usually an unnecessary refetch |
| Synchronous AI calls — extraction, summary, brief, quote flags, drafts | < ~10 s | T2 with a timing assertion against the AIProvider seam, tagged @Tag('performance') so it runs in the nightly full run and in CI, never in a developer’s local build (.forge/design/harness/test-run-cost.md B-4; a bound on elapsed time fails on a loaded laptop, L-125) |
| AI batch work — e.g. 200-lead triage | Runs as a background job with a review queue, never a user wait | T2: assert the request returns immediately and the job completes out of band. AI never sits in a hard user wait for correctness |
| Email send | Respects the provider ceiling, order of ~40 actions/min/account | T2 against the rate limiter |
| Concurrent users | 10–20, thousands of accounts/deals/leads | Seed volume sized to this; no standing load suite in V1 scope (a one-off concurrency check per UAT round is proposed, § System Verification to V1 Sign-off) |
| F23 import | Caps at ~50k rows/file; batch, validated, re-runnable and rollback-safe | T2: seed a large file, assert the job completes, the file is well-formed, and a re-run is safe |
Measuring a response-time limit (lesson L-084). A developer laptop’s own per-request cost can dominate a fast endpoint: in F26-a, endpoints the feature did not touch took 43–50 ms on the laptop, and two readings of a 100 ms limit each breached by about 10 ms while the database part of the read took 0.2 ms. So:
- A spec that sets a response-time limit states where the binding measurement is taken — a deployed environment (dev, once F38 builds it) — and treats a laptop reading as evidence, recorded in the plan’s
## Progresswith any reading over the limit stated as such. Until the environment exists, the binding measurement is a row in.forge/cloud-gated-validation.md§B. - Every timing session measures a baseline beside the new endpoints: an endpoint the feature does not touch, called the same way, with the calls interleaved round-robin. The difference is the feature’s cost; without it a breach cannot be attributed.
- Warm every shape before measuring any, then take 20 measured calls per shape and read the 19th, so one pause does not decide the result while a slow endpoint still fails. On Windows call
127.0.0.1, notlocalhost, which added 7–18 ms per call in F26-a. - Take no other load on the machine during a reading. A timing measurement runs under the machine’s heavy-run lock, through
python3 .claude/lib/heavy-run.py -- <command>(.claude/rules/test-record.md§ One heavy run at a time; lessons L-085, L-315).
Non-Deterministic & External Surfaces
Three surfaces cannot be tested the way the rest of the system is, because their answer is not a fixed function of their input: the AI layer (a model, not a function), Microsoft Graph (someone else’s live server), and the Pipedrive migration (one real attempt on the client’s live data). Left unaddressed, each becomes either an untested hole or a permanent source of red builds. The shape below is what each one needs; where a decision is genuinely still open it says so rather than picking one quietly.
The AI layer
No test calls a paid external service — Amazon Bedrock, Exa or any other per-call API — in any tier or any repo. This covers unit, integration, seam and browser tests, the browser gate, the nightly run, CI, and a test run on a laptop that holds AWS credentials. Tests run under the local or test profile, which wire StubAiProvider and StubSearchProvider. A test that needs the real provider class (a cloud-shape test) builds it on a faked SDK client, as LiveEvaluationIT and BedrockInvocationRecordIT do. In forge-et-be the build also enforces it: the Gradle test task, which the build record, the nightly run and the Slow lane all use, gives the test JVM dummy AWS keys, no profile, session, role or shared config file, and a blank Exa key, so a test that misses its fake fails to authenticate instead of being billed (TestJvmCredentialsSpec). A test that removes a stub, swaps one for the real provider, or builds an SDK client without a fake is a Blocker at review. The one run that calls Bedrock is ./gradlew liveEvaluation, which refuses inside CI and without its confirmation. Why: Empire Teas pays for every Bedrock and Exa call, the account’s Bedrock quota is 10 requests a minute, and a suite runs thousands of tests (the lead, 2026-10-28).
Never assert model output quality in a blocking gate. A non-deterministic assertion in a test suite CI runs (each app repo’s Slow lane) is a permanent flake that trains the team to re-run until green, which costs more than the coverage it buys. Split the problem in two:
What is deterministic — and therefore must be tested (T1/T2, AIProvider stubbed):
| Assert | Why it matters |
|---|---|
| The request shape sent to the seam — model, prompt id, the fields included | The deny-list is here: no PII beyond what the feature needs (PRD § AI data handling). |
| Malformed or hostile model output is rejected by schema validation, not rendered | The backend’s ai package carries schema validation: ai.service.AiOutputSchemaValidator checks every answer against its prompt’s output schema, the call runner retries a failed answer once and then ends INVALID_OUTPUT, and the feature’s own boundary rules treat what passes as untrusted text (F12). A model returning nonsense must produce an error state, never a plausible-looking screen — §0.1’s fabricated-fallback shape with a novel source. |
| The output lands in the human-in-the-loop queue and cannot self-approve | The § Always-Test List’s first row. Assert the absence of the autonomous path. |
| Provenance surfaces in the UI — tagged AI-generated, with confidence and source; the AI-draft → “edited by you” transition | PRD § AI provenance. |
| Provenance is logged per call — provider, model, tokens, cost, latency, outcome | Proposal A.6.2 — this is what makes any output attributable. |
| AI request/response bodies never reach production logs | Proposal A.6.3; coding-standards.md §8 deny-list. Test the logger, not the intent. |
enabled: false cleanly disables the feature |
Per-feature {model, enabled} is runtime config (AD #19) and is a live kill switch — an untested kill switch is not a kill switch. |
| Provider outage degrades to the manual path | AD #19’s stated outage behaviour. The failure mode to prevent is a broken screen; assert the honest error + manual route (coding-standards.md §3.2), not a blank one. |
What is non-deterministic — and belongs to the eval set, not a test tier. Prompt and model quality are scored by the eval set described in .forge/design/ai/ai-provider-strategy.md (“Eval before lock”): built from real anonymised Empire inquiries once the Pipedrive audit yields them, scoring extraction accuracy, draft quality and cost per call across the candidate models. It is a different instrument on a different cadence with a different pass criterion — scored, not boolean; run at Phase 2 eval prep and at any model change; never wired into Slow lane. Treating it as a tier is how a CI budget dies.
Microsoft Graph (mailbox, calendar, contacts)
Graph is the only V1 provider (AD #16) and sits behind the MailboxProvider seam — which is the stub point. Client dependency row 4 supplies one-time admin consent, the test accounts (the provisioned set is listed in design/environments-and-identity.md §5 — the authoritative source; do not restate the count here), and the shared-mailbox confirmation.
Decided by the team 2026-08-25 (Rev 5); FS-6 / the F20 spec implements it, does not re-open it:
- T1/T2/T3 stub the
MailboxProviderseam. That is what the seam is for, and every rule worth asserting (visibility scoping of synced mail, CC/BCC handling, the ~40 actions/min/account rate limiter, envelope-encrypted per-user refresh tokens) is provable against a stub. - Live Graph is exercised against the Empire test accounts manually, recorded as evidence — not as a blocking CI gate. The reasoning, so it can be argued with: a live-Graph CI job needs long-lived Microsoft credentials sitting in CI, which reads badly against the no-static-keys posture that AWS access already satisfies via OIDC → IAM (Graph has no equivalent); it is rate-limited; and a mailbox is shared mutable state across runs. Three independent flake sources for coverage a stub mostly provides.
- Two things a stub can never prove, so each must be exercised live at least once per phase and its evidence recorded: (1) the OAuth consent and refresh-token round trip, and (2) that a sent message actually arrives — which depends on SPF/DKIM on the sending domain, a PRD security clause that no stub can speak to.
- Walk the real Microsoft path early, not only once per phase. The sign-in and mailbox-consent paths are walked against a real Entra account as soon as the credentials exist for the environment being built on (laptop mode, then dev), and again after any change to a sign-in, consent or redirect path — before the work is called done. A test that reaches a step by a shortcut (calling a handler directly instead of going through the filter chain a browser meets) says in a comment which production step it skips. Why: the first day of real Entra sign-ins found five bugs the stub-backed suite had passed (
.forge/lessons.mdL-100). - The email features F20-i, F20-f, F20-j and F20-k are tested this way, in full (the lead asked for thorough testing on 2026-10-17; the design is
design/architecture.md§ Mailbox integration). Against the stub (T1/T2): matching by address and by thread; manual link and unlink with their permission checks; block lists; stored attachments and the size limit; re-linking a message changes nothing; each past-mail trigger (a new record, a mailbox connecting, a bulk import); the automatic connect, a failed connect showing Try again without looping, a deliberate disconnect staying disconnected, losing and regainingemail.manage; a 429 from Graph and the background back-off; and a stub mailbox of tens of thousands of messages so paging, resumption and run time are checked. In the browser (T3/T-E2E and the browser gate): the live inbox, the link and unlink actions, the settings screens, at every supported size. Non-functional, in the same work: security in each PR’s security review (no user reaches another user’s mail or a record outside their reach by changing an id, sanitised HTML bodies, attachment links scoped to the record, tokens never in logs, responses or the browser); data protection (unlinking removes the stored copy; mail content never reaches the logs or the audit trail); reliability (Graph down or slow shows the reconnect or error state; a missed notification is caught by the delta poll); audit events on link, unlink, block and on a connection released by a lost permission. Live on dev, once the mailbox is set up there: each flow walked againstCRM01–CRM05and recorded as evidence; one test mailbox filled with a few thousand emails sent between the test accounts within Microsoft’s sending limits, to time the 12-month scan and the inbox against the 2-second target (§ Performance Baselines method); and 20 parallel inbox requests to check the per-mailbox limiter. No separate load-test tool (§ Performance Baselines keeps a standing load suite out of V1; the per-round concurrency check proposed in § System Verification to V1 Sign-off is the lead’s to decide).
The Pipedrive migration (F4)
AD-O-7 is open, and it is two problems, not one — so what follows is the test shape, and the F4 spec closes it. The reason this needs saying at all: F4 gets one real attempt on the client’s live data, and its failure is discovered by Empire, not by us.
- (a) Replay safety. Run the migration twice and assert nothing duplicates — the PRD requires re-runnable and rollback-safe, and idempotence is the only way to test that claim rather than assert it.
- (b) Delta reconciliation — the part that is easy to miss. Proposal §8 commits a parallel run where both systems are live and Pipedrive remains the system of record, then a freeze window for the final migration. So F4 is an initial load plus a later reconciling pass, and that pass has three cases, each needing its own test: a record already migrated → skipped; a record created in Pipedrive during the parallel run → imported; a record created in the CRM → never overwritten. The third is the one that loses real work if it is wrong.
- A dry-run mode, because the only honest rehearsal of a one-shot job is a full run against a copy of the real data.
- Reconciliation as an artifact, not a log line — row counts and field-level checks before and after, in a migration report Empire can verify against before decommissioning (proposal §8 commits to this, which makes the report itself testable output).
- Test the rollback path, not only the forward one. A rollback-to-Pipedrive position is held open until sign-off; an untested rollback is a plan, not a safety net.
Flaky Tests
Team policy, accepted 2026-08-25 (Rev 5); the FS-4 toolchain wave wires it in. Nothing in the harness said anything about this before, and on a small team shipping browser tests against a December date, an unmanaged flake rate is the single most likely way this suite stops being trusted. A test nobody believes is worse than no test, because it still costs time to run.
The policy:
- Zero retries in a developer’s own runs (
pnpm e2e,pnpm exec vitest,./gradlew test). A flake you cannot see is a flake you will not fix. - One retry in the browser gate and in both records, counted (
.forge/design/harness/test-run-cost.mdF-1 and F-3, the lead’s decision of 2026-10-27, which revises this rule’s earlier “zero retries locally” for those three runs). A test that fails and then passes on its retry counts as passed and is named as flaky: by the gate in its result, on stderr and in theBrowser gate:line (with--pushor--selected, also in a comment on the branch’s open PR marked<!-- browser-gate-flaky, BUG-465), and by the test record (pnpm test:record, Vitest’s--retry=1) and the build record (local-build-evidence.py, theorg.gradle.test-retryplugin with-Pet.testRetries=1) in their PR comments and checks. More than five in one run fail it. The nightly full run retries only in its browser gates, and lists their flaky tests in the nightly log. - Flaky tests are counted before a bug is filed (F-2). The lead runs
python3 .claude/lib/flaky-report.pyonce a week; it reads every published flaky name over the last 14 days (the gate comments, the record comments, the nightly log) and matches each against.forge/bugs.md, closed bugs included. A test that is flaky twice in 14 days, or once on an idle machine (a recorded average below 0.5 busy), gets a bug through the bug flow and the triage below; a single occurrence under load stays in the report. Search the report before filing, so a recurrence of a bug closedwontfixis added to it rather than filed again. - Retries allowed in CI, but a retried pass is never silently green. Record it and investigate — the second attempt passing is information about a race, not a verdict.
- A flake is a defect with a cause. Find it. Never “fix” one by adding a sleep or lengthening a timeout — that converts a race into a slower race and charges the 10-minute budget for it.
- No quarantining.
coding-standards.md §7.8already fails the build on a skipped or.onlytest, for exactly this reason: a quarantined test nobody deletes is a permanent silent hole. - If a flake cannot be fixed quickly, delete the test and record why. An honest gap is auditable; a red-then-green test is not.
- One exception: a failure caused only by an overloaded machine is not a flake in the test, so the test is kept and the bug is closed
wontfix(the triage order below, step 4). The load check in the browser gate and both test records is what stops it recurring.
Triage a flaky failure in this order (harness design .forge/design/harness/load-check-and-flaky-tests.md, D-4):
- Read the failed run’s record. The browser gate’s
.browser-gate/result.jsonholdsload(the CPU busy share and the load average per core at the start, peak and average, and the busiest processes at the peak) andstalls(tests that ran more than three times their timeout); the test records print the same load when they fail. A stall means the test’s process was not scheduled, not that a page or query was slow. On a Mac, first check whether the machine slept during the run:pmset -g log | grep -E " Sleep | Wake "over the run’s window — a sleep whose length matches the stalls is the cause, not a flake (oneforge-et-fegate on 2026-10-22 showed 20 tests stalled 227–953 s against a 30 s timeout, each matching an idle sleep on battery). The gate and both records keep the machine awake while they run (caffeinate -dimson macOS,SetThreadExecutionStateon Windows; designtest-run-cost.mdM-2), which blocks idle sleep but not a closed lid without an external display, so keep the lid open. Run any other long local run (pnpm e2e,./gradlew check) undercaffeinate -dims. - Reproduce it on a machine otherwise within the load limit: under forced load (one busy loop per core beside the test, in every engine — WebKit cannot be throttled through CDP), with the CDP CPU throttle as well for Chromium, and, when neither reproduces it, by holding back the one step that may land late (a route loader, a held response —
.forge/lessons.mdL-162). Start the busy loops only once the run holds the machine’s heavy-run lock ($TMPDIR/et-heavy-run.locknames its process), and stop them when it ends; a run stopped early can leave its app server on the gate’s port (vite previewon 15173), which is stopped too. Loops started while the run waits for the lock hold the CPU above the load limit, so the lock’s holder, the nightly full run included, refuses to start its next suite and neither run moves (.claude/rules/test-record.md). A backend test that waits on an emulator in a container (LocalStack, Postgres) is reproduced by slowing that container instead: freeze it withdocker pauseanddocker unpausein a loop (7 seconds frozen of every 8 reproduced BUG-259), started once the test’s resources exist and aimed only at the container your run started, never another session’s. Busy loops on the host and a CPU limit on the container do not slow it enough (.forge/lessons.mdL-301). - If any of these reproduces it, it is a timing defect in the product or the test, fixed as one.
- If none does, and the record shows the machine was overloaded by work other than the run — it started with
--ignore-load, or the busiest processes at the peak include another gate, build, backend or container — close the bugwontfixbecause the machine was overloaded. ItsResolved:line gives the recorded load, the busiest processes and the reproduction attempts, and the lead is the person who decides; it gets aLesson:line like any bug (.claude/rules/tracker.md § Bug Tracking, step 6). A system-wide measure cannot separate the run’s own load from other work’s, so the process list is the evidence of other work; if it shows none, the bug stays open as a defect to find. A recurrence on a machine within the limit is a new bug that links the first.
The most likely causes on this project, named in advance so they are the first suspects:
- The shared container. T2 specs share a Postgres per test JVM by design, so order-dependence is the built-in hazard — see § Test Data & Isolation’s no-absolute-
COUNT(*)rule. This class of failure characteristically detonates in a sibling work item’s wave, which makes it look like someone else’s bug. - Wall-clock time. The pricing subsystem is effective-dated (AD #14) and the system carries a configured business timezone (
et.business-timezone, required at FS-2). Any test that reads “now” will flake near a validity boundary, a day boundary, or a timezone offset — and will flake differently on a machine in a different zone, on a mixed macOS/Windows team. Inject the clock; never read the system one in a test.StaleClock(FS-13) is the core primitive for the production side of the same problem. - Machine load and operating system. The backend suite runs on each developer’s own machine (macOS or Windows,
team-guide.md § The Team), often beside other Testcontainers suites, because CI does not run it on a pull request. The browser gate and both test records refuse to start while other work keeps the CPU more than 0.65 busy (ET_LOAD_LIMIT;--ignore-loadruns anyway, only when the developer says so) and record the load while they run, so a failure under load can be triaged from its record (the triage order above). They also take one heavy run at a time on a machine: a second gate or record waits for the first instead of running beside it (designtest-run-cost.mdM-5). A case that measures real elapsed time against a fixed bound passes on an idle machine and fails on a loaded one (BUG-077 on Windows, BUG-081 and BUG-082 on macOS). Drive such a bound with an injected clock or a counted step, not real delays (.forge/lessons.mdL-125), and file the flake for the operating system it failed on (.claude/rules/tracker.md § Bug Tracking). - Browser keyboard behaviour that differs by operating system. The browser gate also runs on each developer’s own machine, and Chromium’s native controls do not answer keys the same way on every system: on macOS,
Home,Endand the arrow keys do not change a closed native<select>(they open its picker or do nothing), while on Windows and Linux they do (BUG-103). Change a closed native<select>by type-ahead only (page.keyboard.type('50')), and assert its value after each change; type-ahead is still a keyboard action. Chromium keeps a select’s type-ahead text for about a second, so between two type-aheads on one select move focus off it and back (Shift+Tab, Tab) to start a new search (.forge/lessons.mdL-143). Type into a native<input type="date">in the browser’s part order, never month first: on macOS, Chromium takes the order from the system region, and a Mac with region Sri Lanka puts the year first. Read the order once withdateOrderand type withtypeDate(e2e/browser/custom-fields/values-support.ts), then assert the input’s value (BUG-122, BUG-407). WebKit’s Tab depends on the machine’s settings, not only its system: with keyboard navigation off (macOS’s default), Tab reaches text fields only and Option+Tab reaches buttons and links. A WebKit test that moves focus by key takes the key fromwebkitFocusKey(e2e/support/focusKey.ts), which measures it on the machine, and every focus-trapping shadcn content loops Option+Tab as well as Tab (src/ui/shadcn/loop-option-tab.ts; BUG-231, BUG-236, BUG-241). A browser test that relies on any other key or focus behaviour that differs by system or engine is proved on Windows and on macOS before it merges, or states in the test which system proves it and marks the other with a known failure in the same PR (.forge/lessons.mdL-168). - A react-aria combobox list closed by a page scroll, or opened before its options loaded. react-aria closes an open
Comboboxlist when the page scrolls, and before a click Playwright may scroll the page to the target even when it is in view, so a click on an option can land after the list has closed (BUG-110). A list opened before the read that fills it has landed has its options rebuilt under the test. Open the list once its read has landed (aria-busy="false"on theComboboxinput), and choose an option with the arrow keys and Enter, waiting for the input’saria-activedescendantto move after each key, never by clicking the option (.forge/lessons.mdL-155). - More work in one case than its time allows. A backend test that sends a message through more redeliveries than its assertion needs multiplies every slow emulator call by that number: set the fewest for that test and restore the shared setting afterwards (BUG-259’s dead-letter row, a retry limit of 8 lowered to 2 for its own message;
.forge/lessons.mdL-301). A component test that types long text it does not test, or ticks many boxes one at a time, runs close to its timeout on an idle machine and over it under load (BUG-128, BUG-213, BUG-214, BUG-220’s 51 contacts). Type withuser.pastewhere the typing is not what the test is about, use a select-all control or fewer items where the count is not what it proves, and move a long walk through many items into a hook or unit test (designtest-run-cost.mdF-5). Work that cannot be cut, a git repository fixture built by synchronousgitprocesses, takes the git-fixture timeout on itsdescribe,{ timeout: 60_000 }, as the browser-gate suites do: nothing races there, so it is not the lengthened timeout this section bans (.forge/lessons.mdL-257; BUG-349, BUG-350). The same holds for one wait: a test that releases a held answer and then gives one 1 sfindByorwaitForseveral renders of a large screen releases it insideawait act(async () => …)first (.forge/lessons.mdL-270; BUG-382, BUG-365). - A list that grows with every run. A test that reads every row or every page of a list that other tests, or earlier runs against the same backend, also add to gets slower each run until it times out (BUG-173, BUG-233). Read only the rows the test made (§ Test Data & Isolation), and give the gate’s backend a fresh database where the run allows it (design
test-run-cost.mdP-7).
Test Tooling Summary
One table for “what do we actually use, and does it exist yet”. coding-standards.md §7 remains the authority on the tools themselves — this is the testing-relevant slice of it.
| Tool | Purpose | Repo | Wired at |
|---|---|---|---|
Spock 2.5-groovy-5.0 (spock-core + spock-spring) |
Backend T1 + T2 framework; built-in Mock()/Stub() replaces Mockito. The Groovy suffix tracks the Boot line and is a compile gate (AD #25) |
forge-et-be |
FS-1 ✅ |
bare spring-test — deliberately not spring-boot-starter-test |
Spring test utilities without dragging in Mockito/AssertJ/Hamcrest, which would invite two idioms into one codebase | forge-et-be |
FS-1 ✅ |
| Testcontainers (PostgreSQL) | A real Postgres for T2 — not H2 (see the T2 real-dependency row) | forge-et-be |
FS-2 |
| ArchUnit | web → service → repository layering, feature-package acyclicity, @NullMarked presence, no schema qualifier, fixture unreachability — the hygiene-guard suite (FS-14 retired the split’s boundary + banned-vocabulary rules) |
forge-et-be |
FS-1 ✅ |
| Flyway | Tests run the same migrations as production (one public schema, one sequential version line since FS-14) |
forge-et-be |
FS-2 |
| JaCoCo | Coverage — report from FS-4, gate from Phase 2 (§7.4) | forge-et-be |
FS-4 |
| Trivy (dependency-vulnerability scan) | HIGH/CRITICAL ≡ CVSS ≥ 7.0, report-only on every ready PR, release/* push, manual and weekly CI run, and the nightly full run on the lead’s Mac (the dev deploy runs none; ECR scans each dev image on push — AD #38 as restated 2026-11-03); a UAT or production release stops on an open finding without a .trivyignore.yaml exception (AD #9 / AD #27 / AD #38) |
all three app repos | FS-4 |
| Vitest | Frontend T1 + T2 | forge-et-fe, forge-et-pwa |
FS-1 ✅ |
React Testing Library + @testing-library/user-event |
Component rendering and real interaction, inside Vitest | forge-et-fe, forge-et-pwa |
FS-1 ✅ |
Vitest --coverage (v8) |
Frontend coverage, scoped to src/features + src/**/hooks |
forge-et-fe, forge-et-pwa |
FS-4 |
| Playwright (TypeScript) | T3 browser E2E and the browserless seam tier via APIRequestContext — one tool, two modes |
forge-et-fe (+ forge-et-pwa) |
FS-4 |
@axe-core/playwright |
Accessibility scan on every page visited during E2E; also the assigned closure for the F-004 lint gap | forge-et-fe (+ forge-et-pwa) |
FS-4 |
check-testid-coverage.mjs |
Makes coding-standards.md §4’s data-testid rule mechanical (ships with a --self-test mode) |
forge-et-fe, forge-et-pwa |
FS-3 |
dependency-cruiser |
Frontend boundary rules + the transitive fixture-import rule (§0.1 shape ②) | forge-et-fe, forge-et-pwa |
FS-3 |
| Biome | Format + lint; the whole a11y group as errors — with the coverage limit noted under § Accessibility |
forge-et-fe, forge-et-pwa |
FS-1 ✅ |
gitleaks |
Secret scan in the fast lane, all four delivery repos | all | live ✅ |
✅ = wired and proven red-green as of FS-1 (2026-08-21).
System Verification to V1 Sign-off
The sections above prove each feature as it is built: every acceptance criterion maps to a test, the always-test list holds, the browser gate runs every page at every size, and each PR gets a security review. What they do not cover on their own is the assembled product, tested as Empire will use it, in a deployed environment, and the non-functional requirements measured where they are binding. This section is the plan for both, for each UAT round: Phase 1’s (30 Nov – 10 Dec) and Phase 2’s final round (11 – 29 Jan), client/project-timeline.md. The lead asked for thorough functional and non-functional testing on 2026-10-17. Three kinds of item appear below: commitments already made elsewhere, named with their homes; steps this plan adopts for each round (the per-role regression, the acceptance-criteria walk, the exploratory pass, the real-browser passes and the log sample), which the lead may change; and items marked proposed, which await the lead’s decision (.forge/owed.md).
Functional verification of the assembled product, before each UAT round
| # | Step | What it proves | Home | When |
|---|---|---|---|---|
| 1 | Every feature in the round is done with each acceptance criterion mapped to a passing test (TC-3, /forge-test-verify) |
Each feature works alone | the feature’s spec and plan | as each feature closes |
| 2 | Everything on main retested on dev: the deployed backend’s checks (F38 AC-40) and a per-feature checklist walked by hand on dev with a real Microsoft sign-in (F38 Rev 21; the browser suites depend on the local seed and the stub login, so they are not pointed at dev). Development does not stop while a walk runs, so each walk is followed by a gap retest: the PRs merged since the walk’s list was read are taken from GitHub, each change is marked covered by a walk result or in the gap, and the gap is walked with about eight to ten end-to-end flows rather than per-criterion checklists, never from the start. Each later pass repeats this from the previous pass’s cutoff, starting from the same flows (Rev 33; F38-g spec FR-7) |
The features work on the real platform: migrations, CORS, cookies, Microsoft sign-in, CloudFront routing; and what merged while they were being walked | F38-f, F38-g (its wave 5 is the first gap retest) | before the round’s UAT environment is applied |
| 3 | A regression cycle on the round’s UAT environment: the cross-feature flows /forge-release derives from the acceptance criteria, with evidence per flow; this plan also runs them for each role in the recommended role set |
The assembled product works across features and roles, in the environment Empire will use | .claude/rules/tracker.md § Release & UAT Lifecycle; .forge/release/<id>/RC-n-* |
before handover; the handover gate needs one cycle passed or passed-with-risks |
| 4 | The phase’s acceptance criteria walked one by one on the UAT environment, as they will be demonstrated (Phase 1: every criterion in client/phase-1-acceptance-criteria.md, still a draft), on desktop and on a tablet in Chrome; then once more on real Safari on an iPad and in Edge, since the criteria’s “How the criteria are tested” section names Chrome, Edge and Safari |
Every criterion Empire will check passes before Empire checks it | the cycle’s evidence | in the regression cycle |
| 5 | An exploratory pass by role: each developer takes a role and uses the product without a script, looking for what no criterion names (a manual pass on 2026-10-08 found BUG-153 to BUG-170) | Defects between and around the criteria | findings go to .forge/bugs.md; the pass is recorded in the cycle report |
in the regression cycle |
| 6 | The live external paths walked: Microsoft sign-in and sign-out, mailbox connect, send and receive, SES system mail arriving with SPF, DKIM and DMARC pass (F38 AC-26; F38-e AC-7 sends it under the backend’s task role), and the AI features on Bedrock; in Phase 1’s round, the capture app’s camera and QR capture on a real iPhone and a real Android phone, which a browser test cannot drive |
What a stub or an emulated browser cannot prove (§ Non-Deterministic & External Surfaces) | the cycle’s evidence; register rows | in the regression cycle, once each service is set up in the environment |
| 7 | The migration rehearsed on a copy of Empire’s real export: dry run (F4-a), load, re-run, reconciliation report (§ The Pipedrive migration) | The one-shot migration works on real data | F4-a, F4-b, F4-d | before the parallel run’s first load |
Each phase’s round is its own release in the tracker’s releases: list: P1 for Phase 1’s round, run while Phase 2 is built, and P2 for the final round (.claude/rules/tracker.md § Release & UAT Lifecycle). The rows above run once per release, and a defect Empire raises in Phase 1’s round becomes a bug scheduled into Phase 2 (delivery_phase: 2), so Phase 2 cannot seal until it is fixed.
Non-functional verification
Each row names the requirement, where it is stated, how it is checked, where the binding measurement is taken, and who owns it. “State” is as read on 2026-10-17.
| Requirement | Source | How it is verified | Where and when | Owner | State |
|---|---|---|---|---|---|
| Screens load in < 2 s; interactions respond in < 200 ms | PRD § Performance | In the browser: the time from navigation to the screen’s settled content, and from an interaction to its response, repeated 20 times with the 19th read (as § Performance Baselines does for endpoints), with data at the migrated volume; the endpoints behind a slow screen are then timed by that section’s method to find the cause. Each reading records where the browser was: every API call goes to us-east-1, and the offices are in Sri Lanka, Kenya, the Netherlands, Shanghai and Spain (proposal §5), so each round takes at least one reading from Colombo and one from a distant office or a remote browser in that region |
dev, binding; laptop readings against a local backend are evidence only (L-084). Before each UAT round | each feature’s owner; the regression cycle collects them | not yet measured on dev |
| Synchronous AI calls in < ~10 s; batch work in the background | PRD § Performance | T2 timing against the stub (exists), then measured against Bedrock on dev; the 200-lead batch triage run once at that size on dev, checking the screen stays usable and the job finishes | dev, once the models’ quotas allow | AI feature owners (F12 onwards; F11-c, F11-d for triage) | Haiku 4.5 works on dev; Sonnet 5.5 and Haiku 5.5 are refused (quota 0, register §C-5) |
| Email send and sync throughput (~40 actions a minute per account, PRD; large mailboxes, § Microsoft Graph) | PRD § Performance | The live mailbox tests in § Microsoft Graph (a few thousand messages, 20 parallel inbox requests) | dev, once the mailbox infrastructure exists | the F20-i, F20-f, F20-j and F20-k owners | dev has the mailbox ingest queue and the KMS keys (forge-et-iaac#3, 2026-10-20); the live mailbox tests are not run yet |
| Imports up to ~50k rows; the migration volume | PRD § Performance | T2 with a large file (exists), and the rehearsal on the real export (step 7 above) | local, then dev | F4-a, F4-b, F4-d, F23-b owners | to do |
| 10–20 concurrent users | PRD § Performance | Decided (the lead, 2026-11-19, Rev 44): one concurrency check per UAT round, driven by Playwright workers — 20 signed-in sessions running the main list and detail reads and a few writes for about ten minutes against the UAT environment, recording p95 response time and errors, and the backend task’s CPU and memory and the database’s CPU and connections in CloudWatch. It doubles as the check that dev’s sizing (0.5 vCPU, 1 GiB task; db.t3.micro) holds. Playwright workers can drive it with no new tool; a load tool such as k6 would be a new dependency (ASK FIRST) |
the UAT environment, before each handover | Tharinda to decide and assign | not decided |
| Stress beyond that scale | PRD § Performance | Decided (the lead, 2026-11-19, Rev 44): no separate stress test. The PRD’s scale is 10–20 users of an internal tool, so a traffic spike far above it is not a case to design for. The heavy work is checked at volume in its own rows: the mailbox scan and the AI batch triage run as background jobs, and the imports’ write runs are still to be designed by F4-b and F4-d (F4-a’s dry run answers in the request). The concurrency check above covers the users’ side | — | Tharinda to decide | not decided |
| A failed deploy or task does not leave the system broken (business-hours availability, single Availability Zone, recovery by restore — proposal §A.2) | proposal §A.2; architecture § Deployment Topology; F38 | The deployment circuit breaker rolls a failed deploy back and ECS replaces a failed task (a short gap with one task); each CloudWatch alarm tripped once and its email received (F38 FR-25, AC-25). Since F38 Rev 23 (2026-10-23): in dev, the failed-deploy email and one test alarm; each other alarm is tripped in the environment that builds it | dev; UAT and production for their own alarms | F38-d, F38-e; F38-h, F38-i | dev has the alert topic and the failed-deploy rule (forge-et-iaac#7, 2026-10-23) and, by decision, no other alarms; UAT’s and production’s alarms not built yet |
| Backups restore within the stated times (5 minutes’ data loss, 1–2 hours to restore) | proposal §A.5 | A documented restore rehearsal before go-live and a restore test each quarter after (F38-i; timeline, Phase 2 UAT). Decided (the lead, 2026-11-19, Rev 44): rehearse once on dev before Phase 1’s UAT, timed — a restore of its own, because dev’s database stays unencrypted (the lead, 2026-10-17, F38 Rev 17), so no encrypted-copy restore runs there. It proves the time to restore, not the 5-minute data loss, which needs point-in-time recovery, and dev’s backups are off | dev (proposed), then production | Shyamin Ayesh with Tharinda | dev backups are off (allowed, F38 Rev 11); production rehearsal is F38-i’s |
| The infrastructure controls hold (encryption at rest with KMS, TLS 1.2 or later, the database unreachable from the internet and open only to the backend, S3 public access blocked, least-privilege security groups, an audit trail of infrastructure actions, separate dev and production accounts) | proposal §A.4; F38 | Checked against the Terraform and the account by F38-a and F38-b in dev and F38-i in production (their acceptance criteria), and re-read in the regression cycle | dev, then production | F38-a, F38-b, F38-i | in dev the database is not encrypted, by decision (F38 Rev 17; production undecided), and port 5432 is open to the whole VPC with the database in the public subnets, both optional in dev by decision (F38 Rev 18; environments-and-identity.md §4 rows 2–3); dev has no CloudTrail trail by decision (F38 Rev 13) |
| Security — per change | PRD § Security and Compliance; AD #9, #26, #34–#38 | The security review on every security-sensitive PR (/forge-review-pr step 6a), the always-test list’s access rows, § Security’s table, gitleaks, the bundle check, Trivy on every PR |
every PR | each PR’s author and reviewer | in force |
| Security — per release | AD #38; register §C-8 | The Trivy release check stops a UAT or production deploy on an open HIGH/CRITICAL finding (F38-h FR-35, F38-i); the pre-go-live security review (timeline, Phase 2 UAT) | each UAT and production release | F38-h, F38-i | not built yet |
| Security — the deployed system | PRD § Security and Compliance | Decided (the lead, 2026-11-19, Rev 44): a short threat model before Phase 1’s UAT (none exists, § Security), and a passive OWASP ZAP baseline scan of the UAT environment’s frontends and API in each regression cycle, which checks security headers and cookie flags (TLS versions are set by the load balancer’s and CloudFront’s policies, checked in environments-and-identity.md §4). ZAP is a new tool, approved by the lead with this decision. Both dev distributions send HSTS, X-Content-Type-Options, X-Frame-Options and a referrer policy through a response-headers policy (forge-et-iaac#3, applied 2026-10-20, F38-b spec Rev 7; the capture app’s since its hosting was applied on 2026-10-22), and since forge-et-iaac#14 was applied on 2026-10-28 each also sends its own Content-Security-Policy (curl -sI, 2026-10-29). The CRM’s must keep HTML email bodies working (the register’s §B F20-c row (e)), which is still to be checked signed in. UAT’s overlay sets no CSP yet, so a scan of the UAT environment would flag one until F38-h sets it. An external penetration test is not in the issued proposal, so it is a change request only if Empire asks |
the UAT environment | Tharinda to decide | not decided; the CSP’s signed-in check on dev and UAT’s value owed (.forge/owed.md, “A Content-Security-Policy on both frontends”) |
| Accessibility (WCAG 2.1 AA; keyboard use) | PRD § Accessibility | The browser gate on every PR, and a keyboard-only walkthrough of each core screen when it ships (§ Accessibility). Decided (the lead, 2026-11-19, Rev 44): one screen-reader pass (VoiceOver on Safari; NVDA where a Windows machine is free) over the round’s core screens in the regression cycle | every PR; the regression cycle | each feature’s owner; the cycle | in force; the screen-reader pass not decided |
| Supported browsers, sizes and devices | PRD § Accessibility; Phase 1 acceptance criteria | The gate runs Chromium and WebKit at every size; the regression cycle adds the real-browser passes of step 4 (Safari on an iPad, Edge) and, in Phase 1’s round, the capture app on a real iPhone and Android phone (step 6) | the nightly full run, a frontend PR’s selected run, and the regression cycle (PRD Rev 81, Rev 93) | the cycle | in force for the gate |
| Data protection and AI handling | PRD § Security and Compliance; proposal §A.6 | T2: what each AI request carries, and that request and response bodies never reach the logs (§ The AI layer); a look at a sample of dev’s logs in the regression cycle | every PR; the regression cycle | AI feature owners | in force; Bedrock may process requests in any commercial AWS Region (OQ-15, settled 2026-11-04) |
| Every write and security event audited | AD #36 | The always-test row, and the backend suite failing a write with no audit event | every PR | each writing feature | in force |
| Monitoring and cost | PRD § Observability and Audit | Alarms tripped once (above); AWS Budgets with alerts; AI cost per call logged | dev (the failed-deploy email and the test alarm only, F38 Rev 23), then UAT and production | F38-e, F38-h, F38-i; budgets set up by Shyamin Ayesh, Kavindu Anthony, Malinda Wellalage and Thilina Abeyrathna, not checked by the delivery team (F38 Rev 18) | not built yet |
| Email deliverability (SPF and DKIM, PRD; DMARC, F38) | PRD § Security and Compliance; F38 | A sent message is seen to arrive, once per phase, for user mail through Graph (§ Microsoft Graph) and for system mail through SES, with SPF, DKIM and DMARC pass in its headers (F38-a FR-8 sets SES up in dev; F38 FR-26, AC-26, F38-e, checks arrival under the backend’s task role) |
dev, then production | F20 owners; F38-a, F38-e | SES set up in dev and a first test message delivered (2026-10-21, sent as svc-harness); the send under the backend’s task role is F38-e’s |
| The migration is complete and correct | proposal §8 | The reconciliation report Empire can check (§ The Pipedrive migration) | before each migration run | F4-b, F4-d | to do |
Recording it
Each regression cycle’s plan lists the rows above that fall due in that round, and its report records each one’s result with its evidence, under .forge/release/<id>/. A row that cannot be done in time gets an owed.md row with an owner, or a register row when it waits on the cloud. The phase-boundary checklist checks this matrix is current before a phase seals.
Deliberately Not Tested
Recorded so it is not re-proposed each time someone reads a testing doc from another project, and so the reasoning is inspectable if a trade-off changes. coding-standards.md §7.9 is the equivalent list for tools.
| Not doing | Why |
|---|---|
| A nightly full E2E run in CI | Closed in AD-O-11 — it duplicates the release-tag run without adding a distinct failure class, and GitHub Actions minutes come from an org-wide shared pool. The browser suite does run nightly off CI, on the lead’s Mac, where neither reason applies (.forge/design/harness/test-run-cost.md N-3). |
| Mutation testing (PIT / Stryker) | Genuinely the best measure of test quality, and genuinely too slow for the 10-minute budget. Worth revisiting if the codebase outlives V1. |
| A repo-wide coverage percentage gate | §7.4 — per package or nothing. An aggregate number is dominated by scaffolding and says nothing about whether the quote rules are tested. |
| Dark mode | Not in V1 (design/ui/design-system.md). |
| Cross-tenant / multi-org isolation | Single-tenant by design (AD #3). There is no second tenant to be isolated from, so such a test would assert against a fiction. |
| A standing load / stress suite | The PRD’s scale is 10–20 concurrent users. The seed volume is the load; performance is checked against the § Performance Baselines targets on demand, not in a standing suite. A one-off concurrency check per UAT round is proposed in § System Verification to V1 Sign-off. |
| Native mobile | Deferred in the PRD. The capture PWA is the phone surface and is tested as a PWA. |
| Re-running per-feature suites during release regression | Regression cycles reference the per-feature results rather than re-running them (.claude/rules/tracker.md § Release & UAT Lifecycle) — they add cross-feature assembled-product flows only, so build-time results are not double-counted. |
Mapping to the Forge Workflow
| Phase | Test activity |
|---|---|
| Spec | Acceptance criteria are written so each maps to ≥1 test (observable + verifiable). |
| Decompose | ## Test Strategy Map assigns a tier (T1/T2/T3/T-E2E) to every work item; AC coverage + ownership recorded. |
| Plan | ## Test Approach declares the WI’s tier + the test files; forge-plan-review TC-2 (tier-match) / TC-3 (AC-coverage) enforce it at plan time. |
| Implement | Implementers write + run T1/T2 alongside code; T3/T-E2E are authored by the test specialists. |
| Verify | The orchestrator runs the live tiers + the repair loop; /forge-test-verify guards green status. Pass is the precondition for /forge-pre-pr-review + /forge-pr-open. |
| Review / Ship | Tests exist for every AC; the wave PR’s CI is the merge gate; the final wave runs the full suite. |
| Reflect | Note test patterns worth promoting; update this strategy if the approach changed. |
Test Checklist for Implementation Plans
Every feature plan’s ## Test Approach should cover the applicable rows:
- [ ] T1 unit tests for all business-logic methods + interactive UI components
- [ ] T2 integration tests for all new API endpoints (happy + 401/403/400 paths)
- [ ] T2 integration tests for data scoping (if the feature touches scoped data)
- [ ] T3 seam tests against the frozen wave contract (BE↔FE waves)
- [ ] T3 WI-scope browser E2E for every acceptance criterion with user-visible behavior + an accessibility scan per new page
- [ ] T-E2E coverage folded into the full suite (final wave)
- [ ] Security tests if the feature touches auth, roles, or user input
Plus the § Always-Test List rows this feature touches — each one below is a row in a test table, not a note:
- [ ] AI: if the feature has an AI step — a test that the autonomous path does not exist (no price set, nothing approved, nothing sent without recorded human approval)
- [ ] Permissions: if the feature gates on anything — the AD #26 negative test (holding a role does not confer an ungranted permission)
- [ ] Visibility: if the feature lists or opens a shareable record, or a child of one — it goes through the one
auth.accessrule, and the list and the per-record check agree on the case table, shares and unowned records included (AD #35) - [ ] Audit: if the feature writes — the
audit_eventrow is written in the same transaction and rolls back with the change (AD #36) - [ ] Queue: if the feature enqueues — deliver twice, assert one side effect (AD #13)
- [ ] Temporal: if the feature touches prices or any effective-dated row — no in-place
UPDATE, and an as-of read returns the value in force then (AD #14) - [ ] Truthfulness: if the feature renders server data — the four §0.1 shapes (runtime freshness · no fabricated fallback · empty ≠ error · the server owns the number)
- [ ] State: every valid transition and every invalid one rejected
- [ ] Every new interactive element is addressable (role + name, or
data-testid—check-testid-coverage.mjschecks this), and any testid follows<feature>-<element>[-<qualifier>](§4 — a review check; the script does not check naming) - [ ] AI: if the feature calls the
AIProviderseam — the deterministic rows in § Non-Deterministic & External Surfaces (schema rejection, HITL queue, provenance, kill switch, outage → manual). Model quality is the eval set, never a plan row - [ ] Mailbox: if the feature touches Graph — stubbed at the
MailboxProviderseam, plus a note on what needs live-account evidence this phase - [ ] Accessibility: axe on every new page and a keyboard-only walkthrough of each new core screen, recorded
- [ ] Time: if the feature reads “now” — the clock is injected, not the system one
- [ ] Tier matches the Decomposition Plan’s
## Test Strategy Map(/forge-plan-reviewTC-2), and every AC on hook appears in ≥ 1 row (TC-3)
Writing Tests with Claude Code
- Read the spec’s acceptance criteria — each maps to ≥1 test.
- Write tests alongside implementation, not after.
- Run tests after every change (a Boundaries hard rule).
- Name tests after the behavior, not the method.
- Reuse existing tests for previously-built functions; author new tests only for new functionality.
- Walk the § Always-Test List before writing the plan’s test tables, not after the code. Those rows are anchored to locked decisions; discovering one late means reopening an approved plan.
- Push each assertion to the cheapest tier that can prove it. If a rule can be proven at T1, a T3 test for it is slower, flakier and no more true.
- Never weaken a test to make it pass. A failing test is either a code bug or a test bug — decide which, then fix that. A skipped or
.onlytest fails the build by design (§7.8), and a quarantined test nobody deletes is a permanent silent hole.
Revisions
Rev 44 — 2026-11-19 — the proposed non-functional checks decided
- Changed: § System Verification › Non-functional verification: the five rows marked Proposed now read Decided — a 20-session concurrency check per UAT round driven by Playwright workers, no separate stress test, a timed restore rehearsal on dev before Phase 1’s UAT, a short threat model before Phase 1’s UAT and a passive OWASP ZAP baseline scan in each regression cycle, and a screen-reader pass in the regression cycle.
- Why: the lead adopted all six at the
owed.mdtriage on 2026-11-19, approving ZAP as a new tool; theowed.mdrow “Decide the proposed non-functional checks” is removed. Who carries each out is the row “Plan and run Phase 1’s system verification before its UAT round”.
Rev 43 — 2026-11-18 — keys that differ by system or engine
- Changed: § Flaky Tests, cause 4: a WebKit test that moves focus by key takes the key from
webkitFocusKey, every focus-trapping shadcn content loops Option+Tab (loopOptionTab), and any other browser test that relies on a key or focus behaviour that differs by system or engine is proved on Windows and macOS, or states which system proves it and marks the other as a known failure. - Why: lesson L-168 (F12’s Reflect, a follow-up in
owed.md): F5-h’s WebKit keyboard walk passed on the Mac it was written on and failed on every Windows run, and F1-e’s WebKit trap tests failed on Macs with keyboard navigation off (BUG-131, BUG-231, BUG-241).
Rev 42 — 2026-11-18 — a timing measurement runs under heavy-run.py
- Changed: § Performance, “Take no other load on the machine during a reading”: a timing measurement now always runs under the machine’s heavy-run lock through
python3 .claude/lib/heavy-run.py -- <command>, not only when agents run in parallel, and not under a lock of the timing session’s own. - Why: lesson L-315 — parallel sessions on one machine improvised per-ticket lock folders that only some of them checked, and a backend build took about 5 hours instead of about 85 minutes;
.claude/rules/test-record.md§ One heavy run at a time now puts every heavy run under the one lock (Dinil’s decision, 2026-11-18).
Rev 41 — 2026-11-16 — the Settings keys named as AD #41 split them
- Corrected: the AD #26 row, the fixture-roles paragraph and the privilege-escalation row still named
settings.manage, which AD #41 replaced on 2026-10-30 with one key per Settings page. They now name the page keys (settings.users.manage,settings.roles.manageand the rest). No test changes: the tests were written against AD #41. - Corrected (the same review, 2026-11-17): the last-admin row names both Settings keys (AD #41); the real-phone pass for the capture app is in Phase 1’s round (AD #20); F36-c no longer sends system mail (AD #42), so it leaves the SES rows; the mailbox-load and alarm rows say what dev has now.
- Why: found by the documentation review of 2026-11-16.
Rev 40 — 2026-11-13 — the capture app’s browser gate needs the backend
- Changed: § Browsers & Viewports:
pnpm e2e:gateinforge-et-pwaneeds the local backend running, as the CRM’s does, for itsseamproject — the specs that run against the real backend, in Chromium, once; itschromiumand size projects still mock the API. The header’sLast updatedline, which named Rev 34 after Revs 35 and 36, now names this revision. - Why: F36-a’s Decomposition Plan, § Decomposition Strategy, “The capture app’s browser tests and the backend” (approved under Nanndish’s overnight delegation of 2026-11-12): from F36-a the capture app’s verify work items and its end-to-end suite run against the real backend, signed in through the stub login.
Rev 39 — 2026-11-13 — a test that waits on an emulator
- Changed: § Flaky Tests step 2: a backend test that waits on an emulator in a container is reproduced by freezing that container in a loop, not by load on the host. Likely cause 6: a backend test takes no more redeliveries than its assertion needs.
- Why: BUG-259 reproduced only when LocalStack itself was slowed, and its cause was a row that needed eight receives to prove what two prove (
.forge/lessons.mdL-301).
Rev 38 — 2026-11-13 — the forced-load reproduction waits for the heavy-run lock
- Changed: § Flaky Tests step 2: the busy loops start only once the run holds the machine’s heavy-run lock and stop when it ends, and an app server a stopped run leaves on the gate’s port is stopped too.
- Why: BUG-415’s first reproduction started its loops while its run waited for the nightly full run’s lock; the loops kept the CPU above the load limit, so the nightly’s browser gates refused to start and neither run moved until the loops were stopped.
Rev 37 — 2026-11-13 — Dependabot PRs get a local record; the nightly backend build runs in two JVMs
- Changed: § When Tests Run: the Dependabot and on-demand row becomes on demand only; a Dependabot PR is recorded locally like any other PR. The wall-clock budget names each
Slow lane’s cap (the CRM’s is 45 since 2026-11-12) and says the suite runs there only on demand. “One per test JVM” says the nightly full run uses two test JVMs. - Why: the lead’s decision of 2026-11-13 (
.forge/design/harness/test-run-cost.mdN-5), and the nightly change of 2026-11-12 (B-7, forge-et-harness#1340).
Rev 36 — 2026-11-03 — a suite runs in one place
- Changed: § When Tests Run: the Dependabot and on-demand row (a push to
release/*runs theFast laneonly, its PR’s record having run every test); the release-tag row (onmain, CI’s on-demandFast lanerun and a nightly full run that passed every suite on the commit); the nightly row lists the checks it took over from CI’s scheduled run; the wall-clock paragraph no longer says the backend’sSlow laneruns on a push torelease/*; the two Trivy rows name the weekly CI run and the nightly full run (AD #38 as restated 2026-11-03). - Why: the lead’s decisions of 2026-11-03 (
.forge/design/harness/test-run-cost.mdN-5).
Rev 35 — 2026-11-02 — the browser gate leaves the PR; the records run the tests a change needs
- Changed: § When Tests Run: the once-per-PR row runs the tests the change needs (every test when shared code changed, or with
--full); the “before a PR is ready” row applies only to a PR intorelease/*; the wave-PR row has no full browser gate, only the selected run of the row before it; the nightly row is the only full run of every test and of both browser suites formain. § Browsers & Viewports: the browser suites no longer gate every pull request; they run in full every night onmain, and on a PR intorelease/*. - Why: the lead’s decision of 2026-11-02 (PRD Rev 81;
.forge/design/harness/test-run-cost.mdS-4, revision line of 2026-11-02): a browser gate cost about an hour on a Mac and two to three on Windows on each PR, and the nightly full run checks everything onmainevery night, so problems are found the next morning and fixed together.
Rev 34 — 2026-11-02 — the restore rehearsal no longer rides on an encrypted-copy restore
- Changed: § System Verification’s “Backups restore within the stated times” row: the proposed timed rehearsal on dev is a restore of its own, no longer “the encrypted-copy restore the database needs anyway”, because dev’s database stays unencrypted and no such restore runs. The proposal itself is unchanged and still the lead’s to decide. The header’s
Last updatedline, which named Rev 32 after Rev 33, now names this revision. - Why: the lead’s decision of 2026-10-17 (F38 Rev 17; F38-b spec Rev 1) keeps the dev database unencrypted;
owed.md’s row on the proposed non-functional checks already says the rehearsal is one of its own. Found by the open-items audit of 2026-11-02.
Rev 33 — 2026-11-02 — a gap retest after each walk on dev
- Changed: § System Verification’s step 2: each walk on dev is followed by a gap retest of what merged while it ran — the change list read from GitHub’s merged PRs since the walk’s list, each change marked covered or in the gap, and the gap walked with about eight to ten end-to-end flows, never from the start; each later pass starts from the previous pass’s cutoff and the same flows. F38-g’s wave 5 is the first.
- Why: the lead’s decisions of 2026-11-02 (F38-g spec Rev 3): F38-g’s retest has run since 2026-10-27 on a list read on 2026-10-22 while features, screen reworks and bug fixes kept merging, and the timeline allows neither stopping development nor retesting from the start.
Rev 32 — 2026-10-29 — the frontends’ Content-Security-Policy is served on dev
- Changed: § System Verification’s “Security — the deployed system” row no longer says the CRM sends no Content-Security-Policy: both dev distributions send one since forge-et-iaac#14 was applied on 2026-10-28. What is owed is the signed-in check of HTML email under it and UAT’s value. The header’s
Last updatedline, which still named Rev 29 after Revs 30 and 31, now names this revision. - Why:
curl -sIon both dev hosts showed each app’s policy on 2026-10-29.
Rev 31 — 2026-10-27 — no browser tier inside deploy-prod
- Changed: § When Tests Run’s “Release tag” row: production, like UAT, gets the smoke run by hand after the deploys, with the release track’s regression cycle as the full check; no browser tier runs inside
deploy-prod. Its command column names the smoke suite against the deployed host. - Why: the lead’s decision at F38-i’s spec, 2026-10-27 (F38-i decision 3; F38 Rev 31;
architecture.mdRev 81): the full suite depends on the local seed and the stub login, so it cannot run against production as built.
Rev 30 — 2026-10-27 — the record retries, the run queue, the weekly flaky report and the selected runs
- Changed: § Flaky Tests: one counted retry in both records as in the gate, the weekly flaky report in place of a bug per flaky test, the run queue and the wake lock, and two new first suspects (more work in one case than its time allows; a list that grows with every run). § When Tests Run: the development loop and an implementer’s subtasks run the selected tests (
pnpm test:selected,scripts/select-tests.py,./gradlew check -x test). § Browsers & Viewports: the CRM’s 1440 × 900 Chromium size runs without the axe scan in a full run. - Why: the rest of Phase 1 of
.forge/design/harness/test-run-cost.md(F-2, F-3, F-5, M-2, M-5, P-2, S-1, S-3, W-1), built 2026-10-27 in forge-et-fe#350, forge-et-pwa#99, forge-et-be#322 and forge-et-harness#995.
Rev 29 — 2026-10-26 — access.manage folded into settings.manage
- Changed: § The Always-Test List (the AD #26 row and the last-admin row), the fixture-roles paragraph and the privilege-escalation row name
settings.managewhere they namedaccess.manage; the privilege-escalation row adds that a role without it reaches no page under Settings. - Why: AD #34 refined by Nanndish on 2026-10-26, signed off by Tharinda, 2026-10-27:
settings.manageis the one key for configuration and for user, role and hierarchy administration, and the seeded IT Admin holds it (PRD Rev 79; FS-6 spec Rev 19).
Rev 28 — 2026-10-27 — the nightly full run moves to the lead’s Mac
- Changed: § When Tests Run: the nightly row is the full run of all three app repos — both browser gates included — on the lead’s Mac (
.claude/lib/nightly/run-nightly.py), which raises anightly-redissue; the CISlow lanerow keepsrelease/*, Dependabot PRs and on-demand runs. § Deliberately Not Tested: AD-O-11’s row is about CI. - Why: the lead’s answer to Q-4 of
.forge/design/harness/test-run-cost.md(N-3, approved 2026-10-27): every suite runs in full once a night off CI, so a PR can run less, and the scheduled CI run spends no Actions minutes on the suites.
Rev 27 — 2026-10-27 — the CRM’s PR profile and one counted retry in the browser gate
- Changed: § Browsers & Viewports: an ordinary CRM PR’s gate runs one size per layout band (320, 768 in WebKit, 1024, 1280), and on Windows Chromium only; a wave PR, a PR into
release/*and the nightly run check every size and both engines; theBrowser gate:line names the PR profile and a flaky count. § Flaky Tests: the browser gate retries a failed test once and counts a pass on retry as flaky; more than five fail the run. A Vitest test file is no longer a gate input. - Why: the lead’s answers to the test-run cost design (
.forge/design/harness/test-run-cost.md, Q-2, Q-3 and Q-11, approved 2026-10-27; PRD Rev 80): a full gate took about 50 minutes on a Mac and hours on Windows, and at full load 0.5–0.7% of the tests timed out somewhere, so a full run almost never passed in one attempt.
Rev 26 — 2026-10-26 — a sleeping Mac is checked before a stall is called a flake
- Changed: § Flaky Tests, triage step 1, now checks
pmset -g logfor a sleep in the run’s window and runs long local runs undercaffeinate -dims;.claude/rules/git-conventions.md § Pull Requestscarries the same line for the records and the gate. - Why: an idle sleep on battery stalled 20 browser tests in one gate (2026-10-22); the check had lived only in one developer’s local notes, so another developer’s run could be triaged as a flake.
Rev 25 — 2026-10-26 — the capture app’s security headers
- Changed: § System Verification’s “Security — the deployed system” row no longer says the capture app has no hosting; its dev distribution sends HSTS,
X-Content-Type-Options,X-Frame-Optionsand a referrer policy, as the CRM’s does. The Content-Security-Policy gap the row names is unchanged. - Why: forge-et-iaac#5 applied the capture app’s hosting on 2026-10-22, and
curl -sI https://capture.dev.crm.empire.lk/showed the four headers on 2026-10-26 (the open-items audit of that day).
Rev 24 — 2026-10-26 — corrections from the harness review
- Changed: § When Tests Run’s “Release tag” row follows AD-O-11 as revised by
architecture.mdRev 72: UAT gets a smoke run by hand after the deploys, with the release track’s regression cycle as the full check, and the full browser suite runs insidedeploy-prodonly, once F38-i builds it and if F38-i decides how it signs in (F38 OQ-D). The “Merge tomain” row names the smoke command,pnpm e2e:smoke, which F38-g added toforge-et-fe. § Critical Flows & the Smoke Set no longer says the project is before Gate 3; the decision not to keep a hand-written flow list stands, and the paragraph now says why it still holds. - Why: the harness review of 2026-10-26 found the rows behind
architecture.mdRev 72 andforge-et-femain, and the paragraph describing a state that ended at Gate 3 (2026-09-17).
Rev 23 — 2026-10-19 — the smoke tier is run by hand, and the retest is a checklist
- Changed: § When Tests Run’s “Merge to
main” row: the smoke tier is run by hand by the delivery-team member whose merge or manual dispatch started the deploy, after it finishes, with no CI job, and a failing run is fixed before new work. § Choosing the smoke set: its heading drops “CI”, and F38-g’s plan chooses the set. § Test Data & Isolation’s smoke row: run by hand, with a view-only account. § System Verification row 2: the retest on dev is a per-feature checklist walked by hand, not the browser suites pointed at dev. - Why: the lead’s decisions of 2026-10-19 while F38-g’s spec was drafted (F38 Rev 21,
architecture.mdRev 68): a CI smoke run would have cost about 200–800 billed Actions minutes a month, and the browser suites depend on the local seed and the stub login, which dev does not have.
Rev 22 — 2026-10-17 — system and non-functional verification to V1 sign-off
- Added: § System Verification to V1 Sign-off: the functional checks on the assembled product before each UAT round (retest on dev, a regression cycle by role, the phase’s acceptance criteria walked, an exploratory pass, the live external paths, the migration rehearsal), and a matrix of every non-functional requirement with how, where and by whom it is verified. Changed: § Deliberately Not Tested’s load row now excludes only a standing suite.
- Why: the lead asked on 2026-10-17 for every function and every non-functional area to be tested thoroughly. The tiers above prove each feature as it is built; nothing stated how the assembled product, the deployed environment and the non-functional requirements are checked before Empire’s UAT rounds, and Phase 1’s round falls outside the release lifecycle as written.
- Proposed, for the lead to decide (
.forge/owed.md): a concurrency check per UAT round, no separate stress test, a threat model and a passive ZAP scan, a timed restore on dev, a screen-reader pass, and how Phase 1’s round is modelled in the release track. Gaps found while writing it: the CRM sends no HSTS or Content-Security-Policy header, now anowed.mdrow; the dev database is unencrypted and open to the whole VPC (already inenvironments-and-identity.md§4).
Rev 21 — 2026-10-17 — how the email features are tested
- § Microsoft Graph: a bullet on how F20-i, F20-f, F20-j and F20-k are tested — the stub cases, the browser flows, the non-functional checks, and the live evidence and timings on dev.
- Why: the lead asked on 2026-10-17 for the email flows to be tested thoroughly, including performance and security; the approach was first recorded in
.forge/owed.mdand moved here as its permanent home.
Rev 20 — 2026-10-15 — the load check, and a triage order for flaky failures
- § Flaky Tests: a triage order for a flaky failure (read the run’s recorded load and stalls; reproduce under forced load, the Chromium throttle and a held step; fix a timing defect; close an overload-only failure
wontfixwith its evidence, the lead deciding), stated as the one exception to the deletion rule; cause 3 points at the load check. - Why: flaky failures on overloaded laptops were filed and investigated one by one with no record of the load (BUG-194, BUG-195; BUG-149 asked for the gate’s load), and an overload-only failure could otherwise be closed only by deleting a test that is not at fault. Harness design
.forge/design/harness/load-check-and-flaky-tests.md, H-1, approved by Tharinda (lead), 2026-10-15; the check itself is forge-et-fe#272, forge-et-pwa#79 and forge-et-be#261.
Rev 19 — 2026-10-15 — Spock 2.5
- § T1 — Unit and § Test Tooling Summary: Spock moves from
2.4-groovy-5.0to2.5-groovy-5.0(forge-et-beDependabot PR #257). The Groovy suffix is unchanged, so AD #25’s compile gate still holds. Spock 2.5 changes some test behaviour: an interaction declared outside athen:block now also applies incleanup(), and an interaction on anulltarget throwsInvalidSpecException(Spock 2.5 release notes).
Rev 18 — 2026-10-12 — the suite runs once per PR, and every app repo has a Slow lane
- § When Tests Run: the local-record row becomes “Once per PR, after its last push”: the record is the required
Test recordcommit status, which a later push leaves off the new head and a documentation-only change carries forward. The gate row and § Browsers & Viewports: the push that marks a PR ready, and the last later push that changes a gate input, go throughpnpm e2e:gate --push, then the test record. The wave-PR and nightly rows, and § The AI layer, name each app repo’sSlow lane. § Container-backed specs: one 30-minute cap for everySlow lane. - Why: recording before every push to a ready PR ran the suite, and the browser gate, again for each review fix; the frontends’ tests now run in a
Slow lanejob as the backend’s do, and the backend’s nightly suite hit its 20-minute cap on 2026-10-10 (.claude/rules/git-conventions.md § The test suites run locally, and the PR records them). Lead sign-off: Tharinda, 2026-10-12.
Rev 17 — 2026-10-12 — the frontends’ unit and component tests leave PR CI
- § When Tests Run: a new row, “Before every push to a ready PR”, for the local record in all three app repos (the backend’s build record, the frontends’ new test record,
pnpm test:record); a new row for the nightly run onmain; the wave-PR row no longer lists T1 + T2 as run in PR CI. § Browsers & Viewports › Draft, then ready and the gate row put the test record beforepnpm e2e:gate --push. § Container-backed specs: the frontends’Fast laneruns the tests only on the nightly run, a push torelease/*, a Dependabot PR and on demand, and its cap is a recorded 30-minute exception (coding-standards.md §6.8), beside the backendSlow lane’s 20; “the backend’s one raise” becomes “each raise”. - Why: the frontends’ test step took 445 s and 782 s on recent ready-PR runs, and GitHub Actions minutes are one budget for the organisation; the frontends now do what the backend has done since 2026-09-28 (
.claude/rules/git-conventions.md § The test suites run locally, and the PR records them). Lead sign-off: Tharinda, 2026-10-12.
Rev 16 — 2026-10-02 — the test budget names the job that runs the tests
- § Container-backed specs (the wall-clock budget) and § The AI layer: both said “the
Slow lane” as if every repo had one. The frontends’ tests run in their oneFast lanejob (Rev 13), so the 10-minute budget now applies to each CI job that runs tests, and the 20-minute exception stays on the backend’sSlow lane. No budget changed. - Why: forge-et-fe#76 and forge-et-pwa#30 removed the frontends’
Slow laneon 2026-09-28; Rev 13 left these two mentions for a later pass.
Rev 15 — 2026-10-02 — a share to a deactivated user is inert
- § The Always-Test List: the list/record agreement row adds the case of a record shared with a user who has since been deactivated: the share reaches nobody, the recipient’s managers included, and counts again when the user is reactivated.
- Why: lead decision 2026-10-02 at F5-f spec authoring (PRD Rev 49; F5-f spec FR-R3, AC-4).
Rev 14 — 2026-09-30 — a sharee holding manage may reassign
- § The Always-Test List: the sharing row adds that a user who reached a record through a share and holds
managecan reassign its owner, and that aview-only user can do none of the three actions. - Why: the lead confirmed AD #35’s rule on 2026-09-30 (PRD Rev 46); the F5-f stub’s worked example gains the case.
Rev 13 — 2026-09-29 — how the browser gate works, and the frontends’ one CI lane
- § Browsers & Viewports: the matrix intro drops the instruction to run the sizes by hand and names the
size-<w>x<h>-<engine>projects. The gate paragraph’s description check is now the last step of theFast lane’s full run, not of theSlow lane. A new “How the gate works” list states, once: the gate before ready and the result file.browser-gate/result.json; what a result covers, withscripts/browser-gate/gate-inputs.jsonandscripts/browser-gate/expected-skips.jsonalways gate inputs;pnpm e2e:gate --pushfor every push to an open PR, and why it costs one CI run; draft, then ready; the skip line; the shared checke2e/support/pageCheck.ts, the page-state liste2e/pageStates.tsande2e/support/axeExemptions.ts; andmarkKnownFailure('BUG-NNN', '<reason>'). The capture app’s list of work owed to F40 becomes the current layout. - § When Tests Run: a new row, “Before a PR is ready”, for
pnpm e2e:gate. The Wave PR row no longer names a frontendSlow lane: the two frontends run oneFast lanejob, and the row gains the Stage 12 gate, theBrowser gate:line and later pushes throughpnpm e2e:gate --push. - Why: F40 (spec FR-13) requires these documents to state the gate, the covers rule, the push rule and the skip line; forge-et-fe#76 and forge-et-pwa#30 removed the frontends’
Slow lanejob on 2026-09-28 (F40 spec Rev 2). Not changed: otherSlow lanementions in this file (the wall-clock budget, the Always-Test list’s ArchUnit row, the dependency-scan row, § Non-Deterministic & External Surfaces), which are not about the browser gate. The wall-clock budget’s “10 minutes for the wholeSlow lane” still describes a frontendSlow lane; it is left for a follow-up (F40-WI-1.4 plan## Notes).
Rev 12 — 2026-09-28 — the browser gate runs locally, not in CI
- § Browsers & Viewports: the browser suites still gate every PR, but the run is the developer’s
pnpm e2e:gate, recorded against the commit it ran on and enforced by a harness hook and a last step of theSlow lanethat reads the result line from the PR description. No CI job runs a browser. The capture app’s matrix gains a phone-landscape size (844 × 390) as the orientation check. - Why: F40’s gap check found Rev 10’s CI gate contradicted AD-O-11 (
architecture.md § Browser E2E runs in two CI tiers), which keeps browser runs out of PR CI to save the shared Actions minutes; a CRM CI job would also have to build and start the backend on each run. Developer decision (Dinil, 2026-09-28, with Tharinda’s agreement; PRD Rev 45). AD-O-11’s post-deploy tiers are unchanged.
Rev 11 — 2026-09-27 — AD #37: grants have no scope
- § The Always-Test List: the data-scoping row tests each owned-entity key for a user with nobody below them, a manager with a multi-level subtree, and a holder of the entity’s
allkey, each with and without a share, and tests that<entity>.allis refused without the entity’sviewormanage; the list-and-record row cites AD #37 and names unowned records as visible only with theallkey; the privilege-escalation row drops “every scope”. - Why: AD #37 (lead decision, 2026-09-27; PRD Rev 43) removes the scope from a grant. The lead’s worked example keeps its expected results.
Rev 10 — 2026-09-27 — the browser matrix: sizes, WebKit, and a CI gate
- § Browsers & Viewports now carries the matrix, per lead decisions of 2026-09-27. The CRM runs at tablet and desktop sizes plus 320px reflow (WCAG 2.1 SC 1.4.10). The capture app runs at phone, tablet and desktop sizes. WebKit joins Chromium (the capture app everywhere; the CRM at tablet sizes). Every page at every size gets axe, a no-horizontal-scroll check, a no-clipped-control check, the target-size floor and a no-focusable-inside-
aria-hiddencheck. The browser suites gate every PR in both app repos. - Why: an audit of the built screens (2026-09-27) found the scans ran at desktop size only. That let a tablet-only defect through: BUG-028, focusable links inside the
aria-hiddensidebar. It also found that no CI lane ran the browser tier at all. - Built by F40. Until it lands, verify stages run these sizes by hand.
Rev 9 — 2026-09-27 — corrections from the second harness review
Correction-only except one lead decision; no tier or tool is reversed.
- § The Slow lane budget: records the backend Slow lane’s lead-approved 20-minute cap; the frontends stay at 10 (
coding-standards.md §6.8). - § Test Naming & Layout: the removed
catalogpackage leaves the tree drawing; the*Spec= T1 /*IT= T2 rule now binds new specs, and the existing context- or container-backed*Specfiles are accepted legacy, not renamed (lead decision, 2026-09-27). - § Coverage:
./gradlew checkhas produced the JaCoCo report since FS-4 (it said “no coverage step today”); Vitest scopes includesrc/ui. - § Test hooks and the checklist:
check-testid-coverage.mjschecks that each interactive element is addressable, not that a testid is named correctly — naming is a review check. - Trivy rows: the gate flips when all three app repos read 0 HIGH/CRITICAL (AD #27, FS-4 spec Rev 3), not at a phase boundary.
- § Browsers & Viewports: records what the capture app owes with its first browser tests (F36-a) — a phone Playwright project, the move of
e2e/a11y.spec.tsundere2e/browser/shell/, and a per-area axe spec with the 44 × 44px target asserted — so accessibility is held to the same bar in both frontends. The accessibility check is also added at each stage that lacked it: the spec template andspec-reviewer(an accessibility NFR), the plan template andplan-reviewer(TC-4a),/forge-pre-pr-reviewand/forge-test-verify(lead decision, 2026-09-27).
Rev 8 — 2026-09-24 — the access model, sharing and the audit trail (AD #34, AD #35, AD #36)
- Changed — § The Always-Test List: the AD #26 row uses the catalog key
quote.approve_sub_floorand makes its negative case “the seeded IT Admin (holdingaccess.manage) cannot approve until it grants itself the key, and the grant is audited”; the data-scoping row is restated as keys ×own/team/all× shares (was the(role, resource, level)model). Seven rows added: the list and per-record rules agree (roll-up, unowned only atall, children follow their parent); sharing eligibility, transitivity and authority, with the server refusing an ineligible share; the last-admin invariant at all four points; nobody deactivates themselves; deactivation ends sessions and waits for reassignment and re-parenting; privilege changes apply on the next request (backend, and the frontend/mere-read on 403 and focus); the audit event in the same transaction. - Changed — § The roles the fixtures must cover, T-E2E scope, fixture convention 4 and the full
T-E2E set: no role is predefined, so the fixture roles are sample data in
R__local_seed.sql(the local profile only; tests seed their own), plus the seeded IT Admin and a sample all-permissions test role in place of the retired superadmin; tests assert on keys, never on a role’s name. The Managing Director fixture holdsquote.approve_sub_floor. - Changed — § Security: privilege escalation is restated by key, with self-escalation allowed by design and checked through the audit trail; the session row names session deletion on deactivation; a sign-in-matching row is added (no auto-provisioning; the email-change re-link).
- Changed — § Test Checklist for Implementation Plans: Visibility and Audit items added.
- Why: the lead’s access-model decisions of 2026-09-24. Rationale:
.forge/engagement-log.md§ AD #34, AD #35, AD #36. - Not changed: the tier model, the toolchain, the Graph CI posture and the flaky-test policy. The six role names in Rev 3 stay verbatim — a revision log is a point-in-time record.
- Impact: F5-k rewrites the fixture roles and the E2E sign-in; F5-c, F5-d, F5-f, F5-g and F39 plans carry the new rows.
- Approved by: (owed — the approval of the harness PR carrying the AD #34–#36 changes)
Rev 7 — 2026-09-22 — the Graph test-account count stops being restated here
- Changed: two live-text mentions of “three test accounts” in §Microsoft Graph now point at
design/environments-and-identity.md§5 as the authoritative list instead of carrying a number. The CI posture itself is untouched — stub the seam for T1/T2/T3, exercise live Graph manually per phase against Empire’s accounts, never as a blocking gate. - Why: five accounts are provisioned (
CRM01–05@empire.lk), not three, so the number was wrong; and a count copied out of another file goes stale silently, which is the NEVER-DO in.claude/CLAUDE.md. Naming the source fixes both at once. Surfaced as signal W-1 by F20-a’s gap-check, which resolved it for the spec and left the older documents to be “corrected when next touched” — this is that correction. - Not changed: the counts inside Rev 4 and Rev 5 stay verbatim. A revision log is a point-in-time record and rewriting it would falsify what was decided on 2026-08-24/25 against the information held then.
- Impact: none on the test posture or any suite.
- Approved by: (owed — the approval of the F20-a harness PR carrying this revision)
Rev 6 — 2026-09-10 — the backend test layout follows the single-module collapse (AD #31, FS-14)
Structural, not a policy change: no tier, tool or coverage decision is reversed; the backend test tree, container arithmetic and ArchUnit description are repointed onto the one module FS-14 shipped.
- § Test Naming & Layout — backend tree redrawn onto
src/test/groovy/com/empireteas/{platform,catalog,pricing,…}with the arch guards inarch/and the shared container base insupport/SharedPostgresSpec. The frontend tree dropspackages/*/apps/webforsrc/{ui,features}. - T2 container rules — “three containers” becomes “one”. One module means one test JVM and one shared static container;
java-test-fixturescross-module publication is gone (the base spec sits in the single test source set). The raw-JDBC-vs-Boot distinction is now per spec, not per module. - The Always-Test row and the ArchUnit/Flyway toolchain rows drop the retired boundary + banned-vocabulary rules and the two-schema version partition; the ArchUnit suite is restated as the hygiene guards (layering, feature-slice acyclicity,
@NullMarked, no schema qualifier, fixture unreachability), Flyway as onepublicschema on one sequential version line. - Not changed: T1/T2/T3/T-E2E tiers, tools, the Slow-lane budget, the fixture-unreachability guard, or any coverage target. Approved by Tharinda, 2026-09-09 (the AD #31 lead sign-off).
Rev 5 — 2026-08-25 — the two Rev 4 proposals accepted by the team
The two items Rev 4 deliberately left as proposals are now decided — ruled on rather than merged past, which is what Rev 4 asked for. The § Non-Deterministic & External Surfaces (Graph) and § Flaky Tests markers are lifted accordingly; the Rev 4 entry below is left intact as the point-in-time record of when the items were raised as proposals.
- Microsoft Graph CI posture — accepted. T1/T2/T3 stub the
MailboxProviderseam; live Graph is exercised manually against dependency row 4’s three test accounts and recorded as evidence, not as a blocking CI gate — for the reasons stated inline (long-lived Microsoft credentials in CI sit badly against the no-static-keys posture AWS already meets via OIDC → IAM; Graph is rate-limited; a mailbox is shared mutable state across runs). The two things a stub cannot prove — the OAuth consent/refresh round trip, and that a sent message actually arrives (SPF/DKIM) — are exercised live at least once per phase with recorded evidence. FS-6 / the F20 spec implement this; they do not re-open it. - Flaky-test policy — accepted. Zero retries locally; CI retries permitted but a retried pass is recorded and investigated, never silently green; no sleep-based “fixes”; no quarantining (§7.8 already fails the build on a skip/
.only); a flake that cannot be fixed quickly is deleted with a recorded reason. The FS-4 toolchain wave wires it in.
No tier, tool or policy content changed beyond lifting the two markers and recording the ruling; the body of both sections already stated the accepted shape.
Rev 4 — 2026-08-24 — the surfaces that cannot be tested like the rest of the system
Additive. Two sections are explicitly proposals, not decisions (Graph’s CI posture; the flaky-test policy) and say so inline with a settle-by — they are written down so they are argued about rather than improvised per WI.
- New § Non-Deterministic & External Surfaces, covering the three places where the standard tier model does not reach:
- The AI layer. Splits the problem: the deterministic ring around the call must be tested at T1/T2 with the seam stubbed (request shape and the PII deny-list, schema rejection of malformed model output, the HITL queue with no self-approve path, provenance in the UI and in the per-call log, AI bodies absent from production logs, the
enabled: falsekill switch, and AD #19’s degrade-to-manual on outage); model and prompt quality belongs to the eval set inai-provider-strategy.md, which is a different instrument on a different cadence and is never wired intoSlow lane. The governing rule: never assert non-deterministic output in a blocking gate — that is a permanent flake that teaches the team to re-run until green. - Microsoft Graph. Stub the
MailboxProviderseam for T1/T2/T3; proposed that live Graph is exercised manually against dependency row 4’s three test accounts and recorded as evidence rather than run as a blocking CI gate, with the reasoning stated so it can be contested (long-lived Microsoft credentials in CI sit badly against the no-static-keys posture AWS already satisfies via OIDC; rate limits; a mailbox is shared mutable state). Names the two things a stub can never prove — the OAuth consent/refresh round trip, and that a sent message actually arrives, which depends on SPF/DKIM. Confirm at FS-6 or at the F20 spec. - The Pipedrive migration (F4).
AD-O-7is open and is two problems, so this records the test shape: run-twice idempotence for replay safety, and — the easily-missed half — the parallel-run reconciling pass with its three distinct cases (already migrated → skipped; created in Pipedrive during the parallel run → imported; created in the CRM → never overwritten), plus a dry-run mode, reconciliation as a client-verifiable migration report rather than a log line, and a tested rollback path. Stated because F4 gets one real attempt on live client data and its failure is discovered by Empire, not by us.
- The AI layer. Splits the problem: the deterministic ring around the call must be tested at T1/T2 with the seam stubbed (request shape and the PII deny-list, schema rejection of malformed model output, the HITL queue with no self-approve path, provenance in the UI and in the per-call log, AI bodies absent from production logs, the
- New § Flaky Tests — a proposal, confirm at FS-4. The harness said nothing about this; on a three-person team shipping browser tests to a December date, an unmanaged flake rate is the likeliest way a suite stops being trusted. Proposes zero local retries, CI retries that are recorded rather than silently green, no sleep-based “fixes”, no quarantining (§7.8 already fails the build on a skip), and deletion-with-a-reason over a permanently amber test. Names the two most likely causes on this project in advance: the shared T2 container’s order-dependence (which detonates in a sibling WI’s wave and so looks like someone else’s bug), and wall-clock time — effective-dated pricing (AD #14) plus a configured business timezone (
et.business-timezone) plus a mixed macOS/Windows team means any test reading the system clock will flake near a boundary, and differently per machine. Inject the clock. - New § Critical Flows & the Smoke Set — and a deliberate refusal to list flows. The inventory is derived from feature ACs by
/forge-releaseand lives per cycle under.forge/release/; hand-authoring one now, pre-Gate-3 with no feature ACs written, would produce a list that is regenerated at Gate 3 and then competes with the derived one (the L-005 pattern). What is recorded instead is the selection criteria for AD-O-11’s smoke tier, which do not go stale — login first; prefer the flow whose breakage is silent over one that fails loudly, because a loud failure is reported by a person in minutes; read-mostly and self-cleaning, so a wave spec is never reused unchanged; and if the set outgrows 5 minutes, drop the loudest-failing path rather than the newest. - § Accessibility corrected: an axe pass is necessary, not sufficient. Rev 3 said axe runs on every page, which is true and was being read as “AA achieved”. Automated tooling catches a minority of WCAG failures and is weakest precisely where the PRD is most specific — it can confirm a control is focusable but not that a Kanban board is operable without a mouse, that tab order makes sense, or that focus lands usefully after a dialog closes. A keyboard-only walkthrough of each core screen, recorded as evidence is now a named obligation alongside the scan.
- § Test Checklist gains four rows — AI, mailbox, accessibility (scan and walkthrough), and injected time.
Rev 3 — 2026-08-24 — the gaps filled where other artifacts already point here as the authority
Additive. No tier, tool or policy decision is reversed; every item either records a decision taken elsewhere or names an obligation that already existed implicitly.
- New § Test Naming & Layout.
e2e-test-implementeris instructed to read the test-directory layout from this file and had half a sentence to work from. Now carries both repos’ trees, the*Spec.groovy(T1) vs*IT.groovy(T2) distinction — needed since FS-2 puts container-backed specs in three modules — where thetestFixturesbase spec lives, and the capture-app rule that service-worker assertions run againstpnpm preview, neverpnpm dev(the dev server has no worker by design, so such a test passes for the wrong reason). - New § The Always-Test List, pulled out of the § Coverage Targets paragraph and extended from six items to thirteen, each anchored. The additions are all things this project has already decided and this file did not know: AI never acts autonomously (PRD § Governance — the highest-consequence rule in the product, previously with no test obligation anywhere), AD #26 (a check names the permission, never the role — and the load-bearing test is the negative one), AD #13 (deliver twice → one side effect), AD #14 (no in-place update; as-of reads), AD #12 (no
@Scheduled), AD #17 (ArchUnit is the acceptance test — this file had never mentioned ArchUnit), the sub-floor gate, and three of the four §0.1 failure shapes where only runtime-freshness was listed. - T2 gains the container operating rules settled at FS-2 — one container per test JVM and therefore three, not one; shared via
java-test-fixturesfromcore-platform; no Spring context incore-platform/core-commerceand what that implies for exception tests (SQLState 23505, notDataIntegrityViolationException);et.datasource.*not@ServiceConnection;testprofile neverlocal. Placed here because the FS-2 plan already cites this file as governing, and the next person to add a container spec will read this, not a closed slice’s plan. - The 10-minute
Slow lanebudget is now stated (coding-standards.md §6.8; FS-2 NFR-1/AC-23 make it measured), including the rule that the lever is fewer container-backed specs, never a raised cap. This file previously carried no speed constraint of any kind. - § Test Data & Isolation filled in. The six roles are named rather than counted, with the note that the Managing Director is a permission holder, not a seventh role — building that fixture as a role would bake in the exact conflation AD #26 exists to prevent. Seed data: mechanism settled at FS-2 (a
local-only Flyway location), dataset still open asAD-O-8, whose slice-1 settle-by has passed. Isolation is now a table, with the smoke tier called out as the never-reseeds exception. - § Accessibility & Security split and filled. Accessibility now states the F-004 lint gap — Biome’s
a11ygroup cannot see an input with no label at all, so a greenpnpm checkis not an accessibility pass, and axe at FS-4 is the assigned closure. Security gains an eleven-row concern × tier × what-to-verify table, including the public-bundle secret check that follows from AD #22. - New § Browsers & Viewports. Chromium primary; desktop and tablet per the PRD; whether Firefox/WebKit join T-E2E is explicitly undecided, settle at FS-4 — recorded rather than defaulted, because it is a real cost against the 10-minute budget. Dark mode named as not-tested-because-not-in-V1.
- New § Performance Baselines — the PRD’s § Performance numbers (< 2 s screens, < 200 ms interactions, < ~10 s sync AI, ~40 mail actions/min/account, 10–20 concurrent users, ~50k-row imports) with a verification method each, and the explicit note that they are targets rather than CI gates. They had no home; an unwritten target is unverifiable at UAT, which is where the client measures it.
- New § Test Tooling Summary (what we use, which repo, which slice, and a ✅ for what FS-1 actually proved) and new § Deliberately Not Tested (nightly E2E per AD-O-11, mutation testing, aggregate coverage gates, dark mode, cross-tenant isolation given AD #3’s single tenancy, a load tier, native mobile, and re-running per-feature suites during release regression). The second exists so a testing doc from another engagement does not re-import decisions this project already made.
- § When Tests Run gains a Commands column, and the smoke tier’s read-mostly constraint is restated here rather than only in
architecture.md. Also: the CI workflow ispr-validationwith jobsFast lane/Slow lane— this file still saidpr-validate, a name architecture Rev 20 corrected on 2026-08-20 (“the shipped name wins”). - Two framework-level sentences added: most tests belong at the bottom of the tier stack, and a green T1/T2 with a failing T3 points at the seam rather than the units. This file had no statement of test distribution intent at all.
- § Test Checklist and § Writing Tests extended to reference the always-test rows, and to state that a test is never weakened to make it pass (§7.8).
Rev 2 — 2026-08-24 — six stale or unsupported claims corrected
Correction-only; no tier, tool or policy decision is reversed. Every item was a statement this file made that the rest of the harness no longer supported.
- SP-ARCH-003 is resolved (2026-08-21), not open. The T2 real-dependency row described the pricing subsystem’s temporal scheme as still being settled by the spike and now points at the settled schema,
<backend-repo>/docs/data-model.md§2. This is the same cascadearchitecture.mdRev 23 caught three days earlier: the design text moved when the spike closed, and the what-is-still-unresolved text did not. Two places state a spike’s status; both have to move together. - Coverage tooling is not wired yet. § Coverage Targets claimed JaCoCo was “wired into
./gradlew check”.coding-standards.md §7.2assigns JaCoCo to FS-4, and FS-1’scheckis Spotless +-Werror+ doclint + ArchUnit + Spock. The claim promised a safety net that does not exist; it now names FS-4 and states whatcheckactually runs. - The coverage policy was quoted without its enforcement rules, so ≥ 80% / ≥ 70% read as a live gate. §7.4’s three load-bearing qualifiers are now restated inline: per package never aggregate, report through Phase 1 and gate from Phase 2, and coverage never substitutes for the always-test list.
- MSW removed as the T2 frontend framework. It was named here and nowhere else in the harness — absent from
coding-standards.md §7.3(which enumerates every frontend tool) and from FS-1 FR-F6 (Vitest + RTL +user-eventonly), and the generated API client it was to intercept arrives at FS-10. A test strategy naming a dependency the project has not adopted sends an implementer to install one on their own authority. The row now describes what is true — Vitest + RTL with the API client stubbed at the module boundary — with the transport-level question deferred to FS-10. - The threat-model pointer was dead.
.forge/design/threat-model.mdhas never been written. The security home is.forge/security/; the standing inputs until a threat model exists are the PRD’s § Security and Compliance andcoding-standards.md §7.7. - The capture-app test surface no longer asserts unresolved scope. It named QR parsing and an offline queue as known T1 surfaces; both were verified absent from the POC (L-005) and how much of F36 ships is open as OQ-55. The two settled obligations (capture → lead flow; the service worker’s shell-asset allow-list) are kept and named; the rest is explicitly deferred rather than guessed.
- Header corrected and this log started.
Last updatedread 2026-08-19 while AD #25 content (Spock2.4-groovy-5.0,RestTestClient) had landed on 2026-08-21 — the absence of a revision log is why nobody noticed.Related:now namescoding-standards.md§7 (which owns the tool list this file defers to) and the PRD (which owns the targets tests have to prove); both were missing from a file that depends on them.
Rev 1 — 2026-08-18 → 2026-08-21 — retroactive baseline
No revision log was kept for this period; it is reconstructed from git history rather than from contemporaneous entries. Authored 2026-08-18 with the toolchain decision; CI tiers aligned to AD-O-11 on 2026-08-19; the AD #25 rows (Spock on the groovy-5.0 variant, RestTestClient replacing TestRestTemplate) landed 2026-08-21 without a header date bump. Recorded as one entry rather than invented as three.