source: .forge/design/harness/test-run-cost.md · generated — edit the markdown, not this file

Test-run cost: faster suites, tests chosen by the change, and alerts on nightly failures — Design

Status: approved (Draft 5) — Phase 1 built on 2026-10-27 except the Windows items (§ Where this stands); Phase 2’s selected records switched on, and the browser gate taken off PRs into main, on 2026-11-02 without waiting for the seven nights (the lead, 2026-11-02; S-4, revision line of 2026-11-02; PRD Rev 81) Author: Nanndish Satgunarajah (Drafts 1–4); Draft 5 records the lead’s answers (Tharinda Rajapaksha) Reviewed by: one fresh-context review pass, 2026-10-25 (Draft 1 “Ready after Importants”; Draft 2 applies every finding — see ## Revisions). A second fresh-context pass, 2026-10-26, over Draft 3’s long-run assessment and Draft 4’s bug analysis (“Not ready”: one Blocker, nine Importants, all applied in Draft 4 — see ## Revisions). A third, 2026-10-27, over Draft 5 (“Ready after Importants”, no Blocker; every Important applied in Draft 5). Approval: Tharinda Rajapaksha (lead), 2026-10-27, with the answers in § Open questions and the instruction to build Phase 1 Date: 2026-10-25 (Draft 4: 2026-10-26; Draft 5: 2026-10-27) Scope: the delivery harness and the three app repos.

  • Harness: .forge/test-strategy.md (§ When Tests Run, § Flaky Tests, § Browsers & Viewports), .claude/rules/git-conventions.md § The test suites run locally, and the PR records them, .claude/CLAUDE.md § Quality Gates (the browser-gate paragraph), .forge/design/coding-standards.md §6.8 and §7.1, the F40 spec (.forge/specs/F40-accessibility-responsive-gate-spec.md, NFR-1, FR-7, FR-8, FR-10, FR-11), the FS-4 spec (.forge/specs/foundation/004-build-ci-spec.md, FR-G6 and FR-J3), .claude/commands/forge-deliver.md (Stage 7, Stage 12, Stage 15), .claude/skills/forge-test-verify/SKILL.md, the hooks inject-gate-state.sh and guard-browser-gate.{sh,mjs}, .forge/design/harness/load-check-and-flaky-tests.md (a revision pointer), .forge/team-guide.md, .forge/project-prd.md § Accessibility (P-9), .forge/design/architecture.md AD-O-11, the release and ship rules that count a nightly run (§ What changes elsewhere), and the new .claude/lib/nightly/ and .claude/lib/flaky-report.py.
  • forge-et-be: build.gradle (the test task), src/test/groovy/com/empireteas/support/ (the shared containers), the slowest specs named in § Evidence, scripts/local-build-evidence.py, .github/workflows/pr-validation.yml, CLAUDE.md § Common commands.
  • forge-et-fe: vitest.config.ts, playwright.config.ts, e2e/browser/size-matrix/, e2e/support/pageCheck.ts, scripts/browser-gate.mjs and scripts/browser-gate/ (run.mjs, ci-check.mjs, gate-inputs.json), scripts/local-test-evidence.mjs, .github/workflows/pr-validation.yml, e2e/README.md § Run time, CLAUDE.md § Common commands.
  • forge-et-pwa: the byte copies of forge-et-fe’s shared files (scripts/shared-with-fe.json), vitest.config.ts, src/test/scripts/sharedDrift.test.ts, .github/workflows/pr-validation.yml, CLAUDE.md.

Nothing here is cloud-gated, so .forge/cloud-gated-validation.md gains no row. AWS cost: none. GitHub Actions minutes: fewer than today. The nightly full run moves from GitHub’s runners to the lead’s Mac (N-3), which saves each app repo’s nightly Slow lane (about an hour of minutes a night across the three repos); the M-8 Windows job, which would have added minutes, is not built (Q-10). Every change to a workflow file needs the lead’s approval at its PR (.claude/CLAUDE.md § Boundaries, NEVER DO).

Draft 5 in short (2026-10-27). The lead answered the open questions on 2026-10-26 and 2026-10-27 (§ Open questions gives each answer). Two answers reshape the design:

  • The nightly full run of all three app repos moves to the lead’s Mac (N-3). A scheduled job at 02:00 Sri Lanka time runs the backend suite, both Vitest suites and both browser gates in full on main, and raises a failure as an issue assigned to the whole team (N-1). CI’s scheduled run keeps its Fast lane (the nightly Trivy scan AD #38 requires) and skips its Slow lane; the Slow lane still runs on release/*, Dependabot PRs and on demand. (Since N-5, on demand only: a push to release/* from 2026-11-03, a Dependabot PR from 2026-11-13.)
  • A PR runs less, because the nightly full run covers every test. An ordinary PR’s browser gate checks four sizes, and on Windows only Chromium; a test that passes on its one retry is reported as flaky; a change to unit tests alone needs no new gate; heavy runs on one machine queue instead of overloading it; and the records and the gate later run only the tests a change can affect (Q-1).

Not adopted: WSL 2 for Windows runs, a GitHub self-hosted runner, dropping the Test record, and ego(lite) (§ Rejected alternatives).

Context

CI does not run the test suites on an ordinary pull request (.claude/rules/git-conventions.md § Pull Requests). Each developer runs them on their own laptop — the backend build record (python3 scripts/local-build-evidence.py), the frontends’ test record (pnpm test:record) and the browser gate (pnpm e2e:gate) — and every one of those runs the whole suite, every time. /forge-deliver also runs the whole backend suite once per work item as well as on the wave PR (forge-deliver.md Stage 15 step 4: “Each WI’s run covers the whole backend suite on that WI’s own branch”). The team works on macOS and Windows laptops, often with several Claude sessions, backends and Testcontainers stacks running at once.

The suites have grown much faster than the time anyone budgeted for them. The developers report three effects: runs take too long, failures happen that have nothing to do with the change (more often on Windows), and a machine running a suite cannot be used for parallel work. § Evidence measures all three. The 2026-10-15 design load-check-and-flaky-tests.md stopped runs from starting on an already overloaded machine and recorded the load; it did not make any suite cheaper, and a suite on its own still saturates the machine.

This design collects every change found worth making, in nine tracks, and orders them into three phases: changes that leave every rule as it is, changes that need the lead’s decision, and larger engineering work. Each change states what it saves, what it risks and how its effect is measured.

Evidence

All figures were measured between 2026-10-22 and 2026-10-25 from the sources named. They are point-in-time observations; § Appendix gives the command that re-checks each one.

E-1. How long each run takes

Run Tests macOS, 8 cores Windows, 12–16 cores
forge-et-be build record 10,158 median 21 min (n=20) median 36 min (n=31), slowest 131 min
forge-et-fe test record (Vitest) 8,877 median 2.1 min (n=22) median 4.8 min (n=14), slowest 23 min
forge-et-pwa test record (Vitest) 731 median 3.6 min (n=12) 12 and 31 min (n=2)
forge-et-fe browser gate 5,627 48–51 min at 4 workers (BUG-233, BUG-234, BUG-239) 2.6 h at 2 workers (F1-c Run 4), about 3 h at 3 workers (BUG-218)
forge-et-pwa browser gate 131 a few minutes —

Source: the local-record comments on the last 80 PRs of each app repo (as of 2026-10-22), and the bugs named. Windows takes about twice as long as macOS on twice the cores, which points at the operating system and its file system rather than at the tests (M-1).

E-2. Growth, and the CI caps

  • The CRM browser gate ran 348 tests in about 4 minutes on 2026-09-29 (forge-et-fe e2e/README.md § Run time, inside F40 NFR-1’s 10-minute target) and 5,627 tests on 2026-10-22. NFR-1 says that when a measured run exceeds the target, “the plan says what is cut (for example the 1440 Chromium size in forge-et-fe) rather than silently raising it”. Nothing has been cut, and the README still gives the 348-test figure (owed.md row “Correct the browser gate’s run time”).
  • The backend suite grew 15% in tests and 85% in time between 2026-10-11 and 2026-10-15 (forge-et-be#299); the nightly Slow lane passed its 30-minute cap and was cancelled every night from 16 to 21 October, and the cap became 60 minutes on 2026-10-22 (coding-standards.md revision of that date). The nightly runs of 22 and 23 October passed in 31 and 27 minutes.
  • The forge-et-fe nightly Slow lane is next. On 2026-10-23 (run 40013019756) it ran 24 minutes of its 30-minute cap, Vitest reporting Duration 1421.43s. The backend reached its cap the same way.

E-3. Where the browser gate’s time goes

pnpm exec playwright test --list on forge-et-fe main (2026-10-22): 5,627 tests in 214 files (5,634 in 217 on 2026-10-25).

Project Tests Share
The size matrix: 7 projects (size-<w>x<h>-<engine>) × about 548 page states 3,838 68%
chromium (the functional and per-feature browser specs) 1,140 20%
seam (browserless API specs against the real backend) 639 11%
setup 10 —
  • Every page state runs the full page check at every size, axe included (size-matrix.spec.ts → expectPageCheck, pageCheck.ts AxeBuilder.analyze): about 3,800 axe scans per gate.
  • Outside the size matrix, 51 browser spec files also run axe or the page check. 29 *-a11y files hold 201 tests and 8 *-layout files hold 53. Some of them check page states the size matrix already checks: BUG-246’s one 10px defect failed four tests in four files (pinned-first-column, accounts-layout, contacts-layout and the size matrix).
  • The gate tests against the Vite dev server (playwright.config.ts webServer → pnpm dev), which transforms each module on its first request. Three bugs come from that: BUG-079 (deep links time out while the route is transformed), BUG-196 (the dev server refused a request under load) and BUG-222 (the route warm-up failed on a first start).
  • Only the size matrix spreads its tests across workers. The config sets no fullyParallel, so every other spec file runs its tests in order in one worker; size-matrix.spec.ts:39 is the one file with mode: 'parallel'. A long file such as users/journey.spec.ts is one worker’s work for its whole length. 23 files also set mode: 'serial', which in addition skips the rest of the file after a failure.
  • Every gate runs every test, whatever the change. scripts/browser-gate.mjs has no option to run part of the suite. Its gate inputs (scripts/browser-gate/gate-inputs.json) cover all of src/, e2e/ and scripts/, so any code change needs a full new run — a change to a Vitest test file included, although no browser test loads one (git grep finds no import of src/test or of a .test.ts(x) file under e2e/ or in playwright.config.ts).

E-4. Where the backend suite’s time goes

From the JUnit results of the nightly run of 2026-10-23 (forge-et-be run 40012533032, artifact backend-reports): 10,164 tests in 784 classes, 23.7 minutes for the test task, run in one JVM, one spec at a time (no maxParallelForks).

  • 3.9% of the tests (397, each 0.5 s or longer) take 68% of the test time; 1.4% (143, each 2 s or longer) take 48%. By package, mailbox takes 5.4 minutes, then lead and api 3.6 each and platform 2.4.
  • The slowest tests wait on real time, not on work: a dead-letter cycle through LocalStack (ScanResultQueueIT AC-3, 39 s; PoisonDlqIT, 11 s), a 28-second deadline (EmailSendDeadlineIT), a 17-second rate-limit retry (MailboxIngestFailurePathsIT), a 20-second health check (ActuatorSpec “health reports UP”), and queue round trips in the mailbox, email-send and lead-research specs. forge-et-be#299 trimmed some of these waits.
  • Response-time tests run in the functional suite: PriceEmailResponseTimeIT (7.8 s), EmailTemplateResponseTimeIT (7.0 s), PipedriveDryRunTimingIT (6.3 s). A test that asserts elapsed time fails on a loaded machine by design (BUG-130, L-125).
  • The test Postgres runs with default durability. SharedPostgresSpec starts postgres:18-alpine with withCommand('postgres', '-c', 'max_connections=300') and nothing else. That call replaces Testcontainers’ default command, which sets fsync=off, so fsync is on today; the data directory is not on tmpfs.
  • Spring contexts vary: 93 test files set properties (@TestPropertySource, @DynamicPropertySource or properties =), 31 use ApplicationContextRunner, 7 replace beans with mocks and 5 use @DirtiesContext. Each distinct configuration starts its own context with its own Hikari pool, which is what filled a 2 GB heap (BUG-247, now 3 GB) and the container’s connection limit (BUG-088).
  • JaCoCo instruments every local record: the jacoco plugin is applied, and check depends on jacocoTestReport, although coverage is report-only through Phase 1 and the nightly CI run already produces the report.
  • org.gradle.caching=true is on, but the test task declares all of src/main and src/test as inputs, because several specs read source as text (the comment above inputs.dir('src/main') in build.gradle), so any source change re-runs every test.

E-5. Where the Vitest suites’ time goes

From the nightly CI logs of 2026-10-23 (forge-et-fe run 40013019756, forge-et-pwa run 40012938098):

  • forge-et-fe: 9,113 tests in 484 files. Vitest’s own breakdown of the summed time is “tests 55%, environment 22%, import 15%, setup 6%”: 43% is spent on starting jsdom, importing modules and the setup file, not on tests. About 180 .test.ts files never touch the DOM (no render, document, window or Testing Library), yet run in jsdom; 15 already set @vitest-environment node. The slowest 10% of files take 68% of the test time; one file, AccountListScreen.test.tsx, takes 72 s, then OwnedRecordsList 26 s, LeadFormDialog 24 s and AccountsFilters 24 s. On a 16-core machine a 72-second file bounds the whole run.
  • forge-et-pwa: 731 tests in 29 files; one file, src/test/scripts/sharedDrift.test.ts, takes 61 s of the suite’s ~100 s. It builds two git repositories in a temporary folder for each case.
  • Tests of the harness’s own tooling live in each frontend’s suite (src/test/scripts/, 14 test files in forge-et-fe). They start git and node processes, which are slow on Windows, and account for several Windows-only bugs: BUG-172, BUG-215, BUG-216 and BUG-244.
  • 47 forge-et-fe test files read source files as text (readFileSync, readdirSync), including AD #39’s enforcing test src/app/shadcnOnly.test.ts and src/test/tokenLiterals.test.ts. Selection by imports cannot see what they read (S-7).
  • Vitest’s default worker count is the number of cores minus one. On a 16-core Windows machine that is 15 jsdom workers; the records show a CPU busy share of 0.98–0.99 for the whole run (BUG-220).

E-6. Flaky failures, skips and deferred work

  • 47 of the 263 bugs in .forge/bugs.md on 2026-10-26 are flaky or intermittent test failures, and 42 were found on Windows or are about it, open and closed together; E-10 reads every one of them. Most pass when run alone and fail only in a full run under load. Each takes a long investigation and write-up (test-strategy.md § Flaky Tests, D-4 of load-check-and-flaky-tests.md).
  • 5 of the last 19 ready forge-et-fe PRs merged with Browser gate: skipped. In three (forge-et-fe#287, #311, #329) full runs kept failing on tests the PR did not touch; in #312 a re-run failed when the laptop slept, after a full run on an earlier commit had passed; in #327 the gate failed on BUG-246, a real Windows-only defect already on main. The cost of the attempts was paid each time.
  • Fixes are being put off to avoid a re-run. owed.md holds “the nine Nits left by the forge-et-fe#284 delta review … left so as not to re-run the two-hour browser gate before merge”. Such Nits often touch only unit tests, which today still force a full gate (E-3).
  • At a machine’s full load, about 0.5–0.7% of the browser tests time out somewhere (BUG-218). At 5,627 tests that is 28 to 39 failures a run, so a full run almost never passes in one attempt.

E-7. Nightly runs and who hears about a failure

  • Each app repo’s PR validation workflow runs at 19:30 UTC on main. A job, Nightly — main changed, skips the lanes when main has not moved since the last full run, and fails (“Nightly still red”) when that last run failed. The workflow’s concurrency group cancels a run in progress when another starts on main (cancel-in-progress: true; a manual run on main cancels the nightly, and the other way round). A job that reaches its timeout-minutes also ends cancelled (run 39662818510: “Slow lane cancelled”).
  • No CI job runs a browser. The browser tests run only on developers’ machines, in the browser gate.
  • A failure reaches one person. GitHub sends a scheduled run’s notification to the person who last changed its cron line, according to that person’s own notification settings (GitHub Docs: Notifications for workflow runs). git-conventions.md asks anyone who sees a red run to say so in the team channel. The backend nightly ended cancelled six nights running (16–21 October) and was noticed on the seventh day, during F13’s Reflect (L-223).

E-8. What the size matrix has found

grep -E '^### BUG-[0-9]+ — .*size.matrix' .forge/bugs.md on 2026-10-25: 20 bugs name the size matrix in their title. 10 are titled “Flaky”, and six more describe an intermittent failure (BUG-013, BUG-045, BUG-171, BUG-175, BUG-183, BUG-184). Two were product defects: BUG-232 (a target-size failure at 320 × 640) and BUG-133 (a 1px role select in WebKit at 1024 × 768). The rest are defects in the tests or their marks (BUG-148, BUG-245). Layout defects at 1280 and 1440 have been found too, by specs and people rather than by the size matrix (the centred-column bugs, whose regression tests run at both widths). The matrix is worth keeping at every size; on every PR, most of what its extra sizes produce is failures under load — races in the tests that load exposes (E-10).

E-9. How fast the suites grow

The tracker counts 43 Phase 1 features done and 59 still to build (16 in Phase 1, 43 in Phase 2) on 2026-10-25. Measured against the features done, the suites carry about 130 browser-gate tests and about 235 backend tests per feature, Foundation’s tests included. If later features add tests at that rate, the CRM browser gate reaches about 13,000 tests and the backend about 24,000 by the end of V1. Phase 2’s features (deals, quotes, samples, the capture app) are larger than Phase 1’s, so these are low estimates. A full gate would then take one and a half to two hours on an 8-core Mac even after this design’s cuts to the size matrix, and the backend’s full record about 50 minutes on a Mac and 85 on Windows at today’s rate per test.

E-10. What the flaky and Windows bugs show

Every bug in .forge/bugs.md titled as a flaky or intermittent test failure, or found on or about Windows, was read on 2026-10-26, open and closed, with its tracker.yaml status and notes. The figures are as of that date.

Where they are. Of the 47 flaky bugs, 30 are in the browser gate (14 of them size-matrix states, 11 of those in WebKit), one in the dev smoke run, 9 in Vitest and 7 in the backend suite. 20 are still open or in-progress, and 16 of those 20 are on Windows: all six open Vitest bugs (BUG-213, BUG-214, BUG-215, BUG-220, BUG-223, BUG-225 — each a case that runs past its 20 seconds in a full Vitest run under load) and ten browser bugs. Four of the 20 are in-progress, all on Windows, the oldest found 17 days ago: three WebKit size-matrix states (BUG-013, BUG-129, BUG-183) and a refused dev-server connection (BUG-196); forge-et-pwa’s BUG-184, an intermittent WebKit timeout not titled flaky, is in progress beside them. On macOS, 7 of 13 flaky bugs are fixed, 4 closed wontfix and 2 open; on Windows, 9 of 25 are fixed. The rest were seen on Linux CI or name no system.

They are arriving faster. Bugs titled flaky, intermittent or Windows, by the ISO week of their found_date: 7 of the 56 bugs found in week 40, 8 of 73 in week 41, 24 of 61 in week 42 and 18 of 41 in week 43 — about 40% of the bugs of the last two weeks, while the gate grew past 5,000 tests (E-2).

What the fixed ones were. From the 23 fixed flaky bugs’ notes:

Cause Bugs What addresses it
A race in the test: it acted or asserted before the page or its data had settled, or waited for an element to be visible rather than for what its action produces at least ten: BUG-057, BUG-081, BUG-084, BUG-110, BUG-182, BUG-205, BUG-230, BUG-233, BUG-234, BUG-239 Each fix was permanent. Load exposes these races; it does not cause them. frontend-reviewer’s dimension 14 already looks for them (L-101); F-5
A bound on elapsed time, or more work in one case than its time allows BUG-077, BUG-082, BUG-128 (one case cut from 26 clicks to 6), BUG-130, BUG-216 B-3 and B-4 in the backend; F-5 for Vitest. Five of the six open Vitest bugs have BUG-128’s shape
A test that reads every page of a list that each run adds to BUG-227, and BUG-173 (not titled flaky) P-7; F-5
LocalStack answering slowly inside Docker’s VM during a full build BUG-195, closed wontfix as overload on 2026-10-15, came back as BUG-259 on 2026-10-25 on a machine within the load limit B-2; F-2 matches a recurrence to the closed bug
A product defect first seen as a flaky test BUG-261 (the list search box dropped keys typed while its own debounced write landed — that part fixed in forge-et-fe#338; the bug stays open for the rest), and F11-f’s finding F-001 Why F-1 is adopted only with F-2

All four wontfix closures (BUG-149, BUG-194, BUG-195, BUG-197) are macOS overload outcomes, and BUG-195’s came back ten days later. A closure as overload is a judgement on one run, not proof that nothing else is wrong.

Windows bugs that are not flaky (17, 15 of them fixed):

  • Test and tooling code that assumed macOS: \-separated paths (BUG-038, BUG-242, BUG-244 — the last two the same defect, filed by two people on the same day), ./gradlew where Windows needs gradlew.bat (BUG-073), and platform marks and skip lists in the gate (BUG-131, BUG-188, BUG-245). Each was found only when a Windows developer’s record or gate failed on it, and stopped that developer until it was fixed (M-8).
  • Limits a Windows machine reached first: the Groovy compiler’s and the test JVM’s heaps (BUG-243, BUG-247), a git-heavy self-test (BUG-172), a local database grown by earlier runs (BUG-173).
  • A real product defect macOS did not show: the pinned first column 10px off because Windows scrollbars take space (BUG-246). It is why Windows Chromium runs keep their value (P-9; N-3’s trade-off). (BUG-203, found on a Windows laptop, is not Windows-specific: its list overflows any 1280 × 720 window, which is Playwright’s default viewport.)

What it changes. 38 of the 47 flaky records describe a failure in a full run or under load, so the cuts that lower the load (P-1, P-2, V-1, V-4, M-3, M-5) make them appear less often. But most of the fixed ones were races in the tests that load only exposed: each is still fixed, and F-5 adds the review rules that the remaining causes lack. Windows holds most of the open backlog, which M-1, V-4 and P-9 address. Product defects found through flaky tests and on Windows Chromium are why a retried test is still reported (F-1, F-2) and why Windows keeps its Chromium runs (P-9).

Terms

  • Full run: every test of a suite, as every record and gate runs today.
  • Selected run: the tests a change can affect, chosen by a script from the files the change touches (S-1 to S-3), plus the always-run set.
  • Full profile and PR profile (the CRM’s browser gate only, P-1, P-9): the full profile runs every Playwright project — all seven sizes, Chromium and WebKit; the PR profile runs one size per layout band, and on Windows Chromium only. A gate can be a full run in the PR profile (every spec, four sizes) — what an ordinary PR that touches shared code carries (S-4(a)). The full profile runs at a wave PR, a PR into release/*, with --full, and every night (N-3). The Browser gate: line names the PR profile (· 4 sizes, · 4 sizes, chromium only) and names nothing for the full profile.
  • Shared code: files whose change can affect any feature. Per repo: forge-et-be — com.empireteas.platform, com.empireteas.auth, src/main/resources/ (migrations and configuration), build.gradle, gradle/, src/test/groovy/com/empireteas/support/; forge-et-fe and forge-et-pwa — src/app/, src/ui/, src/lib/, e2e/support/, e2e/fixtures/, playwright.config.ts, vite.config.ts, vitest.config.ts, package.json, pnpm-lock.yaml, scripts/. The lists live in one file per repo (S-5) once built; this paragraph is the starting point.
  • Always-run set: the tests every selected run includes: every test that reads source or project files as text (S-7); in forge-et-be, com.empireteas.arch and the migration specs when a migration changed; in the frontends, the shell, auth and sign-in specs and a small fixed sample of size-matrix states.
  • Flaky test: a test that failed and then passed on an immediate retry of the same code in the same run.
  • Layout band: a width range in which the CRM renders one layout, bounded by src/ui/breakpoints.ts: reflow (below tablet, checked at 320), tablet (768), laptop (1024, the laptop breakpoint) and desktop (1280 and 1440).

Goals

Each goal is measured with the record comments, .browser-gate/result.json and the nightly runs; each phase re-measures before the next starts.

  1. Every nightly failure, and every night the nightly run did not run, reaches the whole team by the next working morning (N-1, N-2).
  2. A PR that changes one feature’s code spends minutes, not an hour, on the browser gate — 15 minutes or less on an 8-core Mac for a typical one-feature PR — and the full gate, run nightly and at the wave PR, falls well below its present length (S-2, P-1 to P-4, P-8).
  3. A full browser run passes with no failed test and no more than Q-2’s limit of flaky tests on an otherwise idle machine on both operating systems — BUG-218’s regression test: three such runs in a row on the team’s Windows machine.
  4. The backend build record falls by a third or more on both operating systems without removing a test that checks behaviour (B-1 to B-9).
  5. A developer can keep working while a record runs: a run uses half the cores by default and never two heavy runs at once (V-4, M-3, M-5).
  6. Each suite’s nightly length and test count are reported every night, with the change since the night before (N-4), so growth like E-2’s is seen in days. (Draft 4 measured this against CI’s timeout-minutes; the nightly run no longer runs in CI, N-3.)

Long-run assessment

The suites will roughly double again before V1 is signed off (E-9). Each change below is judged by what it is worth once that has happened, not only today.

Three kinds of change, with different lifetimes:

Kind Changes What happens as the suites grow
Changes whose cost follows the change, not the suite S-1 to S-3 and S-7 (selected runs), W-1 to W-3, N-3 (the nightly full run, off each PR’s path) A PR runs the tests its change can affect, so its cost stays about the same as the suite doubles. These are the only changes that keep a PR’s test time in minutes at the end of V1.
One-off cuts to what each test costs B-1, B-2, B-3, B-5, B-6, B-7, V-1, V-3, V-4, P-1, P-2, P-3, P-6, P-8, P-9, M-1 Each removes a share of today’s time, and growth takes that time back: a 40% cut is used up once the suite has grown by about two-thirds, some 30 features at today’s rate (E-9). They are still worth having — they make every full run cheaper for the rest of the project, the nightly ones included — but they do not solve the problem on their own.
Rules for new tests H-3, H-5, F-5, M-8 They stop the cuts above being undone, one new slow or flaky test at a time. A refactor that is not paired with such a rule is redone a few weeks later.

What follows from that:

  1. Full runs belong off each PR’s path, and N-3 is the change everything else leans on. Selected runs on PRs are only safe with a nightly full run behind them, and the frontends had none for the browser suite (E-7). The lead decided (Q-4, 2026-10-27) that the full run of all three app repos happens every night on the lead’s Mac (N-3), so a developer’s machine runs only what a PR needs.
  2. The cheap, configuration-only cuts come first (P-1, P-2, P-8, B-1, V-1, V-4): each is a few lines, can be turned back by the same few lines, and removes the largest measured shares. The engineering refactors of Phase 3 are taken only where Phase 1 and 2’s measurements show a need, never all at once.
  3. Every speed fix ships with its rule for new tests (H-5), in the same PR, or the saving is temporary.
  4. New tooling is kept small and can be switched off. The harness’s own tooling tests are already a source of Windows bugs (E-5), and every new script is code the team maintains on two operating systems and copies into forge-et-pwa. So: configuration before code; each selection mode falls back to a full run whenever it cannot decide (an unknown path, a map that fails its test, a missing merge base) and has a --full switch; and a mode that is not used, or is not saving time, two months after it lands is removed.
  5. The retry (F-1) treats a symptom. Load is what makes a flaky test fail, but most of the fixed ones were races in the test that load exposed (E-10); N-3, M-5, V-4 and the one-off cuts lower the load, and fixing each race, with F-5’s review rules against new ones, removes the cause. F-1 stops a passing PR being blocked meanwhile, and F-2’s report keeps every retried test counted. The lead reads the report every week (Q-2), so a race that is a real product defect (BUG-261 and F11-f’s finding F-001 were) is still found.
  6. The size matrix keeps every size, every night. E-8 shows its extra sizes mostly produce failures under load on PRs, but BUG-133 shows a size dropped from PRs (WebKit at 1024) can carry a real defect. Under P-1 such a defect is found at the wave PR’s gate or the next nightly, a day or two later than today.
  7. Each operating system runs what it is evidence for. Safari runs only on Apple systems; Playwright’s WebKit is a patched build, close to Safari’s engine on macOS and an engine no user runs on Windows; Windows Chromium finds layout defects that macOS’s overlay scrollbars hide (BUG-246). P-9 and N-3 place each browser run where it finds real defects — with N-3 on a Mac, Windows Chromium is checked by Windows developers’ gates only (N-3’s trade-off) — and M-8 catches tooling written for one system before a developer on the other hits it.

Is the payback worth the work? Today a ready forge-et-fe PR spends roughly an hour (macOS) to three hours (Windows) on its gate, often more than once, and a backend PR 20 to 40 minutes on its record, each repeated after every later push. Phase 1 is a few days of work in total, mostly configuration and small scripts, and is paid back within its first week across four developers. Phase 3’s items are larger and are each taken on their own measured case.

Decisions

Each decision names its phase (§ Rollout). Phase 1 leaves every rule about what a PR must carry as it is; its workflow edits still need the lead’s approval at their PRs. Phase 2 needs a decision by the lead (§ Open questions). Phase 3 is larger engineering work, started after Phase 2’s measurements.

Track N — Know when the full suite fails

# Decision Why Phase
N-1 The nightly job (N-3) raises every red night as an issue assigned to the team. When a suite fails, or the job cannot run it, the job opens one GitHub issue in that app repo titled Nightly red on main, labelled nightly-red, assigned to the four delivery-team members from the * line of that repo’s .github/CODEOWNERS, or adds a comment to the open one. The issue names the suite, the commit, the commits since the last green night and the failing tests (or why the suite did not run). When a later night passes, the job closes the issue with a comment; so does a run by hand, --only included, when every suite it ran passed and every other suite of that repo last passed (2026-11-13). A suite whose repo (and, for the CRM’s browser gate, forge-et-be) has not moved since its last green night is not run, and nothing is raised. The job uses the lead’s own gh login; no secret and no workflow. GitHub does not notify the person who opens an issue, so the lead learns of a red night from N-2. A red night caused by a defect still gets a BUG-NNN through the bug flow (.claude/CLAUDE.md § Boundaries), linked from the issue. (Draft 4 built this as a CI job; with the nightly run off CI, the job that runs the suites raises it.) An assigned issue notifies every assignee (each by their own notification settings, which for an assignment are on by default) and stays visible until it is closed, so a red night no longer depends on one person (E-7). 1
N-2 Each Claude session shows open nightly failures, and a missed night, at its start. inject-gate-state.sh adds one line per app repo with an open nightly-red issue (Nightly red on main: forge-et-be since 2026-10-16 (#NNN)), and, after 06:00 Sri Lanka time, a line when the nightly log (N-4) has no entry since that day’s 02:00 (Nightly run: none since <date> — on the lead's Mac, clear the cause (mains power, lid open, the gates' ports 15173 and 15174 free) and run python3 .claude/lib/nightly/run-nightly.py --now this morning, since 2026-11-03; the ports since 2026-11-04); the job writes an entry every night, an unchanged night included, so a weekend does not read as a missed night. It makes one gh call for each, caches the answers for 30 minutes in the user’s temp folder, bounds the calls with a watchdog that works on macOS (which has no timeout command), and prints nothing when gh is missing, offline or slow. git-conventions.md already says a failed nightly is fixed before new work starts; the session start is where that rule is acted on. A Mac that slept through the night raises nothing itself, so the missed night has to be noticed by its absence. 1
N-3 The full run of all three app repos happens every night on the lead’s Mac (Q-4, 2026-10-27), by a scheduled job — a user LaunchAgent at 02:00 Sri Lanka time, not a GitHub runner — that runs only main. In its own worktrees ($WS/worktrees/nightly/<repo>, reset to origin/main each night), one suite after another: forge-et-be’s full build with the response-time tests and coverage (B-4, B-6); forge-et-fe’s and forge-et-pwa’s whole Vitest suites with coverage (pnpm test:coverage); then the backend on an isolated stack with a fresh database (the compose.verify.yaml stack, project et-nightly, removed afterwards, which also gives each night P-7’s empty database) and forge-et-fe’s full browser gate against it, all seven sizes, Chromium and WebKit (P-1, P-9); then forge-et-pwa’s full gate, against a backend started the same way on a fresh stack of its own (since 2026-11-13, for F36-a’s seam specs; revision line of that date). It runs under caffeinate, takes the machine’s run lock (M-5) so it waits for any gate or record already running (for at most an hour), and never stops anything else on the machine. Its own ports, so the lead’s own stack can stay up: the compose.verify.yaml stack on Postgres 56432 and LocalStack 46666 (not that file’s own 55432 and 45666, which a developer’s verify stack may hold; NIGHT_DB_PORT, NIGHT_LOCALSTACK_PORT), and the backend on SERVER_PORT=18080 (ET_BACKEND_PORT=18080 for the gate). Each gate serves the app on a port of its own, the CRM’s on 15173 and the capture app’s on 15174 (since 2026-11-04; each frontend’s src/lib/config/e2eAppPort.ts), so a pnpm dev on 5173 can stay up overnight; the job retries a gate whose port is still held within the lock wait, and reports that gate not run when it stays held. Limits: each suite has a time limit (backend 60 minutes, each Vitest suite 20, the CRM gate 180, the capture app’s gate 90) and the job a deadline of 07:30; a suite that reaches either is stopped and reported red through N-1. When the Mac slept: launchd starts a missed job when the Mac wakes, so a job that starts after 05:00 runs nothing and reports the night as missed; the lead schedules a wake at 01:55 (sudo pmset repeat wakeorpoweron MTWRFSU 01:55:00, which only the lead can run). Its worktrees are locked (git worktree lock), so /et-cleanup does not remove them, and recreated when missing. The job’s own script lives in the harness (.claude/lib/nightly/) and each night runs the version on origin/main, so a change to it is reviewed like any harness change. CI’s scheduled run no longer runs the suites: in the three repos’ PR validation workflows the Slow lane skips the schedule event, while the scheduled Fast lane stays, because AD #38 requires Trivy on each app repo’s nightly run on main. The Slow lane still runs on release/*, on Dependabot PRs and on demand, unchanged. Revised by N-5 (2026-11-03): a push to release/* runs the Fast lane only, the scheduled run is weekly and kept to refresh main’s caches, and the nightly Trivy scan is this job’s *-scan suite; and from 2026-11-13 a Dependabot PR does not run the Slow lane either. What it costs: the Mac must be on, plugged in and awake at 02:00, and one machine now carries every nightly full run; a missed night is shown at every session start (N-2). This revises AD-O-11’s “no nightly” (architecture.md), whose reasons — a nightly CI run duplicated the tag run and spent Actions minutes — do not apply to a run off CI. The trade-off of a Mac (Draft 4’s Q-4): Windows Chromium is checked every night by no machine, only by Windows developers’ own gates, which S-2 will narrow to their own features; a Windows-only defect caused by a Mac developer’s change (BUG-246’s kind, E-10) may wait until a Windows developer’s wave PR runs the full profile. It uses no Actions minutes and saves the hour the CI nightlies used. Why not a GitHub runner: no spare machine (Q-4), and a runner would execute any workflow pushed to a branch; this job executes only reviewed main. Selected runs and cut-down gates on PRs are only safe when everything still runs every night; the browser suite ran nowhere but on laptops (E-7), and the lead chose one machine for all three repos so the night’s results are in one place. 1
N-4 Each night’s results are written to one place: a comment on the harness issue Nightly full run (label nightly-log), one every night (an unchanged night says so), giving for each suite its result, test count, duration, coverage percentage where the suite measures it (B-6, FS-4 FR-J3), the 15 slowest tests and the change in count and duration since the previous night, with links to any nightly-red issue. Growth like E-2’s is then seen in days, not when the next timeout or bug appears; the log is also what N-2 reads to tell a missed night. 1
N-5 A suite runs in one place, never in two (the lead, 2026-11-03). The test suites and the browser gates run on the lead’s Mac (N-3) and on the developer’s machine (the records, the gate); CI runs them only on a manual run that asks for them (gh workflow run "PR validation" --ref <branch>; on main, -f tests=false runs the Fast lane only — on any other branch the suite still runs). A Dependabot PR is recorded locally like any other PR (the lead, 2026-11-13): before it merges, a delivery-team member runs the repo’s local record on its final head, and forge-review-dependabot.yml sets only Forge review. CI keeps the quick checks: the Fast lane on every ready PR, on every push to release/* and in a weekly scheduled run on main (Sunday 19:30 UTC) that skips a week main did not move and is kept to refresh the caches every PR restores. N-3 runs every check the Fast lane runs on main except the two that read a pull request (its title, and the comments it adds) and the wrapper validation. A PR into release/* records every test locally (its base decides it) and runs the full browser gate, on a head the protect-releases ruleset requires to be up to date before it merges; a tag on main needs the Fast lane (-f tests=false) and a fully green night on its commit. A suite run twice spends minutes and time and adds nothing; a slow suite in CI also needs a timeout long enough for its growth (E-2), which the Mac does not. The cost is that no suite runs on a clean Linux runner for release/* or a tag, which the lead accepted. 1

Track S — Run the tests the change needs

# Decision Why Phase
S-1 Vitest selects through its module graph. The test record and the development loop gain a --selected mode that calls vitest --run --changed <merge-base>, which runs the test files that import a changed file, directly or through other modules, plus the always-run set (S-7). A change to a shared-code path (S-5) runs everything. Built into Vitest; the module graph is exact for unit and component tests that import what they test. 1 for the development loop and work items; 2 for the record (decided in Q-1; after N-3’s two weeks, S-4)
S-2 The browser gate selects by feature, through an explicit map. Playwright’s own --only-changed follows imports, and browser specs do not import the screens they test (they open a URL), so it would miss product changes. Feature and spec names also do not line up: e2e/seam/ is flat (*.seam.spec.ts), src/features/priceEmail is tested under e2e/browser/price-email, several feature folders (leadTriage, leadResearch, notesAndTags, accountBriefing, aiDrafts, timeline) have no e2e folder of their own, and several e2e folders (access, mailbox, shell, effective-permissions) have no feature folder. So one file, e2e/feature-map.json, lists for each feature folder the browser spec files, seam spec files and size-matrix routes that test it; a test fails when a feature folder, a spec file or a page-state route is in no entry. browser-gate.mjs run --selected maps changed paths through it: a changed feature → its entry; a changed spec or helper → itself and the specs that import it (--only-changed covers this half); a shared-code path (S-5) → the full gate. The always-run set (§ Terms) is added every time. The gate’s result records selection: "selected" or "full" and the features chosen; the Browser gate: line gains · selected: leads, pricing or · full. 68% of the gate is the size matrix, and a one-feature PR touches a small share of the states (E-3). 2 (decided in Q-1; after N-3’s two weeks, S-4)
S-3 The backend selects by feature package. AD #31’s feature-first packages, with no cycles between them, make a reverse-dependency list cheap: for each changed com.empireteas.<pkg>, the packages that import it, followed until no new package is added (a scan of import com.empireteas. lines, or jdeps on the compiled classes). The record’s --selected mode runs ./gradlew test --tests 'com.empireteas.<pkg>.*' … for those packages, the api specs whose controllers belong to them (a map kept beside the shared-code list, checked by a test like S-2’s), and the always-run set (S-7). A shared-code path (S-5) runs the whole build. The record comment says selected: lead, account (+ always-run) or full. No built-in tool; the package layout makes a simple, checkable rule possible. 1 for the development loop and work items; 2 for the record (decided in Q-1; after N-3’s two weeks, S-4)
S-4 What a PR’s records must be (Q-1, decided 2026-10-27; revised by the lead on 2026-11-02). (a) A PR into main, a wave PR included, runs no browser gate; the nightly full run (N-3) is the only run of the browser suites for main. (b) Every PR’s build record and test record run the tests its change needs (S-1, S-3), and every test when it touches shared code or with --full; the backend record still runs the whole build with only the test set narrowed. © A PR into release/*, which N-3 does not test, keeps the full-profile browser gate. (d) N-3 is the full run behind (a) and (b), and N-1 makes its failures visible by the next morning; a red night is fixed before new work starts. Superseded on 2026-11-02: the 2026-10-27 form kept a full record and a full browser gate for a PR touching shared code and for every wave PR, allowed a selected gate only for a one-feature PR, and started the selected modes only after seven green or explained nights of N-3 (the lead, 2026-10-29). The lead, 2026-11-02: a browser gate cost about an hour on a Mac and two to three on Windows on every PR, for defects N-3 finds the next morning; the cost of the change is that a cross-feature break or a broken screen is found after merge, on main and on dev for up to a day, and fixed with the others that night found. Refined 2026-11-12 (fewer-nightly-reds.md R-1 to R-3, PRD Rev 93): a frontend PR into main runs the browser tests its change selects before it is marked ready, and again when main moves in the same area; still no full gate and no Browser gate: line. 2
S-5 One shared-code list per repo, a JSON file, scripts/shared-code.json in each app repo (built there in the frontends too, not beside the gate inputs), read by every selection mode. Each entry carries a one-line reason, as gate-inputs.json does. One definition, used the same way by the record, the gate and the hook, and reviewable in a PR. 1
S-6 A narrower run by hand is for the development loop only. A developer may run any subset while working (pnpm exec vitest <file>, ./gradlew test --tests '<Spec>', playwright test <file>), as git-conventions.md already says; a record or gate result always comes from the script’s own selection (S-1 to S-3) or a full run, never a hand-picked list. A hand-picked list depends on its author remembering everything the change can break; the script’s rule can be read and reviewed. 1
S-7 Tests that read files as text are always run. A scan, run by the selection scripts, finds them: in the frontends, test files that call readFileSync, readdirSync or glob (47 in forge-et-fe on 2026-10-25, among them shadcnOnly.test.ts and tokenLiterals.test.ts); in forge-et-be, the specs that read src/ or docs/ by path (RetiredKeySourceSpec, PermissionCatalogSpec, VisibilityPermissionSpec, AuditCodeDocumentSpec, StubSearchProviderDocumentSpec, the arch confinement specs — the same specs build.gradle declares file inputs for). A test of the scan fails when a file that reads source by path is not found by it. Neither a module graph nor a package closure sees what these tests read, and they enforce project-wide rules (AD #39’s shadcn check among them). 1

Track B — A faster backend suite

# Decision Why Phase
B-1 The test Postgres trades durability for speed: the data directory on tmpfs (withTmpFs(['/var/lib/postgresql': 'rw']), the parent of the PGDATA path in the PostgreSQL 18 images) and -c fsync=off -c synchronous_commit=off -c full_page_writes=off beside the existing max_connections. fsync=off restores Testcontainers’ own default, which the existing withCommand replaced. A test database lost on a crash loses nothing that matters. Every container-backed spec writes to this database; disk writes through Docker Desktop’s VM are slow on macOS and Windows. Measured before and after with the build record. 1
B-2 Queue specs call the consumer directly, except a few per queue. For each SQS queue keep the specs that prove the queue itself — delivery, redelivery after a failure, dead-lettering — and move the business-rule specs to call the listener’s handling method with a built message, without the LocalStack round trip. The kept dead-letter specs use a test queue with a short visibility timeout and a maxReceiveCount of 1 or 2. The 0.5-second-and-longer tests, mostly queue round trips and dead-letter cycles, are 68% of the test time (E-4). 3
B-3 Specs that wait on real timers use an injected clock or scheduler: the 28-second deadline, the 17-second rate-limit retry, the 20-second health check, as L-125 already requires for bounds on elapsed time. Each such test waits out its timer on every run and fails on a loaded machine. 3
B-4 Response-time tests run nightly, not in the record. PriceEmailResponseTimeIT, EmailTemplateResponseTimeIT, PipedriveDryRunTimingIT and any test that asserts elapsed time carry a tag (@Tag('performance')); a local build excludes the tag, and a build with -Pet.fullSuite=true (the nightly job, N-3) or in CI (CI=true: the Slow lane, which runs on demand only since N-5) includes it. The ACs they prove stay proved, once a night on the same machine, which gives comparable numbers. Their results depend on the machine’s load, not on the change (E-4, BUG-130). 1 (Q-8, decided 2026-10-27)
B-5 Fewer distinct Spring contexts. Measure first: one run with logging.level.org.springframework.test.context.cache=DEBUG gives the number of contexts loaded and the cache’s hits and misses. Then group the property variants that differ only in a value no other spec reads into shared base specs, replace @DirtiesContext where a reset of the data is enough, and move ApplicationContextRunner specs that start a whole application onto a slice. Each context costs seconds to start and its own pool and heap (BUG-088, BUG-247). 3
B-6 No coverage agent in the local record. The test task runs JaCoCo only under the same condition as B-4 (-Pet.fullSuite=true, or CI=true), so the nightly job (N-3) and the CI Slow lane still produce the report, and the Phase 1 → Phase 2 threshold (FS-4 FR-J3) uses the nightly figure. Once FR-J3’s thresholds bind, a coverage drop is then found after merge, by the nightly run, rather than before it; FR-J3 is revised to say so. Coverage is report-only through Phase 1; the agent slows every local test run (E-4). 1 (Q-8, decided 2026-10-27; owed.md has the coverage-measuring question for the record)
B-7 Two test JVMs where the machine has the memory, after B-2 and B-3: maxParallelForks read from -Pet.testForks (default 1). Each JVM starts its own Postgres and LocalStack through the existing static containers. Adopted as the default only if a measured run on the Windows machine and on an 8-core Mac is faster with no new failures. (2026-11-12: built in forge-et-be#411, default 1. The nightly full run passes -Pet.testForks=2: on the lead’s Mac (8 cores, 16 GB, mains power) the whole -Pet.fullSuite=true build took 31m 23s with 0 failures, against 43m 8s in one JVM the night before (forge-et-be#412), and at least 34% of memory stayed free. An earlier try the same evening, at a load average of 21 from other sessions’ builds, lost one JVM at start-up — Testcontainers could not connect to its Ryuk container, so that JVM’s 289 specs failed with initializationError — and a heavily loaded night can do the same. GitHub’s 2-processor, 7.9 GB runner cannot hold two JVMs, so CI keeps one. Developer machines keep the default of one.) Much of the suite’s time is waiting, which a second JVM overlaps; the cost is a second 3 GB heap and a second set of containers. 3
B-8 A quick unit-only task. The container-backed base specs (SharedPostgresSpec, DatasourceContextSpec, MigratedSchemaSpec, the messaging and mailbox context specs) and every @SpringBootTest spec carry an integration tag; the task’s first PR proves with a spec of its own that a subclass inherits it. unitTest excludes the tag and runs with Spock’s parallel execution on, through its own configuration file (systemProperty 'spock.configuration', 'SpockConfigUnit.groovy' on that task only, holding runner { parallel { enabled true } }); integrationTest includes the tag and runs as today; check runs both. Developers run unitTest in the development loop. Most specs are fast, and today they wait behind the container-backed ones in the same task. 3
B-9 The slowest specs of E-4 are each looked at once, starting with ScanResultQueueIT, MailboxIngestFailurePathsIT, PricingQueryPlanIT, LeadResearchEnqueueIT, EtPropertiesSpec and ActuatorSpec. Each is changed only where the change keeps what the spec proves. The top 30 classes hold a large share of the time; one look each is cheap. 3
B-10 Not adopted now: Testcontainers reuse between runs (withReuse(true)). It saves the container starts but keeps data between runs, and leftover containers are what docker-resource-labels.md exists to clean up. Revisit after B-1 and B-5 are measured. — —

Track V — Faster Vitest suites

# Decision Why Phase
V-1 Two Vitest projects per frontend: node and dom. Files that do not touch the DOM run in the node environment; the rest stay in jsdom. The split is by a rule the config can read (for example *.test.ts under src/lib/ and src/test/scripts/ → node, *.test.tsx → dom), plus the existing @vitest-environment comment for exceptions. 43% of the summed time is environment start, imports and setup (E-5); a node file skips jsdom entirely. 1
V-2 The tooling tests run only when the tooling changes, and nightly. A third project, scripts, holds src/test/scripts/; the test record runs it only when scripts/ or src/test/scripts/ changed since the merge base (S-1), and the nightly full run (N-3) always runs it. They test the harness’s gate scripts, not the product; they are slow and a source of Windows-only bugs (E-5). 2 (decided in Q-1, since the record no longer runs every test; after N-3’s two weeks, S-4)
V-3 The long files are split, and their slow interactions shortened — the four files over 20 s first. AccountListScreen.test.tsx and the others are split by behaviour into files that run in parallel. Within them, typing that is not what the test is about uses user.paste or fireEvent, and lists ticked one box at a time (BUG-220’s 51 contacts) use a select-all control or fewer items when the count is not what the test proves. sharedDrift.test.ts in forge-et-pwa builds its two git repositories once per file, not once per case. One 72-second file bounds the run on a 16-core machine; BUG-213, BUG-214 and BUG-220 are slow interactions that time out under load; the forge-et-fe Slow lane is near its cap (E-2). 1 for the four longest files and sharedDrift; 3 for the rest
V-4 Half the cores by default. maxWorkers: '50%' in both frontends’ vitest.config.ts; VITEST_MAX_WORKERS still overrides it, and is documented (the owed.md row “Document VITEST_MAX_WORKERS”). Not adopted (2026-10-27): measured first, as this row asked (§ Measurements), and the run queue (M-5) removes the contention between heavy runs it was for; VITEST_MAX_WORKERS is documented as the per-machine override, a whole number. The record runs at a 0.98 busy share with cores-minus-one workers (E-5); half the cores leaves room for the developer’s other work at a small cost in time, measured before adoption. 1
V-5 Not adopted: happy-dom in place of jsdom (faster, but behaves differently in places the tests rely on), and isolate: false (shared module state between files). Either may be measured for the node project only, later. — —

Track P — A cheaper browser gate

# Decision Why Phase
P-1 Four sizes on a PR, seven nightly and at the wave PR. A PR’s size matrix runs one size per layout band: size-320x640-chromium (reflow), size-768x1024-webkit (tablet), size-1024x768-chromium (laptop, the narrowest laptop layout, where overflow is likeliest) and size-1280x800-chromium (desktop). The nightly run (N-3) and the wave PR’s gate run all seven. The PRD’s supported sizes are unchanged: every size is still checked, every night and before every wave merges. 1440 repeats the 1280 desktop layout, and the Chromium tablet and the WebKit laptop sizes repeat a band the PR set already covers in one engine; four sizes cut the PR size matrix by about 43% before any selection (S-2). F40 NFR-1 itself offers “the 1440 Chromium size” as an example of what to cut. 1 (Q-3, decided 2026-10-26; an F40 spec revision)
P-2 Axe once per layout band, in the nightly and wave-PR runs. The CRM’s markup changes with the viewport (components that render differently through useMediaQuery or matchMedia, for example TagSettingsTable, LoginRoute, LeadTagsSection, AccountDetailHeader and ComposeEmailDialog), so axe is not run at one size only: it runs at P-1’s four sizes, and the other three projects (size-768x1024-chromium, size-1024x768-webkit, size-1440x900-chromium) run the geometry checks only — overflow, target size, frame height and nothing focusable inside aria-hidden. In a Windows PR profile (P-9), size-768x1024-chromium stands in for the WebKit tablet project and runs axe, so the tablet band is still scanned. Axe is the slowest part of the page check; the three projects it leaves each repeat a band already scanned (E-3). About 3,800 axe scans a full run become about 2,200; a PR run already scans only P-1’s four sizes. Narrowed when built (2026-10-27): axe findings can differ by engine at one size — BUG-271’s axe:target-size is marked at 768 Chromium only — so a full run leaves the scan out only at size-1440x900-chromium, which repeats 1280 in the same engine, and keeps it at 768 and 1024 in both engines. A Vitest test (forge-et-fe src/lib/config/sizeMatrixAxe.test.ts) pins which projects scan and fails an axe mark on a project that does not. 1 (Q-3, decided 2026-10-26)
P-3 Test a production-like build instead of the dev server — a spike first. The stub sign-in is removed from a build by import.meta.env.DEV (each call site is written import.meta.env.DEV && …, which a production build replaces with false; scripts/check-bundle-stubs.mjs), so a vite build --mode e2e drops it too. The spike therefore chooses between (a) changing the removal condition at every call site to one a test-mode build keeps — a change to AD #28’s removal mechanism, needing the lead — and (b) a development build (NODE_ENV=development vite build) served by vite preview, which keeps DEV but is no longer a production build. Either way the spike shows that check-bundle-stubs.mjs still fails a production bundle that contains a stub. Adopted only if the full gate is faster and passes with no new failures on both operating systems. Pages load from one bundle instead of hundreds of transformed modules, removing the BUG-079, BUG-196 and BUG-222 class. Cost: a build of about a minute per run, and with (a) a change to a security control. 3
P-4 The per-feature accessibility and layout specs are audited against the size matrix. For each of the 51 files that also run axe or the page check, a test that checks the same page state as a size-matrix entry is removed, and one that checks something the matrix cannot (a keyboard path, a focus order, a live region, a behaviour across steps) stays. The rule for new tests is written into test-strategy.md: a check of a page in a state is a pageStates.ts entry; a feature spec tests behaviour. BUG-246 failed four tests in four files for one defect (E-3). 3
P-5 The seam specs are audited against the backend’s integration tests. A seam spec proves the contract — the response shape, status codes, the CSRF handshake, a multi-step flow across the seam; a business rule the backend’s own spec already proves is removed from the seam set. 639 seam tests run against the real backend on every gate; some may repeat what forge-et-be proves. 3
P-6 Long spec files spread across workers. Either fullyParallel: true in playwright.config.ts, with every file whose tests share state set to mode: 'serial' (the 23 that already are, and any others the first full run shows), or mode: 'parallel' on the longest files one at a time, starting with users/journey.spec.ts. The 23 serial files are also reviewed, and those whose tests share no state lose serial, so one failure no longer skips the rest of the file. Every file except the size matrix runs its tests in order in one worker today, so a long file sets a floor on the run’s length (E-3). 3
P-7 A fresh, pre-migrated database for each gate run: the gate’s backend starts from a Postgres template database made once after migration (CREATE DATABASE … TEMPLATE), so a run never inherits the rows of earlier runs. BUG-173 (a grown local database slowed list reads until the gate refused to run) and the cards that BUG-233 found arriving from other tests’ data. 3
P-8 A change to unit tests alone needs no new browser gate. gate-inputs.json excludes **/*.test.ts and **/*.test.tsx (as built: the gate-input grammar cannot name src/test/**, so the helpers under src/test/ that are not test files stay gate inputs), and a check in pnpm check:boundaries fails when anything under e2e/ or playwright.config.ts imports one of them. F40 FR-8 is revised to match. No browser test loads a unit-test file (E-3), and deferred review fixes are often of exactly that kind (E-6). 1 (Q-3, decided 2026-10-26; an F40 spec revision)
P-9 In forge-et-fe, an ordinary PR’s browser gate on Windows runs the Chromium projects only; WebKit runs on macOS. On a Windows machine the CRM’s gate leaves out the WebKit projects for a PR that is not a wave PR or into release/*, runs size-768x1024-chromium in place of P-1’s size-768x1024-webkit so the tablet band is still checked, and the Browser gate: line says so (· chromium only). A wave PR’s gate and the nightly run (N-3) run every project on any system, and a PR from a Mac runs WebKit as today. forge-et-pwa keeps WebKit on every PR: the capture app is phone first, and the PRD names Safari on iPhone. This revises a PRD commitment: project-prd.md § Accessibility says both apps “are tested in Chromium and WebKit … and the accessibility and responsive browser checks gate every pull request”, so the PRD, .claude/CLAUDE.md § Quality Gates › Browser QA, F40 FR-7 and the hook and CI check that read the line all change with it (§ What changes elsewhere). Safari runs only on Apple systems, so Playwright’s WebKit build for Windows is an engine no user runs, and it is where Windows-only browser failures sit: BUG-131 (Tab never reaches a link, a Windows WebKit behaviour) and three of the four flaky bugs in progress, all Windows (E-10). The trade-off: BUG-261’s product defect was first seen as a Windows WebKit failure, because a slow engine exposes a race sooner; the wave PR’s gate and the nightly still run WebKit, so such a race is found there, a day or two later. 1 (Q-11, decided 2026-10-27 — a PRD and F40 spec revision)
P-10 A change to the local test record’s scripts alone needs no browser run. In forge-et-fe and forge-et-pwa, gate-inputs.json excludes **/local-test-evidence.*, **/fast-lane-scope.*, **/test-selected.* and **/vitest-run-facts.* (the grammar cannot name one file under scripts/, so each entry matches the name at any depth). A pnpm check:boundaries rule, gate-loads-no-record-script, reads the list’s **/<name>.* entries and fails when anything the gate loads imports a file with one of those names; the config stops on any other exclude shape it could not guard, and a test checks that every tracked file such an entry matches sits directly under scripts/ (a spec Playwright finds by glob is never seen by an import rule). F40 FR-8 is revised to match (Rev 11). It saves a run in forge-et-fe only: in forge-et-pwa the record scripts are copies of forge-et-fe’s, and every re-copy also changes src/ui/SHARED-WITH-FE.sha256, a gate input the gate reads, so such a change still selects the capture app’s whole gate (the list is a shared copy and keeps the entries). A capture-app PR that only re-copies record scripts records Selected browser run: skipped — <reason> rather than teaching the selector the hash list (the lead, 2026-11-18). The record, the Fast lane’s documentation-only decision, the selected Vitest run and the record’s Vitest reporter use these scripts; no browser test loads them. All of scripts/ was a gate input, so forge-et-fe#567 (W-5), which changed only local-test-evidence.mjs, its declarations and its Vitest test, selected the full CRM gate, about an hour; it recorded a skip, as did its copy forge-et-pwa#157. The nightly full run still runs every browser test. 1 (the lead’s decision, 2026-11-18)

Track F — Flaky tests

# Decision Why Phase
F-1 One automatic retry in the browser gate, and a test that passes on retry is reported as flaky, not passed. The gate passes --retries=1; Playwright’s JSON report marks such a test flaky. The result records each flaky test; the Browser gate: line gains · N flaky; the run fails if more than 5 tests are flaky (Q-2). At a machine’s full load, 0.5–0.7% of tests time out somewhere (E-6), so a full run almost never passes in one attempt and the developer ends at a skip, which records less than a retry would. The trade-off: a retry can let a real race pass, which “zero retries locally” exists to prevent (BUG-261 and F11-f’s finding F-001 were product defects first seen as flaky tests). F-2’s weekly report and its threshold for filing a bug are what keep such a race visible, so F-1 is adopted only together with F-2. 1 (Q-2, decided 2026-10-27 — it changes “zero retries locally”)
F-2 Flaky tests are counted before a bug is filed — from records already kept, not a new committed file. The gate’s result.json, the record comments (F-3) and N-3’s nightly results list every flaky test with the run, operating system and recorded load. A harness script, .claude/lib/flaky-report.py, reads them through gh api (the PRs’ Browser gate: lines and record comments, and the nightly log, N-4) and lists each test’s occurrences over a window. A test that is flaky twice in 14 days, or once on an idle machine, gets a BUG-NNN through the bug flow, as today; a single occurrence under load stays in the report. The report also matches each test against the bugs already filed, closed ones included, by the spec path and test title in each bug’s Affects line: a test whose bug was closed wontfix as an overload outcome and that fails again is listed against that bug, and the report is searched before a new bug is filed. The report is run and read once a week by the lead (Q-2). Each flaky bug today costs a full investigation, including for one-off failures under load (E-6). BUG-195, closed as overload, came back ten days later as BUG-259, and BUG-242 and BUG-244 were the same defect filed twice on one day (E-10). A ledger file appended by every branch would conflict on every merge (L-049), and CI has no harness checkout to write to. 1 (Q-2, decided 2026-10-27)
F-3 The same for the other runners: Vitest’s retry: 1 in the test record only, with retried tests listed in the record comment; Gradle’s org.gradle.test-retry plugin in the build record with failOnPassedAfterRetry = false and the retried tests listed. Both are read by F-2’s report, and the same limit of 5 applies. One rule for all three suites. 1 (Q-2, decided 2026-10-27; the Gradle plugin is a new dependency, approved with Q-2)
F-4 Unchanged: a sleep or a longer timeout is not a fix; no quarantining; a flaky test that cannot be fixed is deleted with the reason recorded; the triage order of load-check-and-flaky-tests.md D-4 applies to every bug F-2 files. — —
F-5 The causes the fixed flaky bugs share become review checks, where none exists yet. Added to frontend-reviewer: a Vitest test asserts no bound on elapsed time (L-125, as backend-reviewer already checks for the backend); a test that reads a list reads only the rows it made, never every page of a list that each run adds to; a component test case keeps its interactions few, moving a one-by-one walk to a hook or unit test (BUG-128’s fix). Added to .forge/test-strategy.md § Flaky Tests as named first suspects: more work in one case than its time allows, and a list that grows with every run. The race causes are already checked (frontend-reviewer dimension 14, L-101) and stay as they are. E-10: after the races, these are the causes of the fixed flaky bugs, and five of the six open Vitest bugs have BUG-128’s shape (the sixth, BUG-215, is git-fixture time in a tooling test). § Test Data & Isolation’s rule against absolute row counts is the nearest existing rule. 1

Track M — The machines

# Decision Why Phase
M-1 Windows machines are set up for builds, and measured before and after. Either (a) the repositories, the pnpm store, ~/.gradle and Docker’s data on a Windows 11 Dev Drive, whose Defender “performance mode” scans these files without blocking each open; or (b) Defender exclusions for the same folders and the java, node and docker processes. Docker Desktop’s WSL 2 VM is given a memory and processor cap in .wslconfig that leaves room for the editor and other sessions. The before and after build records are compared on the same commit. Windows takes about twice as long as macOS on twice the cores (E-1), and BUG-220’s record names Defender among the busiest processes. 1 (Q-6 for the choice and any security approval)
M-2 A run keeps the machine awake. The gate and both records hold a wake lock while they run: caffeinate -dims -w <pid> on macOS, SetThreadExecutionState through PowerShell on Windows. Runs have failed because the laptop slept (BUG-226; forge-et-fe#312’s skip). 1
M-3 Half the cores by default for every runner. The browser gate already uses Playwright’s default of half the cores (ET_E2E_WORKERS overrides it); Vitest follows with V-4; the backend keeps one test JVM by default (B-7). Leaves room for parallel work (Goal 5). 1 (no change beyond V-4)
M-4 The Windows load reading is checked. One Windows record reads “busy 1.00 at start, 1.00 peak, 0.17 average”, an average below both its start and its peak, which the sampling should not produce. The Get-Counter sampling in the load module is compared with Task Manager on the Windows machine. The load check (load-check-and-flaky-tests.md) depends on the reading. 1
M-5 One heavy run at a time per machine, by waiting, not refusing. The gate and both records take a lock file in the user’s temp folder (holding the PID, the command and the start time). A second heavy run started on the same machine waits, printing which run holds the lock, and starts when the lock is released; a lock whose PID is no longer running is removed. --no-wait refuses instead. Parallel sessions on one machine are the normal case (E-6); two heavy runs together make both fail. This reverses a rejected alternative of load-check-and-flaky-tests.md, whose accepted gap (“two sessions that start a gate or build within seconds of each other … overload the machine together”) is what the records now show; that design gains a revision pointer to this one. The nightly job (N-3) takes the same lock, so it waits for a run a developer left going. The lock file is et-heavy-run.lock in the user’s temp folder (Node’s os.tmpdir(), Python’s tempfile.gettempdir()), holding the holder’s PID; a run started by a lock holder sees ET_HEAVY_RUN_LOCK_HOLDER set to that PID and runs without waiting, since its parent holds the lock — only when the lock file names that same PID and that process is alive, so a stray exported variable never skips the lock. 1 (Q-7, decided 2026-10-27)
M-6 Optional: heavy runs at lower scheduling priority (nice -n 10; BELOW_NORMAL_PRIORITY_CLASS on Windows) behind --low-priority, so interactive work stays responsive. Not the default, because a starved test is a stall (load-check-and-flaky-tests.md § Terms). — 2
M-7 Dropped (Draft 5). It ran a PR’s record on N-3’s self-hosted runner; N-3 is a scheduled job on the lead’s Mac that runs only main (Q-4), so there is no runner to send a record to. — —
M-8 Tooling is checked on both systems before a developer meets it. Built: the reviewer path rule below. Not built: the Windows job (Q-10). As proposed, in each frontend’s nightly, a job on a GitHub-hosted windows-latest runner runs Vitest’s scripts project (V-2) and S-7’s tests that read files as text, only when one of those files or a script they test changed since the last nightly. Added to both reviewer agents: a script or test builds and compares paths with the platform’s path functions (path.join, path.relative, os.path), and starts a wrapper by its platform name (gradlew.bat on Windows). The backend’s scripts/ have no tests of their own (git -C forge-et-be ls-files 'scripts/*' on 2026-10-26), so there the reviewer rule is the check. 7 of the 17 Windows bugs that are not flaky were test or tooling code written for macOS, each found when a Windows developer’s run failed on it (E-10). The job would have caught two of those defects first (BUG-038; BUG-242 and BUG-244, one defect), and BUG-172 from the machine-limits group; the reviewer rule covers BUG-073’s kind; BUG-131, BUG-188 and BUG-245 are in the browser gate’s marks and need a Windows browser run, which Windows developers’ own gates remain. Cost: about 6 minutes a run (install and the two test sets), billed at twice the Linux rate, so about 12 minutes per frontend on a night the tooling changed and none on other nights. 1 for the reviewer rule; the Windows job is not built (Q-10: the lead stated no preference on 2026-10-27, so the job, which adds minutes, waits until the lead asks for it)

Track W — The delivery flow

# Decision Why Phase
W-1 A work item’s own runs are selected runs. /forge-test-verify and /forge-deliver Stage 7 run S-1 and S-3’s selection on a work item’s branch instead of the whole backend suite and the whole Vitest suite; the wave PR’s record stays a full run and remains the integration build that Stage 15 step 4 already reads. The browser runs of a type: verify work item are already scoped to the wave’s specs and stay as they are. A wave of four work items runs the whole backend suite five times today (§ Context); the wave PR’s full record is the one the merge depends on. No rule about what a PR must carry changes. 1
W-2 After a fix, re-run what failed first. --last-failed (Playwright) and vitest --run --changed re-run the failures and the code the fix touched; the full run is repeated once, at the end. Written into /forge-deliver Stage 7 and Stage 12. A full re-run after each one-line fix repeats the hour each time. 1
W-3 A record carried forward over a small change (Q-1). After a recorded commit, a later push may be recorded with a selected run over the change since the recorded commit (S-1 to S-3), when that change touches no shared code; the comment names both commits. The gate’s covers rule (F40 FR-8, FR-10, FR-11) gains the same case: a passing result on an ancestor plus a selected gate run over the change covers the head. This extends the existing carry-forward for documentation-only changes. Today any push after the record, however small, needs a full new record and gate (E-6’s deferred Nits). 2
W-4 The wave’s full gate runs once, at Stage 12, and its fixes are re-checked with W-2 then one final full run. Stage 7’s browser runs stay scoped to the wave’s own specs and size-matrix states. When W-1 lands, forge-deliver.md is read for any step that runs the whole browser suite earlier, and such a step is narrowed in the same PR. A full gate costs most of an hour on a Mac and hours on Windows (E-1); one per wave is the most the flow needs. 1
W-5 A record carried across a merge of main. On a PR into main, the build record and both test records (local-build-evidence.py, local-test-evidence.mjs) set the check without running when the head is reached from a recorded commit only through merges of main whose own paths touch no test or build input, and commits that change only such documentation. A merge’s own paths are the paths git’s automatic merge of its parents left in conflict and the paths where the merge commit differs from that automatic merge (git merge-tree --write-tree, git 2.38 or later). The conflicted paths count even when the merge kept what git left there, because a modify/delete, rename/rename or binary conflict leaves no markers (git show --remerge-diff alone misses those). The merge’s second parent must be on origin/main. In forge-et-be, a merge that brings a Flyway migration into a branch that adds or changes one of its own is built, because two migrations can share a version and merge cleanly. The walk follows first parents for at most 50 commits, and any git error builds. The check names the recorded commit and says main was merged in. A resolution in code, or any other code edit in the merge, runs the record as before. Never on a PR into another base, never on release/*, and never with --full. Without the resolutions, the merged code is what GitHub’s own merge of the PR makes, and § Branching accepts that merge without a new record; the record diffed from the recorded commit, so the base branch’s own changes counted as the PR’s and a docs-only conflict cost a full run. forge-et-be#457 (2026-11-17) adds a migration, so its record runs every test (about 100 minutes). Its record on 07e40690 passed; merging main to resolve a conflict in docs/api/README.md alone (44af099e) then needed a second full run, and the first attempt at it was stopped before it finished. The lead’s decision of 2026-11-18. 2

Track H — Keeping the suites lean

# Decision Why Phase
H-1 Every record lists its 10 slowest tests in its PR comment (from the JUnit XML or Vitest’s JSON), and the gate’s result its 10 slowest. What is not shown is not looked at. 1
H-2 A slow-test list, not a gate. A test over 5 s (backend, Vitest) or 15 s (browser) appears in N-4’s nightly summary; a new one is explained in the PR that adds it. Keeps growth visible without failing anyone’s PR. 1
H-3 A spec that adds page states says which sizes each needs (default: P-1’s four), and a plan that adds more than a handful states the expected run-time cost. Added to spec-reviewer and plan-reviewer checks. The size matrix multiplies every new state by seven (E-3). 2
H-4 F40 NFR-1 is restated to the targets of § Goals, and forge-et-fe e2e/README.md § Run time records the measured figures after each phase, closing the owed.md row “Correct the browser gate’s run time”. NFR-1 is broken today and says so (E-2). 1 (Q-3, decided 2026-10-26)
H-5 Every speed fix ships with its rule for new tests, checked at review. Added to backend-reviewer: a new queue consumer’s business rules are tested by calling its handler, with at most the per-queue specs of B-2 going through LocalStack; a bound on elapsed time uses an injected clock (L-125); a new Spring property variant or @DirtiesContext says why a shared base spec cannot serve; a response-time assertion carries B-4’s tag. Added to frontend-reviewer: a check of a page in a state is a pageStates.ts entry, not a per-feature a11y or layout spec (P-4); a test file that does not touch the DOM is in the node project (V-1); a new spec file whose tests share no state is not serial (P-6). A new test over H-2’s limits is explained in its PR. Each rule is added in the PR that makes the matching cut. A cut without a rule is undone by the next features (§ Long-run assessment). 1 for the rules whose cuts are in Phase 1; with each later cut
H-6 The trend is read every week. The lead (Q-2) reads N-4’s nightly log and F-2’s report once a week and raises anything growing out of line (a suite’s time rising faster than its tests, a nightly run that no longer finishes before the working day, a test flaky twice) as a bug or an owed.md row. A trend nobody reads is the six cancelled nights again (E-7). 1

Rollout

Phase Contents Gate to the next phase
1 — the nightly full run covers every test, so a PR runs less (the lead’s answers of 2026-10-26 and 2026-10-27 moved the decided Phase 2 items here) First, the PR-gate changes: P-1, P-8 and P-9 (the PR profile), F-1 to F-3 with F-2’s report, M-5 (the run queue), M-2. Until N-3’s first run, the wave PR’s gate (full profile) is the only run of the three sizes and the WebKit projects a PR profile leaves out, as it was each wave’s integration check before. P-2 waits for N-3, since it changes only full-profile runs. Then the nightly run: N-3 with N-1, N-2 and N-4; when N-3 has completed one full run of all three repos and reported, CI’s scheduled Slow lane is switched off (workflow PRs, approved by the lead, Q-9; the scheduled Fast lane and its Trivy scan stay). Then the suite cuts: B-1, B-4, B-6, V-1, V-3 (the four longest files and sharedDrift), V-4. Alongside: S-1 and S-3 for the development loop and work items, S-5, S-6, S-7; W-1, W-2, W-4; H-1, H-2, H-4, H-6, and H-5’s rules for these cuts; F-5, M-8’s reviewer rule; M-1 and M-4 on the Windows machines (Q-6). Measured: the build record, test record and gate on one commit before and after, on macOS and on Windows. N-3 has had seven green or explained nights in a row (the lead, 2026-10-29, revising two weeks); the measurements are in this document’s ## Measurements
2 — selected runs on PRs S-1 and S-3 for the PR records, and no browser gate on a PR into main (S-4 as revised on 2026-11-02; switched on that day without waiting for seven nights), W-3, V-2, H-3, each with its H-5 rule; M-6 The rules that change are rewritten in the same PRs (§ What changes elsewhere)
3 — engineering Only where Phase 1 and 2’s measurements show the need: B-2, B-3, B-5, B-7, B-8, B-9; V-3 (the rest); P-3 (spike first), P-4, P-5, P-6, P-7 (beyond the fresh nightly database N-3 already gives) Each item is measured and recorded when it lands; an item whose measured saving does not justify its work is dropped here with the figures

Rejected alternatives

  • Remove tests to hit a number. It removes coverage without knowing which. Tests are removed only where an audit shows a duplicate (P-4, P-5) or the check moves to a cheaper place (B-2, B-4).
  • Turn the suite off and pick tests by hand for each bug or feature. A hand-picked list depends on its author knowing what the change can break; S-1 to S-3 choose by a rule that can be reviewed, with a full run whenever shared code changes and every night (S-6).
  • Run the full browser suite nightly on GitHub-hosted runners. A run that takes about 50 minutes on an 8-core Mac would take hours on a standard hosted runner, every night, from the organisation’s one Actions budget; sharding it spends the same minutes. The nightly job on the lead’s Mac (N-3) costs no minutes.
  • A GitHub self-hosted runner for the nightly run (Draft 4’s N-3). No spare machine is available (Q-4, 2026-10-26), and a runner on a daily laptop would execute any workflow pushed to a branch; N-3’s scheduled job executes only main.
  • A small cloud machine for the nightly run. Offered on 2026-10-27; the lead chose the Mac job, which costs nothing.
  • Run the Windows developers’ gates and records inside WSL 2. Likely faster than native Windows, but those runs would prove Linux browsers rather than Windows Chromium; the lead declined it on 2026-10-27, and P-9, P-1 and selected gates cut the Windows gate instead.
  • Drop the Test record. Every merge to main deploys to dev, and the record is the only test run before a merge; the lead kept it, made cheaper by selection (S-4, W-3), on 2026-10-27.
  • ego(lite), an AI-agent browser (suggested 2026-10-27). It is not a test runner — its own comparison with Playwright says to keep Playwright for deterministic test suites — it drives Chromium only, its Windows build is a beta, and it shares the user’s whole Chrome login profile with the agent. It would not shorten the gate; the built-in browser pane and Claude in Chrome already cover an agent’s ad-hoc checks.
  • Raise the timeouts. It turns a race into a slower race (test-strategy.md § Flaky Tests).
  • Quarantine flaky tests. A quarantined test is a silent hole (coding-standards.md §7.8); F-1 and F-2 keep every flaky test running and counted.
  • A committed flaky-test ledger appended by every branch. It conflicts on every merge and cannot be written from CI (F-2).
  • A paid test-selection service (for example Gradle Develocity’s predictive test selection). A new running cost for something the package layout allows us to do with a short script.
  • Swap jsdom for happy-dom, or turn off Vitest’s isolation, across the suite (V-5).

Open questions

Every answer below is the lead’s (Tharinda Rajapaksha), given in the working session of 2026-10-26 and 2026-10-27, except Q-6, which is still open.

# Question Recommendation Owner Answer
Q-1 May a PR that touches no shared code carry selected runs for its test record, build record and browser gate (S-4, W-3), with the nightly run as the full run behind it? Yes, once N-1 has run for two weeks and N-3 exists; full runs stay on shared code, wave PRs and release/* Tharinda (lead) Yes, split by suite (2026-10-27). Selected build and test records on any PR that touches no shared code; a selected browser gate only on a PR that touches one feature and no shared code; full runs on shared code, wave PRs and release/*. They start once N-3 has run for two weeks (S-4). Revised 2026-10-29: built now, switched on after seven green or explained nights in a row, about 5 Nov (S-4).
Q-2 May the browser gate and the records retry a failed test once, reporting it as flaky, with F-2’s report in place of a bug for a single occurrence under load (F-1 to F-3)? What flaky count fails a run? Who reads F-2’s report every week (F-1 is safe only with that owner, H-6)? Yes; fail the run above 5 flaky tests; the lead names the owner with the answer Tharinda (lead) Yes (2026-10-27). One retry; more than 5 flaky tests fail a run; the lead reads the weekly report (F-2, H-6).
Q-3 May a PR’s size matrix run four sizes, one per layout band, with all seven nightly and at the wave PR, axe skipped on the three repeated projects (P-1, P-2), unit-test files leave the gate inputs (P-8), and F40 NFR-1 be restated (H-4)? Yes Tharinda (lead) Yes, all four parts (2026-10-26). P-2 was built narrower than answered, scanning more, not less: axe is left out only at 1440 Chromium (P-2’s row, 2026-10-27).
Q-4 Which always-on machine runs the nightly full run (N-3), and who keeps it running? Draft 4: a spare machine (a Mac if one can be found) with a self-hosted runner; the trade-off of a Mac is in N-3 Tharinda (lead) No spare machine (2026-10-26); the lead’s Mac, by a scheduled job at 02:00 Sri Lanka time, running all three app repos’ suites in full (2026-10-27). CI’s scheduled Slow lane is switched off once the job has run; the scheduled Fast lane and its Trivy scan stay (N-3, AD #38). The lead keeps the Mac on and awake overnight.
Q-5 Should N-1 also post to the team’s chat, which needs a webhook secret per repo? Not at first; the assigned issue notifies everyone Tharinda (lead) No preference stated (2026-10-27); not built, as recommended.
Q-6 Dev Drive or Defender exclusions on the Windows machines (M-1), and does either need Altrium’s security approval? Dev Drive first: it keeps scanning on Nanndish, with whoever approves machine security Open. Nanndish’s measured causes and proposed fixes, 2026-11-02, are in the revision line of that date; the recommendation stands, with the before-and-after record M-1 asks for.
Q-7 May heavy runs wait for each other on one machine (M-5), reversing the rejected lock of load-check-and-flaky-tests.md? Yes Tharinda (lead) Yes (2026-10-27).
Q-8 May response-time ACs be proved by a nightly tagged task instead of the record (B-4), and coverage come from the nightly run only (B-6), revising FS-4 FR-J3? Yes Tharinda (lead) Yes, both (2026-10-27).
Q-9 Approval of the workflow edits in each app repo’s pr-validation.yml Approve at each PR Tharinda (lead) The one workflow edit left, skipping the Slow lane on the scheduled run, may be merged once N-3 has completed one full run and reported (2026-10-27).
Q-10 May each frontend’s nightly run the tooling tests on a Windows runner when the tooling changed (M-8), at twice the Linux minute rate? Yes: a few minutes on the nights the tooling changed Tharinda (lead) No preference stated (2026-10-27); the job is not built, since it adds minutes; M-8’s reviewer rule is built.
Q-11 May an ordinary forge-et-fe PR’s browser gate on Windows leave out WebKit (P-9), with WebKit kept in the wave PR’s gate, the nightly and every Mac’s gate? This revises the PRD’s “tested in Chromium and WebKit … gate every pull request” (project-prd.md § Accessibility, through .claude/rules/prd.md). Yes, once N-3 runs WebKit nightly; until then, only with the wave PR’s gate as the WebKit run Tharinda (lead) Yes (2026-10-27). N-3 runs WebKit every night; until its first run, the wave PR’s gate (full profile) is the WebKit run (§ Rollout).

Where this stands, and the next steps

As of 2026-10-27: Draft 5 records the lead’s answers (§ Open questions) and reorders the rollout around them, and Phase 1 is built apart from its Windows items. The owed.md rows “Start Phase 2 of the test-run cost design” (the lead) and “Run the test-run tooling on Windows” (Nanndish) hold what is left. The dated account is .forge/engagement-log.md, 2026-10-26 and 2026-10-27.

Built on 2026-11-18: W-5 — forge-et-be#463, forge-et-fe#567, forge-et-pwa#157 (the record scripts) and forge-et-harness#1470 (the rules). The records also carry over an empty commit on a recorded one. P-10 — forge-et-fe#575 and forge-et-pwa#163 (the gate-input list and the boundary rule) and forge-et-harness#1479 (this design, F40 Rev 11).

Built and merged on 2026-11-03: N-5 — forge-et-be#352, forge-et-fe#387, forge-et-pwa#107 (the workflows and record scripts) and forge-et-harness#1092 and #1099 (the nightly run’s added checks, the rules, the missed-night step).

Built and merged on 2026-10-27:

  • The PR gate profile — P-1, P-8, P-9 and F-1 (with F-3 still to come for the records): forge-et-fe#346, forge-et-pwa#97, and the hook and rules in forge-et-harness#978.
  • The nightly full run — N-1 to N-4: forge-et-harness#979, with the review fixes it carries.
  • The backend cuts — B-1, B-4 and B-6: forge-et-be#317.
  • F40 NFR-1 restated (H-4): forge-et-harness#978, with the measured figures in forge-et-fe#346’s e2e/README.md § Run time.

Merged with this document’s rule changes, after the first full nightly run: the workflow PRs that skip CI’s scheduled Slow lane (forge-et-fe#347, forge-et-be#318, forge-et-pwa#98).

Built on 2026-10-27 and merged on 2026-10-28, the rest of Phase 1 (forge-et-fe#350, forge-et-pwa#99, forge-et-be#322, forge-et-harness#995):

  • The run queue and the wake lock (M-5, M-2): the gate and both records take et-heavy-run.lock and wait for each other and for the nightly run; they keep the machine awake while they hold it.
  • The counted retry in both records (F-3), the flaky names published (the gate’s PR comment on --push, the record comments, the nightly log) and the weekly flaky report (F-2, .claude/lib/flaky-report.py).
  • The slowest tests in every record and gate result (H-1), and the nightly log’s slow-test counts and lists (H-2).
  • Selected runs for the development loop and a work item’s own runs (S-1, S-3, S-5, S-6, S-7, W-1): pnpm test:selected and python3 scripts/select-tests.py, each with its repo’s scripts/shared-code.json. S-3 follows production code only: following every reference, test code included, reached all 20 feature packages from any change, because test fixtures name many packages; a test file is chosen by the packages it names, or by the class of a changed test file it names. In place of S-3’s map from api specs to packages, every api spec runs when a reached package serves HTTP, since an api integration spec can call its controller without naming its package and most changes select most of them anyway. The backend’s filter goes to Gradle in an init script, because some 190 --tests arguments passed Windows’ 8,191-character command limit. The records still run every test until Phase 2 (S-4).
  • The Vitest cuts: V-1 in forge-et-fe (two projects; the 65 *.test.ts files that need the DOM say so); V-3’s sharedDrift in forge-et-pwa (one git cat-file --batch read, a fixture built once per file, the per-file cases in files of their own). V-3’s four longest forge-et-fe files are not split: the longest takes 38 s of a 168 s run that starts the slowest files first, so a split would not shorten it. V-1 is not applied in forge-et-pwa, whose test files are nearly all script tests already in Node.
  • P-2, narrowed: a full run leaves the axe scan out at 1440 Chromium only, because axe findings can differ by engine (BUG-271), so the scans saved are one size in seven rather than three.
  • The delivery flow (W-1, W-2, W-4) and the reviewer rules (H-2, H-5 for the cuts made, F-5, M-8’s path rule).

Decided against on 2026-10-27, with the measurement: V-4 (half the cores by default). The run queue (M-5) removes the contention between heavy runs that V-4 was for; VITEST_MAX_WORKERS (a whole number — Vitest reads the variable with parseInt, so 50% means 50 workers) is documented in both frontends as the per-machine override, which closes its owed.md row. The measurement is in § Measurements.

Built and merged on 2026-10-30, the first of Phase 2 (forge-et-fe#361, forge-et-pwa#102; the lead’s decisions of 2026-10-29):

  • S-2, the selected browser gate: pnpm e2e:gate --selected, scripts/browser-gate/select.mjs and e2e/feature-map.json, with src/app/featureMap.test.ts keeping the map complete. The size matrix is narrowed by each page state’s route, which the map names, and the specs go to Playwright in a --test-list file. Host pages: src/app/ places one feature’s panel on another feature’s page, so a panel feature’s entry names that page’s specs, and a selected run also adds every spec that names one of the feature’s test ids (testIdMentions), leaving out helpers imported by more than 20 specs. A full run always passes ET_GATE_SELECTION=all. Until the switch-on, a selected line does not match the passed-line pattern, so it covers no commit, and --selected --push is refused. A live selected run of a one-line activities change ran 240 tests in 2 min 51 s, against about an hour for a full PR-profile gate.
  • The ad-hoc run’s lock (M-5): e2e/support/heavyRunLock.ts, the first Playwright global setup in both frontends, takes et-heavy-run.lock and releases it in its global teardown. A run still waiting already holds its dev-server port (5173), because Playwright starts its dev server before global setup; since 2026-11-04 the gates serve the app on ports of their own (forge-et-fe 15173, forge-et-pwa 15174; each repo’s src/lib/config/e2eAppPort.ts), so neither a waiting run nor a pnpm dev on 5173 stops a gate. The gate still gives its own port up to 15 seconds to free before it refuses. Under playwright test --ui or the VS Code extension the lock is held while the test server is open, and a reload in the same process counts its own lock as held.

Switched on on 2026-11-02, the lead’s decision that day (S-4 as revised; PRD Rev 81): no browser gate on a PR into main (each frontend’s Fast lane last step runs only on a PR into release/*, and the harness hook on gh pr create / gh pr ready is unregistered), and the build and test records run the tests a change needs, every test for shared code. The PRs that build it are named in the revision line of 2026-11-02. S-4©'s one-feature limit on a selected gate is no longer needed, since no PR into main carries a gate result.

Not built yet:

  • M-1 and M-4 on the Windows machines (Q-6, Nanndish), with a first Windows run of M-5 and M-2 (the owed.md row “Run the test-run tooling on Windows”). Nanndish’s Windows findings of 2026-11-02 are in the revision line of that date.
  • H-6, the weekly reading, which starts with the first weekly report.
  • The rest of Phase 2: W-3, V-2, H-3 and M-6.
  • Phase 3, each only where the measurements show the need: B-2, B-3, B-5, B-7, B-8, B-9, P-3 to P-7, and V-3 for the rest of the long files.

The question that started it — who hears about a failed nightly run? Nobody reliably. GitHub emails a scheduled run’s result only to the person who last edited its cron line (E-7). N-1 now opens an issue assigned to the team, raised by the nightly job itself (N-3).

To continue:

  1. Windows (Nanndish): Q-6 and M-1, then M-4, and a first run of the queue and the wake lock, with Windows before-and-after figures in § Measurements (AC-4).
  2. The weekly reading (the lead, H-6): python3 .claude/lib/flaky-report.py and the nightly-log issue, once a week.
  3. The rest of Phase 2: W-3, V-2, H-3 and M-6. The selected records and the browser gate’s removal from PRs into main were switched on on 2026-11-02.
  4. Keep N-3 reliable, since it is now the only full run: the lead’s Mac on, plugged in and open overnight, and a red or missed night fixed first (owed.md, “Keep the nightly full run reliable”).

What changes elsewhere

Each change lands in the PR that builds the decision it belongs to, with every pointer updated in that PR (.claude/CLAUDE.md § Compounding Engineering, item 5).

  • .forge/test-strategy.md: § When Tests Run (the rows that say “the whole browser matrix” — line 180 on 2026-10-25 — and the nightly row: selected and full runs, the nightly browser run), § Flaky Tests (F-1 to F-3, after Q-2; F-5’s two first suspects), § Browsers & Viewports (P-1, P-2, P-9), the page-state rule (P-4), a revision entry.
  • .claude/rules/git-conventions.md § The test suites run locally, and the PR records them: the selected record (Q-1), the carried-forward record (W-3), VITEST_MAX_WORKERS, the wait-for-lock behaviour (M-5), the nightly alert issue (N-1), and every sentence that places the nightly full run in CI’s Slow lane (§ Branching’s “caught after the merge, by each app repo’s nightly run on main”, § The test suites run locally’s description of the Slow lane schedule and “A failed nightly run on main is fixed before new work starts”): the nightly run is N-3’s, on the lead’s Mac.
  • Each app repo’s .github/workflows/pr-validation.yml: the Slow lane skips the schedule event (the scheduled Fast lane, with its Trivy scan, stays — AD #38), and the header comments that describe the nightly run point at N-3 (Q-4, Q-9). .forge/design/coding-standards.md §7.1’s description of the nightly Slow lane changes with it.
  • The rules that count a green nightly CI run as proof the suite ran, changed in the same harness PR as the workflow change, because a scheduled run then tests nothing: .claude/rules/release-and-branch-protection.md § Release Process (“A green nightly run on the same commit counts”), .claude/commands/forge-release.md step 4, .claude/commands/forge-deliver.md’s ship check (“a nightly Slow lane run counts”) and .claude/rules/tracker.md’s ship_check_resolved_by. Each says that only a manual dispatch (gh workflow run "PR validation" --ref main) or a release/* push counts.
  • .forge/design/architecture.md AD-O-11 (“no nightly”) and .forge/test-strategy.md § Deliberately Not Tested (“A nightly full E2E run — Closed in AD-O-11”): each gains a note that N-3 runs the full browser suite nightly off CI, which AD-O-11’s reasons (it duplicated a CI run and spent Actions minutes) do not apply to. .forge/test-strategy.md § When Tests Run’s nightly row and the hook guide’s tier table (.claude/hooks/README.md, “nightly (Slow lane)”) change with it.
  • Each frontend repo’s CLAUDE.md browser-gate rule and e2e/README.md (the scope, --full, the retry and the flaky count).
  • .claude/lib/nightly/ (new): the nightly job, its installer and its tests (N-3); .claude/CLAUDE.md § Reference Map and .forge/team-guide.md gain a line on where the nightly run happens and what to do with a nightly-red issue.
  • .claude/CLAUDE.md § Quality Gates › Browser QA: “a responsiveness and accessibility pass at every supported size … At each: … axe clean … Chromium and WebKit” changes for P-1, P-2, P-9 and S-2 (Q-1, Q-3, Q-11): an ordinary PR checks one size per layout band, on Windows in Chromium only, and every size and both engines are checked at the wave PR and every night.
  • .forge/design/coding-standards.md: §7.1 (the lanes: the scheduled run keeps the Fast lane and skips the Slow lane; the nightly full run is N-3’s; the “whole browser matrix” wording near line 633 on 2026-10-25), and a revision entry.
  • .forge/specs/F40-accessibility-responsive-gate-spec.md: a revision for NFR-1, FR-7 (the result’s selection and flaky fields, the line’s new suffixes), FR-8 (P-8’s exclusions, selection, W-3’s covers case), FR-10 and FR-11 (the hook and the CI check reading the new line) (Q-1, Q-3).
  • .forge/specs/foundation/004-build-ci-spec.md: FR-J3 (coverage from the nightly run, B-6) and FR-G6 (retries outside CI, F-1 to F-3).
  • forge-et-fe and forge-et-pwa scripts/browser-gate/ci-check.mjs and the harness hook .claude/hooks/guard-browser-gate.mjs (its bash wrapper was deleted when the hook was unregistered): they parse the Browser gate: line, which gains · selected: …, · full, · N flaky and · chromium only (P-9). forge-et-fe scripts/browser-gate/run.mjs chooses the projects by operating system for P-9.
  • .forge/project-prd.md § Accessibility (the “Chromium and WebKit … gate every pull request” sentence), through .claude/rules/prd.md, with a PRD revision entry (P-9, Q-11).
  • .claude/commands/forge-deliver.md Stage 7, Stage 12 and Stage 15, and .claude/skills/forge-test-verify/SKILL.md (W-1, W-2, W-4).
  • .claude/hooks/inject-gate-state.sh and .claude/hooks/README.md (N-2).
  • .claude/agents/spec-reviewer.md and plan-reviewer.md (H-3); .claude/agents/backend-reviewer.md and frontend-reviewer.md (H-5, F-5, M-8’s path rule).
  • .forge/design/harness/load-check-and-flaky-tests.md: a revision pointer on its rejected machine-wide lock and accepted gap (M-5), and on D-6’s “zero retries locally” (F-1).
  • Each app repo’s CLAUDE.md § Common commands (the new modes and options).
  • .forge/bugs.md: BUG-218’s Related: line points here (done in Draft 1’s PR); BUG-218 is resolved when Goal 3 is met.
  • .forge/owed.md: the rows on the gate’s run time and on VITEST_MAX_WORKERS point here (done in Draft 1’s PR), and close when H-4 and V-4 land (both closed by 2026-10-27; V-4 by documenting the variable rather than changing the default).

Acceptance criteria

  • AC-1 (N-1, N-3). The nightly job’s tests prove, against a stubbed gh, that a failed suite opens one nightly-red issue assigned to the four delivery-team members, a second failure comments on it, a passing night closes it, and an unchanged repo is not run. A first real run on the lead’s Mac runs all three repos’ suites in full and writes the nightly log (N-4).
  • AC-2 (N-2). With an open nightly-red issue, a new session’s start lists it; with no nightly log entry in 36 hours, it says so; with gh unavailable or slow, the hook prints the rest of its output, no error, within its existing time.
  • AC-3 (S-1 to S-3, S-7). For a change confined to one feature, each selected mode runs that feature’s tests and the always-run set, and names them; for a change to a shared-code path, each runs the full suite; every test that reads files as text is in every selected run. Proved by the scripts’ own tests on fixture diffs, and by S-2’s map test failing on an unmapped spec.
  • AC-4 (Phase 1 measurements). The build record, test record and browser gate are measured on one commit before and after Phase 1 on an 8-core Mac and on the Windows machine, and the figures are added to this document.
  • AC-5 (Goal 3). A full browser gate passes three times in a row, as Goal 3 defines a pass, on the Windows machine and on a Mac with nothing else heavy running (BUG-218’s regression test).
  • AC-6 (Goal 2). After Phase 2, a one-feature forge-et-fe PR’s gate takes 15 minutes or less on an 8-core Mac, measured on three PRs.
  • AC-7 (Goal 4). After Phase 3’s backend items, the build record’s median over ten PRs is at least a third lower than E-1’s on each operating system, with no test that checks behaviour removed.
  • AC-8 (Goal 6). Each night’s log entry gives every suite’s test count and duration and the change since the previous night; proved by the job’s tests on two recorded nights.
  • AC-9 (P-1, P-9). On an ordinary PR, the CRM’s gate runs P-1’s four sizes, on Windows Chromium only, and its Browser gate: line says which; on a wave PR, a PR into release/* or with --full, it runs every project; the Fast lane’s check refuses a wave PR or a release/* PR whose line is not a full-profile run. Proved by the gate scripts’ tests.
  • AC-10 (F-1, F-3). A test that fails and then passes on its retry is counted as passed, named as flaky in the result and the line, and a run with more than 5 such tests fails; proved by the gate’s, the test record’s and the build record’s tests on fixture reports.
  • AC-11 (M-5). A second heavy run on the same machine waits for the first and starts when it ends; a lock whose process is gone is removed; --no-wait refuses instead; the nightly job’s wait ends after an hour and reports the suite not run. Proved by the lock module’s tests.
  • AC-12 (B-4, B-6). A local ./gradlew build runs no performance-tagged test and no JaCoCo agent; the same build with -Pet.fullSuite=true, or with CI=true, runs both. Proved by one run of each.
  • AC-13 (the scheduled run). After the workflow change, a scheduled PR validation run runs the Fast lane, Trivy included, and skips the Slow lane; a release/* push, a Dependabot PR and a manual dispatch still run the Slow lane. Proved by the next scheduled run and one manual dispatch. (Superseded by N-5: a release/* push runs the Fast lane only, the schedule is weekly, and from 2026-11-13 a Dependabot PR does not run the Slow lane.)

Measurements

Point-in-time figures, each with how it was taken. AC-4 asks for before-and-after runs on one commit on an 8-core Mac and on the Windows machine; the rows below are what exists so far, and the Windows rows are owed (.forge/owed.md, “Build Phase 1 of the test-run cost design”).

What Before After How measured
CRM browser gate, 8-core Mac, 4 workers full profile: 5,627 tests, 48–51 min (E-1, 2026-10-22) PR profile: 4,004 tests, 33.7 min, passed at the first attempt — 3,999 passed, 5 skipped, 0 flaky, 0 stalls — with the machine 0.86 busy on average pnpm e2e:gate --push on forge-et-fe#346’s head 733aa51, 2026-10-27; .browser-gate/result.json startedAt/finishedAt
Tests in each CRM gate scope full: 5,656 Mac PR profile: 4,004 (−29%); Windows PR profile: 3,981, no WebKit ET_GATE_SCOPE=<scope> pnpm exec playwright test --list on forge-et-fe f61040d, 2026-10-27
Capture app browser gate, 8-core Mac — full (unchanged): 133 tests passed pnpm e2e:gate --push on forge-et-pwa#97’s head 917d6be, 2026-10-27
Backend build record, 8-core Mac (B-1, B-4, B-6) median 21 min for 10,158 tests (E-1, n=20, 2026-10-22) 10,335 tests in 18 min 55 s on an otherwise quiet machine (0.21 busy on average); 20 min 36 s on the next head with two Vitest records running beside it python3 scripts/local-build-evidence.py on forge-et-be#317’s heads e5e38cc and 9c819cd, 2026-10-27. Not the same commit as the baseline, so the figure is indicative; AC-4’s same-commit comparison and the Windows rows are owed
forge-et-pwa shared-file tests, 8-core Mac (V-3) sharedDrift.test.ts: 144 tests, 179 s (the nightly run of 2026-10-27) the same 144 tests in three files: 195 s with the fixture built once per file alone, 61 s with the batched read alone, 22.5 s with both and the split; the whole capture-app Vitest suite 180 s → 55 s pnpm exec vitest --run src/test/scripts/sharedDrift* on forge-et-pwa#99’s branch, 2026-10-27, with other sessions’ builds running (load average 13 to 19)
forge-et-fe Vitest, 8-core Mac (V-1) one project, all in jsdom: 9,537 tests in 241 s, 1,105 s of CPU two projects: the same 9,537 tests in 168 s, 955 s of CPU (−14%); the 149 *.test.ts files that need no DOM: 32.7 s in jsdom, 10.9 s in Node /usr/bin/time -p pnpm exec vitest --run --config <old or new> back to back on one commit of forge-et-fe#350, 2026-10-27; the machine was loaded by other sessions (load average 47 before the second run, 51 after it), so the CPU time is the fairer figure
forge-et-fe Vitest workers (V-4) the default, 7 workers on 8 cores: 342 s and 340 s; 1,132 s and 1,147 s of CPU 4 workers: 262 s and 333 s; 958 s and 1,040 s of CPU. All four runs passed (9,571 to 9,582 tests) /usr/bin/time -p pnpm exec vitest --run, then with VITEST_MAX_WORKERS=4, twice each, alternating, on forge-et-fe#350’s branch, 2026-10-27, while two review agents and other sessions ran (load average 8 to 38): too noisy to show a cost or a saving, so the default is kept and the variable documented. An earlier run with VITEST_MAX_WORKERS=50% ran 50 workers and is discarded
forge-et-be selected runs (S-3) the whole suite: 1,139 s of test time over 809 classes a change in one package of quote or priceemail: about 5% of it; pricing 16%; lead, account, tag or ai about half; auth, users, mailbox, attachment, platform or shared code: everything select-tests.py’s choice for one changed file per package, priced with the nightly run’s JUnit times, 2026-10-27
The first nightly full run, 8-core Mac (N-3) — all five suites green, run one after another: backend build 10,341 tests in 19 min 39 s; CRM Vitest 9,219 in 3 min 00 s; CRM browser gate, full profile, 5,656 (5,651 passed, 5 skipped, 0 flaky) in 48 min 13 s; capture app Vitest 807 in 3 min 02 s; capture app browser gate 133 in 40 s — about 76 minutes from the first suite’s start to the last one’s end run-nightly.py --now on be 81d6a19, fe 6d52831, pwa 80d3dd3, 2026-10-27; ~/.et-nightly/logs/2026-10-27/summary.md, posted to the nightly-log issue (forge-et-harness#981)

Appendix — re-checking the evidence

  • Record durations: gh api repos/altrium-io/<repo>/issues/<N>/comments and the local-build-evidence / local-test-evidence comments’ time column.
  • Browser gate composition: pnpm exec playwright test --list in forge-et-fe, counted by project.
  • Backend per-test times: gh run download <nightly run id> -R altrium-io/forge-et-be -n backend-reports, then the time attributes of build/test-results/test/TEST-*.xml.
  • Vitest breakdown: gh run view <nightly run id> -R altrium-io/forge-et-fe --log --job <Slow lane job id> — the Duration line and the per-file times.
  • Files that read source as text: git -C forge-et-fe grep -l -E 'readFileSync|readdirSync' -- 'src/**/*.test.ts' 'src/**/*.test.tsx'.
  • Flaky and Windows bugs (E-6, E-10): the ### BUG- headings of .forge/bugs.md whose title contains “Flaky” or “intermittent”, or whose title or Found in line names Windows or macOS, joined to yq '.bugs[] | [.id, .status, .found_date, .notes]' .forge/tracker.yaml for the status, the week found and the fix. The causes in E-10 are read from each bug’s notes and Fix line, not counted by a script.
  • Nightly results: to 2026-10-27, gh run list -R altrium-io/<repo> --workflow "PR validation" --branch main --event schedule; from N-3, the comments on the harness issue labelled nightly-log (gh issue list -R altrium-io/forge-et-harness --label nightly-log).

Revisions

  • Draft 1, 2026-10-25: first draft.
  • Draft 2, 2026-10-25: applies the fresh-context review of Draft 1 (“Ready after Importants”). P-1 keeps a 1024 laptop size on PRs (four sizes, not three); P-2 runs axe once per layout band rather than at one size per engine, because markup changes with the viewport; S-2 uses an explicit feature map with a test, since feature and spec folders do not line up; new S-7 always runs tests that read files as text; P-3 names that the stub is removed by import.meta.env.DEV and what each spike option costs; E-3 and P-6 corrected — only the size matrix runs in parallel, and fullyParallel is the lever; N-1 tells a replaced run from a timeout, alerts on “still red”, adds a dispatch test input and the label, and keeps the BUG-row rule; N-3 gains its security risks and the token it needs; Phase 1’s workflow edits need the lead’s approval (Q-9); E-2 records the forge-et-fe Slow lane at 80% of its cap, N-4 warns at 80% (Goal 6, AC-8) and V-3’s longest files move to Phase 1; new P-8 takes unit-test files out of the gate inputs; F-2 reads existing records instead of a committed ledger; § What changes elsewhere adds the hooks, ci-check.mjs, F40 FR-10/11, FS-4 FR-J3/FR-G6, coding-standards §6.8 and the load-check design; Nits: N-2’s single cached call, B-1’s Testcontainers default, NFR-1’s wording, E-5’s count of tooling tests, E-6’s account of #312 and #327, BUG-218’s Related: line, Goal 3’s definition, B-8’s tag and configuration file.
  • Draft 3, 2026-10-25: a review of whether the changes still help as the suites grow. New E-8 (what the size matrix has found: mostly load failures, two product defects) and E-9 (the suites’ growth per feature, projected to the end of V1). New § Long-run assessment: only selected runs and full runs off the laptops scale with growth, the other cuts are one-off, and each needs a rule for new tests; configuration-only cuts first; tooling kept small, with a fallback to a full run and removal when unused. N-3 gains a fallback when no machine is found, and Q-4 is decided first; F-1 states its trade-off and is adopted only with F-2, whose report gets a weekly owner; new H-5 (each speed fix’s rule for new tests, in the reviewer agents) and H-6 (the weekly trend reading); Phase 3 items are taken only on measured need.
  • Draft 4, 2026-10-26: every flaky, intermittent and Windows bug read, open and closed, with its tracker status and notes (Draft 1 had counted titles only). E-6’s counts corrected (47 flaky, 42 on or about Windows, against Draft 1’s title counts of 2026-10-25, 39 and 15). New E-10: where the flaky bugs are (16 of the 20 still open on Windows; all six open Vitest bugs; WebKit in most of the oldest), that they are arriving faster, what the fixed ones were (mostly races in the tests that load exposed; elapsed-time bounds; a growing list; LocalStack), a product defect found through one (BUG-261), an overload closure that came back (BUG-195 → BUG-259), and the Windows bugs that are not flaky (tooling written for macOS, machine limits, one real layout defect). Long-run item 5 corrected (load exposes the races, it does not cause them) and new item 7. New F-5 (review checks for the causes that lack one), M-8 (tooling tests on a Windows runner when the tooling changes, and a path rule for reviewers), P-9 (Windows PR gates run Chromium only; WebKit on macOS, at the wave PR and nightly); F-2 matches recurrences to closed bugs and is searched before filing; Q-4 recommends a Mac; new Q-10 and Q-11. A second fresh-context review recounted every figure (they hold) and found: P-9 revises the PRD’s “Chromium and WebKit … every pull request” (Blocker — now stated in P-9, Q-11 and § What changes elsewhere); P-9 keeps a tablet size on Windows and covers forge-et-fe only; four flaky bugs in progress, not five; BUG-195 closed on 2026-10-15, so it came back ten days later; BUG-203 is not Windows-specific; M-8 catches two of the seven tooling defects, not all; S-4 and Q-1 now allow N-3’s fallback; Q-4 states the Windows Chromium trade-off; Q-2 asks for F-2’s weekly owner. All applied, with its Nits (macOS 13 flaky bugs, five Vitest bugs of BUG-128’s shape, the match key for F-2, M-8’s phase and billed minutes, Track H’s order, wording on WebKit and load).
  • Draft 5, 2026-10-27: records the lead’s answers to Q-1 to Q-5 and Q-7 to Q-11 (§ Open questions gains an Answer column; Q-6 stays open with Nanndish). Q-3 and Q-4’s first answer were given on 2026-10-26, the rest on 2026-10-27. The changes they make:
    • N-3 rewritten. No spare machine (Q-4); the nightly full run of all three app repos — backend, both Vitest suites, both browser gates — runs on the lead’s Mac at 02:00 Sri Lanka time as a scheduled job (not a GitHub runner) that runs only main, on a fresh isolated database. CI’s scheduled Slow lane is switched off once the job has run; the scheduled Fast lane stays for AD #38’s nightly Trivy scan. N-1 is raised by that job rather than by a CI job; N-2 also reports a missed night; N-4 writes one nightly log on a harness issue. Goals 1 and 6, long-run item 1 and AC-1, AC-2 and AC-8 follow; new AC-9 for the PR gate profile.
    • Phase moves. Q-3, Q-7, Q-8 and Q-11 move P-1, P-2, P-8, P-9, M-5, B-4 and B-6 into Phase 1; Q-2 moves F-1 to F-3 in, with a limit of 5 flaky tests and the lead as the weekly reader of F-2’s report and H-6’s trend. B-4 and B-6 include the timing tests and coverage under -Pet.fullSuite=true or in CI, so no workflow change is needed for them. Q-1 decides S-4: selected records on any PR that touches no shared code, a selected browser gate only on a one-feature PR, starting once N-3 has run for two weeks (Phase 2).
    • Not built: M-7 (no runner), M-8’s Windows job (Q-10, no preference stated), a chat post for N-1 (Q-5).
    • Rejected alternatives gain a self-hosted runner, a cloud machine, WSL 2 for Windows runs, dropping the Test record, and ego(lite), each with the reason.
    • § Rollout, § Where this stands and § What changes elsewhere are rewritten to match.
    • A fresh-context review of Draft 5 (2026-10-27, “Ready after Importants”, no Blocker) found, and Draft 5 applies: the release and ship rules that count a green nightly CI run, and AD-O-11’s “no nightly”, must change with the workflow change (§ What changes elsewhere); N-2’s missed-night check works to the morning, with an N-4 entry every night; N-3 gains its own ports, time limits, a deadline, a wake time, a rule for a late start and locked worktrees; Draft 4’s trade-off of a Mac is restored in N-3 and long-run item 7; Q-11’s interim (the wave PR’s gate is the WebKit run until N-3’s first run) is restated in § Rollout, and P-2 waits for N-3; § Terms defines the full and PR profiles; the Windows tablet stand-in runs axe; N-4 records coverage; AC-10 to AC-13 are new; stale phase cells and wording are corrected.
  • 2026-10-27, the rest of Phase 1 built (no new draft): § Where this stands lists what forge-et-fe#350, forge-et-pwa#99, forge-et-be#322 and forge-et-harness#995 built, and the findings of the build and its reviews that refine decision rows: S-3’s package closure follows production code only (test code linked every package to every other), runs every api spec when a reached package serves HTTP instead of a map, and passes its filter in a Gradle init script; the build record counts each JUnit <testcase> by its final outcome, because Gradle’s suite attributes still count a retried attempt (F-3); P-2 leaves the axe scan out only at 1440 Chromium, because axe findings can differ by engine (BUG-271); a lock taken before the machine last started is a leftover (M-5), so a reused PID cannot keep runs waiting; and V-4 is not adopted, with the measurement. The V-3 split of forge-et-fe’s longest files and V-1 in forge-et-pwa are not done, each with the reason. New measurement rows (V-1, V-3, V-4, S-3). Two owed.md rows replace “Build Phase 1”: “Start Phase 2 of the test-run cost design” and “Run the test-run tooling on Windows”.
  • 2026-10-28, N-3’s first scheduled night (no new draft): the first run started by the LaunchAgent (the 2026-10-27 run was started by hand) hit every suite’s limit: the backend build at 60 minutes against 19 the night before, the capture app’s Vitest at 20 against 3. Cause: the agent’s ProcessType was Background, which confines the job to the efficiency cores; one CPU benchmark took 12.8 s as a Background launchd job and 3.3 s as an Interactive one. install.sh now writes Interactive (the installed agent on the lead’s Mac was changed the same day). A second defect turned a suite stopped at its limit into “could not run”: macOS answers EPERM to killpg when one member of the group cannot be signalled, which kill_group did not expect; it now never raises, with a self-test case. The capture app’s gate did not start because another session’s dev server held port 5173 all night. Also found finishing forge-et-fe#350: P-2’s 1440 Chromium project without axe failed ten size-matrix states, because the check that each of a state’s expectExemptions was applied ran there too (now only where the project scans, forge-et-fe 16141cda); CI’s Fast lane caught the skipped-test check refusing an it.skipIf (forge-et-fe e018cdf4) and an unpinned encoding (forge-et-be 6443b631), both of which a local run of every Fast lane check would have caught. The CRM full gate took four runs to pass: one stopped when the laptop was closed, one failed on a sleep on battery, one on the local database over the gate’s limit, and the fourth passed in about 55 minutes. Separately, BUG-275 failed the templates list’s axe test in forge-et-fe gates that include main; the session that took it found a test-timing defect rather than a colour one (axe ran while the row menu was still fading in) and fixes it in its own PR. Later the same day, forge-et-fe#350 needed main merged in for a conflict, and so a fifth full gate, which passed on f69662f8 in about 55 minutes with no flaky test; its review found that the C-9 CLI test started a real gate that could wait on the developer’s machine lock and hang pnpm test (fixed: each CLI run gets its own temp folder, and the gate refuses a dirty tree before the wait). That gate refused twice to start because another session’s ad-hoc pnpm exec playwright test runs, which take no machine lock, each started a dev server on port 5173; the sessions agreed to use only pnpm e2e:gate while one ran. An ad-hoc browser run outside the lock is a gap M-5 does not cover.
  • 2026-10-29, Phase 2 brought forward (no new draft): the lead decided to build Phase 2 now and switch it on after seven green or explained nights in a row of N-3, about 5 Nov, in place of Q-1’s two weeks (S-4’s row, Q-1’s answer, § Rollout and § Where this stands updated); S-2 and S-4’s selected modes are built first and stay unused on PRs until the switch-on PR rewrites the rules. Until then, a small contained PR may carry a browser-gate skip after its changed specs and its screen’s specs pass at every size in both engines, the reason naming what ran (owed.md); forge-et-fe#356, a wide PR, was merged that day on a skip after two gate runs were invalidated by the machine sleeping and the backend stopping, with the size matrix left to the nightly run. The lead also decided that an ad-hoc playwright test takes et-heavy-run.lock through a Playwright global setup in both frontends (M-5).
  • 2026-11-02, three nights lost to a closed lid (no new draft): the nights of 31 October, 1 November and 2 November were red or missing because the lead’s Mac was closed while travelling (pmset -g log: “Clamshell Sleep” at 02:19 on 31 October, 01:55 on 1 November and 01:14 on 2 November, each until the morning). With the lid closed the 01:55 wake gives only dark wakes of a few seconds every 15 minutes, so the backend build crawled from 02:10 to 10:09, about 25 tests took 750–940 s each, a timing test failed (ScanResultQueueIT AC-3 on 2 November; ScheduledJobRoutingIT and PriceExpirySweepIT on 31 October), and the deadline left the four frontend suites not run. Not a code defect: forge-et-be#344, which contains main’s head ef022576, passed the full build locally the same day (10,980 tests, none failed). Two runner defects made it worse, both fixed in run-nightly.py: a suite’s time limit stopped counting while the Mac slept (Python’s time.monotonic() stops on macOS), so the build ran on into the working day holding the machine lock; and the failure was reported as a code failure. A suite’s limit now counts the sleep, and a red suite the Mac slept through for more than two minutes is reported as not run, naming the sleep, with its timings dropped so they are not the next night’s comparison base. Such a night counts as explained for Phase 2’s seven-night count (§ Rollout).
  • 2026-11-02, the browser gate leaves the PR into main, and the records run what a change needs (no new draft; the lead’s decision): without waiting for the seven nights, the lead decided that a PR into main, a wave PR included, runs no browser gate, that the build and test records run the tests the change needs (every test for shared code), and that as much as possible is left to the nightly full run, whose findings are fixed together the next morning. S-4 is revised, the Rollout’s Phase 2 row and § Where this stands updated, and the rules rewritten (PRD Rev 81; .claude/CLAUDE.md § Quality Gates › Browser QA; git-conventions.md § Pull Requests; test-strategy.md Rev 35; architecture.md Rev 83; the skills and commands that ran the gate). A PR into release/* keeps the full-profile gate, because N-3 tests only main. Built in forge-et-fe#379 and forge-et-pwa#105 (the test record’s selected mode and the Fast lane step only on a PR into release/*) and forge-et-be#348 (the build record’s selected mode); measured on their own heads: a selected frontend record ran 63 files and 2,482 tests, and a selected backend record 520 tests in 1 min 45 s against 57 min for the last full one. The harness hook on gh pr create / gh pr ready is unregistered and its bash wrapper deleted; its Node file stays, because the Docker-naming hook reads commands through its parser (owed.md). Recommended to the lead and recorded, not decided: run pnpm e2e:gate --selected (about 3 minutes for one feature) before marking a screen-changing PR ready, written into the Browser QA rule as the development check; cut a release tag only from a commit with a fully green night (owed.md, a question for the lead); keep the secret scan, the Fast lane’s static checks and the review record on every PR, since a pushed secret cannot be fixed the next morning. Risks named to the lead: N-3 on one laptop is a single point of failure (three of the four nights before the decision were lost to a closed lid); a red night covers several PRs, and main and dev carry the break for up to a day. Separately, Nanndish reported why the backend build is slower on Windows (2026-11-02, from a record run on the Windows machine): Microsoft Defender scanning every file the build touches (MsMpEng the busiest process, about 80% of a CPU), small-file costs on NTFS, Docker reached through a named pipe into WSL 2 with the database port forwarded across it, the search indexer reading the build output, and one test JVM for every test (no maxParallelForks, confirmed in build.gradle). The fixes Nanndish lists are M-1’s (a Dev Drive, or Defender exclusions; indexing off for the workspace; a .wslconfig memory and processor cap; a quiet machine on mains power); the recommendation recorded with Q-6 is a Dev Drive first, because Defender keeps scanning there, and the before-and-after record on one commit that M-1 asks for. Parallel test JVMs are B-7, kept in Phase 3 after B-2 and B-3 because the shared databases and timing tests would turn flaky; no new proposal is needed.
  • 2026-11-02, the lead’s follow-up — PR gates as light as possible, everything removed carried by N-3 (no new draft): the lead accepted the three recommendations above (the selected gate as the development check, expected and not enforced; a release tag only on a commit with a fully green night, now in .claude/rules/release-and-branch-protection.md § Release Process and /forge-release; the full gate kept on PRs into release/*), and asked that N-3 carry everything a PR no longer runs, plus the checks a PR skips. N-3 gains, in run-nightly.py: a scan suite per repo, run every night whether or not main moved (gitleaks over main’s whole history, which also covers the DevOps team’s deploy-file pushes that skip PRs, and fails on a secret; Trivy over the lockfiles, reported without failing, AD #38), and a static-checks suite per frontend (Biome, the typecheck, the folder boundaries, the test-hook coverage, the production build, the bundle budget, and in the capture app the live comparison of its copies of forge-et-fe’s files), which no alerting run checked on main before, so two cleanly merged PRs could break them unseen. After the fresh-context review of forge-et-harness#1075: a gitleaks error is reported as not run rather than as a secret, a found secret stays a failure even if the Mac slept, a Trivy run that did not happen is listed with the findings, a static check skipped at the suite’s limit makes the suite not run rather than failed, and a run started with --now says so in its log heading (“started by hand at HH:MM”), so neither the missed-night check nor the release rule reads it as a night. The first full run by hand (2026-11-02, 18:32, on the runner of that afternoon): the three scans, the backend build (11,019 tests, 24 min 56 s, 96.3% lines), both static suites and both Vitest suites (10,649 and 897 tests) passed, which closed forge-et-be#343; both browser gates did not run, because another session’s F20-i verify stack held the CRM stack’s ports 55432 and 45666 and its Vite server held port 5173. The nightly stack now has host ports of its own (56432, 46666, laid over compose.verify.yaml by a file the runner writes), checked by starting it beside that F20-i stack; port 5173 is still shared (owed.md, “Keep the nightly full run reliable”). Later runs by hand the same evening: at 20:00 the scans and both static suites passed on the final runner, every new check included (test hooks, bundle budget, the capture app’s live shared-file comparison); at 20:25, once port 5173 was free, the capture app’s gate passed (134 tests, 54 s); a run of the CRM gate alone failed to write its ports file into a log folder that did not exist yet (fixed in forge-et-harness#1078); the CRM gate then ran from 20:28 on the nightly stack’s own ports, but the Mac hibernated from 21:58 to 22:29 on a 4% battery, so the runner stopped the gate at its limit on waking and reported it not run, as designed. By then it had reached test 4,764 of 5,973, with 1,154 distinct tests failing in 125 spec files — the revamped record pages, notes, sharing, the lead page’s AI panels, email compose and the size matrix — the specs forge-et-fe#364 left behind, which forge-et-fe#371 updates (its comparison with this run is posted on #371). The stopped gate left Playwright’s vite preview serving port 5173, because it runs in a process group of its own; the runner now stops whatever still runs from its own app-repo worktrees after each suite and on exit. The full run by hand from 23:18 (every suite forced) passed everything but the CRM gate, and closed forge-et-pwa#104; the CRM gate, with every one of its 1,533 failing tests retried, outran its 120-minute limit and was reported as “no gate result”. The CRM gate’s limit is now 180 minutes (a green gate takes about 50), and a gate stopped before writing its result reports the failures it had printed — how many, and the spec files with the most. Their first run (by hand, 2026-11-02) passed everywhere and reported one HIGH Trivy finding in the backend (Guava CVE-2026-102554, owed.md). A run started with --now now writes its logs to a folder of its own, after one replaced the night’s summary (restored from the nightly-log issue). /forge-deliver Stage 7’s final-wave type: e2e run is narrowed to the item’s own specs, settling that owed.md question the same way. What a PR into main still runs: the Fast lane in CI (static checks, secret scan, build, Trivy report — minutes, no developer time), the selected test record, and the review record with its security pass.
  • 2026-11-03, a suite runs in one place (N-5; no new draft; the lead’s decisions): the night of 2–3 November was lost when the Mac hibernated on a 1% battery at 02:56, 18 minutes into the CRM gate; a full run by hand from 09:51 (1 h 28 min, on mains power) passed everything but the CRM gate (5,880 passed, 9 failed, 129 known failures), whose nine failures were triaged into BUG-324 (four specs left behind by forge-et-be#342’s AD #41 keys), BUG-309 (now also at 1024 and 1280, the role editor having grown) and BUG-313’s and BUG-308’s marks (.forge/bugs.md). The lead then decided, each on Claude’s recommendation unless stated: the scheduled CI run becomes weekly, still skipping a week main did not move but no longer failing a quiet week after a red run (“Nightly still red” is gone), and is kept only to refresh the caches; N-3 gains every check the Fast lane ran on main (the frontends’ work-marker, skipped-test, fixture-leak, stub-sign-in and Knip checks; the backend’s guard scripts, its DDL drift check and its jar fixture check, run after the build); a PR into release/* records every test (the lead’s decision of the same day), and, on the lead’s own call, a push to release/* runs the Fast lane only, with the protect-releases ruleset now requiring the PR to be up to date (set by Claude with the lead’s admin rights, 2026-11-03); a tag on main checks the Fast lane only (-f tests=false, a new input of the manual run). Also decided: the nightly run stays on the lead’s Mac, with no always-on machine (owed.md), and, to cover the nights it misses, a missed or interrupted night is run again by hand the next morning (run-nightly.py --now; git-conventions.md § Pull Requests; the session start’s missed-night line now names that step). Forge-et-be’s Guava finding is fixed by a constraint on the errorprone configuration (33.4.6 → 33.7.2; Trivy clean).
  • 2026-11-04, the gates’ own ports (no new draft; the lead’s decision): a held port 5173 had stopped a gate on 28 October, 2 November and 3 November, because both gates served the app on the port pnpm dev and hand-run Playwright specs use. Under the gate (E2E_GATE=1) forge-et-fe now serves the app on 15173 and forge-et-pwa on 15174 (forge-et-fe#399, forge-et-pwa#108). Every other local Playwright run stays on 5173 and reuses a running pnpm dev. The backend needs no change: the dev proxy removes Origin, the stub sign-in goes through the API, and the mailbox specs open the SPA’s marker URLs directly. The nightly runner’s missed-night line names the new ports. A hand-run spec that waits on the lock still holds 5173 (BUG-330), but that no longer stops a gate. Both PRs merged the same day (8cae2485, 498c09cb), and no separate follow-up check is kept. If a gate fails on its new port, the nightly run opens a nightly-red issue (N-1). Windows is covered by ordinary use: every local Playwright run now passes --port to the app server, so a Windows teammate’s first pnpm e2e or gate exercises it, and a server that does not start fails the run. This answers frontend-reviewer’s Nit 7 on forge-et-fe#399, that only macOS had run it.
  • 2026-11-08, the night checks the backend’s local config (no new draft): on the night of 7 November, 21 of the CRM gate’s failures came from the lead’s forge-et-be application-local.yml, which N-3 copies into its backend worktree: it lacked the two lead-ranking queue keys F11-g had added to .example, so every lead stayed pending (triage on forge-et-fe#430; the code defects of that night are BUG-377 and BUG-378). run-nightly.py now compares the local file’s key names with .example on origin/main before the suites run, without reading a value, since the file holds local secrets. A gap is listed with the CRM gate’s reported findings, and on a red gate it leads the summary the nightly-red issue shows; it fails nothing, because a key the gate does not exercise must not stop the night. --dry-run prints the same check.
  • 2026-11-13, Dependabot PRs get a local record (N-5; no new draft; the lead’s decision): N-5 had kept one exception, the Slow lane on a Dependabot PR, because no developer stood behind one. In the 30 days to 2026-11-12 those runs took about 560 runner minutes (the CRM’s 397, the backend’s 147, the capture app’s 14), the CRM’s raised by every @dependabot rebase, and the backend’s Slow lane had reached 47 of its 60 minutes and was growing 5–6 minutes a week, which a GitHub-hosted runner cannot shorten (two test JVMs do not fit it, B-7). The Slow lane now runs only on demand. Before a Dependabot PR merges, a delivery-team member runs the repo’s local record on its final head — after any @dependabot rebase — as on any other PR, and forge-review-dependabot.yml sets only Forge review. The review a Dependabot PR already needed (release notes, where the package is used, the checks that exercise it: .claude/rules/review-record.md) is unchanged. Changed with it: the three app repos’ pr-validation.yml and forge-review-dependabot.yml, forge-review-gate.md D-4, and the rules and documents that stated the exception.
  • 2026-11-13, the night’s CRM gate was lost to another session (N-1, N-3; no new draft; the lead’s decision on closing): the scheduled run passed every suite but the CRM gate. That gate was reported not run within seconds, because another session’s backend was listening on port 18080, the port the gate’s own backend uses. The same session’s BUG-415 load reproduction (eight yes busy loops) kept the CPU 0.98 busy while its Playwright run waited for the night’s lock, so the capture-app gate refused to start until the loops were stopped. Run by hand afterwards (--now --only fe-gate), the CRM gate passed: 8,365 passed, 5 skipped, no failures, 51 min 36 s. Changed: the runner waits up to 30 minutes for 18080, checking every five minutes, and names the process holding it (forge-et-harness#1351). test-record.md names the nightly’s ports and says load started on purpose begins only after its run holds the lock (L-293). A hand run, --only included, now closes a repo’s nightly-red issue when every suite it ran passed and every other suite of that repo last passed (forge-et-harness#1359, N-1; the lead). Until then a partial run never closed one, so forge-et-fe#484 was closed by hand the same morning.
  • 2026-11-13, the capture app’s gate starts the backend (N-3; no new draft): from F36-a, forge-et-pwa’s browser gate has a seam project whose specs talk to the real backend through the Vite /api proxy, and a seam-backend setup project that fails the gate at once when nothing answers on ET_BACKEND_PORT (F36-a WI-1.2 plan Decision 17). run-nightly.py now runs pwa-gate as it runs fe-gate: it starts the isolated stack and the backend on the night’s own ports, runs the gate with ET_BACKEND_PORT=18080, and stops both afterwards, each gate on a fresh stack of its own. pwa-gate takes forge-et-be as an input, so a backend change runs it again, and the local-config check’s note goes on both gates. N-3’s stack ports are corrected with it to the 56432 and 46666 the runner uses. It settles the owed.md row “Start the backend for the capture app’s nightly browser suite”, which closes on the first night after F36-a’s wave 1 merges.
  • 2026-11-17, the capture app’s gate limit is 90 minutes (N-3; no new draft): the capture app’s browser gate grew from 142 tests (under a minute, 2026-11-14) to 759 (3 min 35 s, 2026-11-15) and to about 5,260 by 2026-11-17, as F36’s specs added the phone, tablet and desktop sizes in Chromium and WebKit. Its 30-minute limit, set when it took under a minute, stopped two hand runs on 2026-11-17 before the end: at 11:15 with 22 failures, and at 16:30 after 2,796 tests had started while another session’s runs kept the load near 63 on 8 cores. run-nightly.py now gives it 90 minutes, room for each failing test’s second run and some load. N-3’s limits sentence is corrected with it to the CRM gate’s 180 minutes, which the runner has used since the night of 2026-11-02 outran 120.
  • 2026-11-18, a record carried across a merge of main (new W-5; no new draft; the lead’s decision): Nanndish could not merge forge-et-be#457 for most of 2026-11-17: each merge of main to resolve a conflict made a new head that needed another 100-minute full record, and more PRs landed on main while it ran. Only one 4-line resolution in docs/api/README.md separated the merged head from its recorded commit. The lead chose to carry the record across a merge of the base branch whose resolutions touch no test or build input, in all three app repos (W-5). git-conventions.md § Branching and § The test suites run locally and test-record.md state it, and each app repo’s CLAUDE.md says it beside the documentation carry. Run on #457’s own commits, the new rule covers 44af099e from the record on 07e40690, and does not cover 25184660, whose merge resolved AuditCodeDocumentSpec.groovy. The backend review of forge-et-be#463 found two gaps in the first version, both closed before merge: git show --remerge-diff lists nothing for a conflict that left no markers (modify/delete, rename/rename, binary) when the merge keeps git’s version, so the conflicted paths now come from git merge-tree; and a clean merge can bring in a migration with the same version as the branch’s own, so such a merge is built.
  • 2026-11-18, the record scripts leave the browser gate’s inputs (new P-10; no new draft; the lead’s decision): W-5’s frontend PRs changed only the test record’s script and its Vitest test, yet pnpm e2e:gate --selected chose the full gate, because all of scripts/ was a gate input. The lead chose to exclude the four record scripts by name, with a boundary rule that reads the list (rather than computing gate inputs from the import graph, or keeping the full run with a skip each time), and to build it the same day. On a throwaway forge-et-fe commit that changed only three of those scripts, the selector answered none — no gate input changed; a change to scripts/browser-gate/select.mjs still answered full. The review found that it saves nothing in forge-et-pwa, where each record-script change is a re-copy that also changes the shared-file hash list, a gate input; whether the capture app’s selector should look past those hash lines was held in a new owed.md row in its place. Decided the same day (the lead): no. Teaching the shared selector the capture app’s hash list would put a capture-app case in code both apps share, to save a run on a PR that comes a few times a month; such a PR records Selected browser run: skipped — <reason>, and the nightly full run checks the capture app’s whole gate. Revisit only if re-copy-only PRs become frequent. The row is removed.
  • 2026-11-21, the nightly run has the next turn at the lock (M-5, N-3; no new draft; the lead’s decision, BUG-521): the night of 2026-11-21 ran no suite. It waited its hour for the lock while other sessions’ runs passed it from one to the next, each within two seconds of the last release, and the nightly run looked only every 30 seconds. While it waits, the nightly run now writes a reservation, et-heavy-run.next, beside the lock, and every taker (heavy-run.py, both frontends’ machine.mjs, the build record) leaves a free lock to a reservation that names a live process other than itself, written since the last boot and less than 90 minutes ago. A run the holder started is not affected. The cost: a session’s run started during the nightly run waits until it ends.
  • 2026-11-18, M-5’s lock covers every heavy run (no new draft; Dinil’s decision from F11-i’s Reflect, L-315): this log’s earlier note that an ad-hoc run outside the lock is a gap M-5 does not cover proved costly: three sessions shared one machine overnight, agents’ direct Gradle builds, bootRun and test runs took no lock, and the sessions improvised per-ticket lock folders that only some of them checked; one backend build took about 5 hours instead of about 85 minutes. Every heavy command not already taking the lock now runs through python3 .claude/lib/heavy-run.py -- <command>, which takes the same et-heavy-run.lock with the same steps as the records, and a backend a suite needs is started inside the same wrapper call as that suite (test-record.md § One heavy run at a time; the agent prompt template’s § Shared machine lock). Per-ticket lock folders are no longer used.
  • 2026-10-26 (no new draft): new § Where this stands, and the next steps, so a later session can continue from this file and its owed.md row. Draft 4’s revision line corrected from “two real layout defects” to one; E-10 has said one since the review.

Generated from .forge/design/harness/test-run-cost.md by .claude/lib/render-docs-html.js. Markdown is canonical.