Test runs explained — what runs on your PR, what runs at night
For the whole team. The rules are in
.claude/CLAUDE.md § Quality Gates,.claude/rules/git-conventions.md § Pull Requestsand.forge/test-strategy.md; the reasons and the evidence are in the design.forge/design/harness/test-run-cost.md(approved 2026-10-27; the lead’s decision of 2026-11-02 in its revision lines). True when written — check those files for the current state.
The short version
- Every suite runs in full, every night, on
main, on the lead’s Mac at 02:00 Sri Lanka time. That is a secret scan and a dependency scan of every repo, the backend build, each frontend’s static checks and build, both Vitest suites, and both browser gates at every size and in both browser engines. It is the only full run (the lead’s decision of 2026-11-02). - A release tag goes only on a commit the nightly run passed in full — check the latest scheduled night’s
nightly-logcomment before tagging (a run started by hand says so in its heading and does not count). - Your PR runs only what its change needs:
- no full browser gate on a PR into
main, wave PRs included, and noBrowser gate:line. But inforge-et-feandforge-et-pwa, runpnpm e2e:gate --selectedafter the test record and beforegh pr ready. It runs only the browser tests your change can affect, usually in a few minutes. Paste the line it prints into the PR’s test evidence asSelected browser run: …(none selectedwhen it picks nothing). Run it again if a PR touching the same screens merges intomainbefore yours does (.forge/design/harness/fewer-nightly-reds.md, since 2026-11-12); - the test record (
pnpm test:record,python3 scripts/local-build-evidence.py) runs the tests your change can affect, and everything when you changed shared code (scripts/shared-code.json); its comment says which ran and why.--fullruns everything; - the backend record still runs the whole
./gradlew build(compile, static checks, ArchUnit) with only the test set narrowed; the backend’s timing tests and coverage run only at night (and in a CI run you ask for).
- no full browser gate on a PR into
- A suite runs in one place, never in two. The test suites and browser gates run on our machines (the records, the gates, the nightly run). CI runs the quick checks (the
Fast lane) on every ready PR, on every push torelease/*, and once a week onmainto refresh the caches; the weekly run is skipped whenmainhas not moved. CI runs a suite only when you ask:gh workflow run "PR validation" --ref <branch>(onmain,-f tests=falsefor theFast lanealone). A Dependabot PR gets a local record from a delivery-team member before it merges, like any other PR. - Run the browser specs your change needs while you work:
pnpm e2e:gate --selected(about 3 minutes for one feature) or one spec withpnpm exec playwright test <spec>. Not required, but it saves a red night. - A red night is fixed the next morning, before new work starts, together with anything else that night found. Sort each failure by re-running only the failing tests, never the whole suite (§ The nightly full run). It opens an issue assigned to all four of us, and every Claude session start lists it.
- A PR into
release/*still runs the full browser gate (pnpm e2e:gate --full --push), and its test record runs every test by itself, because the nightly run tests onlymainand CI runs no suite onrelease/*. It must be up to date with the release branch before it merges, so the merge holds what you tested. - A test that fails and then passes on its one retry is counted as flaky, not failed — in the browser gate, the test record and the build record. More than five flaky tests fail the run. Each run names its flaky tests, and the lead’s weekly report decides which need a bug.
- Heavy runs on one machine queue instead of colliding. Start a second gate or record while one is running and it waits for the first, saying which. Runs keep the laptop awake while they go (keep the lid open).
What runs where
flowchart LR
subgraph DEV["While you work"]
D1["pnpm test:selected<br/>select-tests.py<br/>pnpm e2e:gate --selected<br/>(only what the change needs)"]
end
subgraph PR["Your PR into main (your laptop)"]
R1["Build / test record<br/>(the tests the change needs;<br/>everything for shared code;<br/>one retry, flaky named)"]
end
subgraph REL["PR into release/*"]
G2["Every test in the record<br/>+ browser gate, full profile<br/>(7 sizes, Chromium + WebKit);<br/>must be up to date to merge"]
end
subgraph NIGHT["Every night, 02:00 (the lead's Mac)"]
N0["gitleaks + Trivy, every repo"]
N1["Backend build + every test + timing tests + coverage,<br/>then the Fast lane's guard scripts"]
N4["Frontends: Biome, typecheck, boundaries,<br/>test hooks, work markers, no skipped test,<br/>build, bundle budget, fixture and stub checks,<br/>shared-file drift (capture app), Knip"]
N2["Both Vitest suites + coverage"]
N3["Both browser gates, full profile"]
end
subgraph CI["GitHub CI"]
C1["Every ready PR and every release/* push:<br/>Fast lane (static checks, gitleaks, Trivy)"]
C2["Weekly on main, if it moved:<br/>Fast lane only, to refresh the caches"]
C3["A manual run only:<br/>Slow lane (the test suite)"]
end
NIGHT -- "red" --> ISSUE["nightly-red issue<br/>assigned to the team"]
NIGHT -- "every night" --> LOG["nightly-log issue:<br/>results, slow tests, flaky tests"]
How long each run takes (measured on 3 November 2026; the times grow with the suites, and the night’s nightly-log comment has the latest figures):
| Run | When | About | CRM and capture app |
|---|---|---|---|
| Nightly full run | 02:00, the lead’s Mac | 1½ hours on a green night: scans 1 min, backend build and tests 30–32 min, CRM checks and Vitest 4 min, CRM browser gate 45–50 min, the capture app’s three suites 2 min | The same three suites each (checks and build, Vitest, browser gate). The CRM’s gate needs a backend, which the run starts; the capture app’s does not. The CRM tests tablet, desktop and 320px; the capture app phone, tablet and desktop |
| A PR’s Fast lane in CI | every ready PR, every release/* push |
backend 6–10 min, CRM 1–2 min, capture app 1 min | The same lane, plus a check of its own each: no stub sign-in in the CRM’s bundle; the capture app’s shared files and PWA build |
| The local record | once per PR, after the last push | backend 2–32 min (32 when shared code changed), CRM 1–4 min, capture app 1 min | The same script, byte for byte; the capture app’s needs no backend |
| The weekly scheduled CI run | Sunday 19:30 UTC | 1 min to skip, up to 10 min to run (the backend’s) | The same |
| A manual CI run with the suite | when asked | backend about 35 min, CRM about 24 min | The same tests switch |
A PR into release/* |
rare (hotfixes) | about an hour on the laptop | The same: every test, the full gate, up to date before it merges |
| After a dev deploy | each merge to main |
about 2 min to deploy; the smoke run by hand | Only the CRM has a smoke run so far; the capture app’s is owed by F38-g |
| UAT and production | a release tag | UAT: built for the backend and the CRM by F38-h, first run on 2026-11-16; production: not built yet (F38-i) | The capture app is first released from Phase 1’s round (AD #20; the lead, 2026-11-12) |
Your PR, step by step
The end of a PR into main: commit → git push → pnpm test:record → gh pr ready in the frontends; commit → git push → python3 scripts/local-build-evidence.py → gh pr ready in the backend. A PR into release/* in a frontend: commit → pnpm e2e:gate --full --push → pnpm test:record → gh pr ready.
- If another gate or record is already running on your machine, yours says
Waiting for … to finishand starts by itself when it ends.--no-waitrefuses instead. Don’t add--ignore-loadto get past a busy machine: wait, or close what you can. - If tests needed their retry, the records list them in their comments (and a gate’s line says
· N flaky). You don’t file a bug for each one: the weekly report does that for a test that keeps coming back (twice in 14 days, or once on an idle machine). - Every record and gate lists its slowest tests. If a test you added is among them (over 5 s for a backend or Vitest test, over 15 s for a browser test), say in the PR why it needs the time.
- Add a
PAGE_STATESentry for a new route or data state (e2e/pageStates.ts), so the nightly run checks it at every size. - A frontend record that fails only on timeouts under its own load can run fewer workers:
VITEST_MAX_WORKERS=4 pnpm test:record(a whole number).
When you do run the browser gate
| Command | What it runs | Its Browser gate: line ends with |
|---|---|---|
pnpm e2e:gate --selected |
the specs and size-matrix states e2e/feature-map.json maps to the features you changed; everything when you changed shared code, and always everything in forge-et-pwa, which has no map (under two minutes) |
· selected … (it covers no commit) |
pnpm e2e:gate on an ordinary branch, on a Mac |
every spec; size matrix at 320, 768 (WebKit), 1024, 1280 | · 4 sizes |
pnpm e2e:gate on an ordinary branch, on Windows |
every spec except WebKit ones; size matrix at 320, 768, 1024, 1280, all Chromium | · 4 sizes, chromium only |
pnpm e2e:gate --full, or on a wave, release/* or hotfix/* branch |
everything: 7 sizes, Chromium and WebKit | (nothing) |
The capture app (forge-et-pwa) always runs every size and both engines: it targets Safari on iPhone.
While you work: run what your change needs
| Repo | Command | What it runs |
|---|---|---|
forge-et-fe, forge-et-pwa |
pnpm test:selected |
the tests that import a file you changed since origin/main, plus every test that reads files as text; everything when you changed shared code (scripts/shared-code.json, such as the Vitest config) |
forge-et-be |
python3 scripts/select-tests.py |
the packages you changed, every package whose code uses them, every spec that names those or a test file you changed, every api spec when one of those packages serves HTTP, the arch specs and the specs that read files; everything for shared code (scripts/shared-code.json, such as build.gradle, migrations, platform) or a change that reaches most packages |
Each prints what it chose and why first; --list prints the choice without running it. On the backend a change in a small package runs about 5% of the suite’s test time, and one in lead or account about half. /forge-deliver uses these for a work item’s own checks, and the PR’s record uses the same selection.
The nightly full run
- Where: a scheduled job on the lead’s Mac (
.claude/lib/nightly/run-nightly.py, installed by.claude/lib/nightly/install.sh). It runs onlymain, in its own locked worktrees underworktrees/nightly/, with its own database stack and backend port, so it never touches your work. It takes the same machine lock as a gate or record, so it waits for one the lead left running. - What it reports:
- a red suite opens a
nightly-redissue in that repo, assigned to the four of us, or comments on the open one; the next green night closes it; - every night writes one comment on the harness issue labelled
nightly-log: each suite’s result, test count, time, coverage, how many tests are over the slow limit, the change in each since the night before, the tests over the limit, the 15 slowest tests, and the browser gates’ flaky tests; - every Claude session start shows open
nightly-redissues, and a night that did not run.
- a red suite opens a
- What to do with a red night: pick it up, fix it before starting new work, and file a
BUG-NNNif it is a product defect. The steps, as on 3 and 6 November:-
Read what failed. The
nightly-redissue names each failing test and the commits since the last green night. The detail is on the lead’s Mac:~/.et-nightly/logs/<date>/summary.md(a hand run’s folder is<date>-now-<HHMM>), one log per suite beside it, and each browser gate’s.browser-gate/result.jsonandplaywright-report.jsoninworktrees/nightly/forge-et-feorworktrees/nightly/forge-et-pwa. Both browser gates run against a backend the night starts. That backend reads the lead’s ownforge-et-beapplication-local.yml, and the night compares its key names withmain’s.example(values are never read). When keys are missing, each gate’s line in the issue starts by naming them. Add them to the local file first, because every test that reads one fails without it (21 lead-ranking tests on 7 November).run-nightly.py --dry-runruns the same check. -
Re-run only the failing and flaky tests, never the whole suite (
git-conventions.md § Pull Requests). Run them at the commits the night tested, in the nightly worktrees$WS/worktrees/nightly/<repo>, which stay at those commits until the next night resets them, with retries off. A browser spec takes seconds this way; the CRM gate takes about 50 minutes. The capture app’s browser specs mock the API, except itsseamproject (from F36-a), whose specs talk to the backend through the app’s/apiproxy. The CRM’s specs and the capture app’sseamspecs need a backend, started the way the night starts it, on the night’s own ports, so a developer’s stack can stay up:# from the harness checkout: the folder that holds the harness and the app repos WS="$(dirname "$(dirname "$(git rev-parse --path-format=absolute --git-common-dir)")")" cd "$WS/worktrees/nightly/forge-et-be" docker compose -p et-rerun -f compose.yaml -f compose.verify.yaml -f ~/.et-nightly/logs/<date>/compose.nightly-ports.yaml up -d --wait ET_DATASOURCE_PORT=56432 ET_FILES_STORAGE_ENDPOINTOVERRIDE=http://localhost:46666 SPRING_CLOUD_AWS_SQS_ENDPOINT=http://localhost:46666 SERVER_PORT=18080 ./gradlew bootRun --no-daemon "--args=--et.mailbox.rate-limit.refill-per-minute=60000 --et.mailbox.rate-limit.interactive-reserve=0" # in a second terminal, once http://localhost:18080/actuator/health answers UP: cd "$WS/worktrees/nightly/forge-et-fe" E2E_GATE=1 ET_BACKEND_PORT=18080 pnpm exec playwright test <spec> <spec> --project=chromium --retries=0 # or, for a capture-app seam spec (its seam-backend setup project runs first): cd "$WS/worktrees/nightly/forge-et-pwa" E2E_GATE=1 ET_BACKEND_PORT=18080 pnpm exec playwright test <spec> --project=seam --retries=0 # afterwards: stop bootRun, then cd "$WS/worktrees/nightly/forge-et-be" && docker compose -p et-rerun -f compose.yaml -f compose.verify.yaml -f ~/.et-nightly/logs/<date>/compose.nightly-ports.yaml down -vE2E_GATE=1is required. With it, Playwright builds the app and serves it on the gate’s own port, 15173 for the CRM and 15174 for the capture app, instead of usinglocalhost:5173. Without it, Playwright reuses any app already on 5173, such as yourpnpm dev, so the tests write their fixtures through that app into your own database (forge-et-fe#484, 2026-11-10). The ports and arguments are the onesrun-nightly.pysets (BACKEND_ENV,BACKEND_ARGS,NIGHT_PORTS_COMPOSE). For a failed Vitest file or backend spec, run that one file in its nightly worktree (pnpm exec vitest <file>,./gradlew test --tests '<Spec>'). Then compare each test’s time with the same test in earlier nights’ logs (grep '<spec>:<line>' ~/.et-nightly/logs/*/fe-gate.log) and the night’sMachine load:line in the suite’s log with earlier nights’. If the test also fails at the night’s commits, run it on currentmaintoo, since a later PR may already have fixed it. Then sort it into one of these:- Load: it passes alone in its usual time, and the night’s recorded load was well above earlier nights’. Record it on the issue. If the same test fails again on a loaded night, file a bug and triage it by
.forge/test-strategy.md § Flaky Tests. (On 6 November the CRM’s axe scan timed out at 30 s twice, with 13.7 load per core at the peak, then passed alone in 6 s.) - Environment: the Mac slept, or a port was held; the suite says “not run”. Clear the cause and run the night again.
- Stale test: another PR changed what the test expects, such as the shape of a backend answer.
- Real defect.
- Flake: it passes on its retry, or fails only sometimes. Check
pmset -g logand the recorded load first, and file it once for each operating system.
The triage backend (
et-rerun, on 18080) uses the night’s ports and runs from the nightly worktree, which the next 02:00 run resets. Take it down well before 02:00, or that night’s run fails on a held port. Another session may use it too, for example a fix PR’spnpm e2e:gate --selectedwithET_BACKEND_PORT=18080when nothing answers on 8080. Whichever session finishes last takes it down and tells the others. Each run adds fixtures to its database, and the browser gate refuses to start once the database holds more than 1,000 active users or leads (“over the browser gate’s limit”). When that happens, empty it: stopbootRun, run thedown -vandup -d --waitcommands above, then startbootRunagain. - Load: it passes alone in its usual time, and the night’s recorded load was well above earlier nights’. Record it on the issue. If the same test fails again on a loaded night, file a bug and triage it by
-
Record it. Post what the re-run showed on the
nightly-redissue: the command, the commits, each test’s time against the night’s, and the verdict. A defect, a stale test or a flake becomes aBUG-NNNin.forge/bugs.mdand the tracker, linked from the issue; claim an existing bug with/et-claimfirst, never file a duplicate. -
Fix it in a
fix/BUG-NNN-…PR, with a regression test that fails before the fix, its test record and its review record. When the fix must wait, mark the test a known failure under its bug id, so the night is green. -
Confirm it: the next scheduled night re-runs the red suite and closes the issue when it passes. Do not re-run the whole suite by hand to confirm a fix: the fix PR’s own run of the failing tests is the proof. A hand run (
--now, with or without--only) closes the issue only when what it ran passed and every other suite of that repo last passed. Its log still says “started by hand”, so it never counts as a scheduled night for a release.
-
- What to do with a missed or cut-short night. You know one happened when the session start says there was no run since the night before, or the red issue says a suite did not run. The lead fixes the cause on the Mac: mains power, the lid open, and the gates’ ports 15173 and 15174 free. Then the lead runs
python3 .claude/lib/nightly/run-nightly.py --nowthat morning, before new work starts. CI runs no test suite onmain, so otherwise nothing checks it until the next night. - It needs the Mac on, plugged in and awake at 02:00, with the lid open. The Mac is set to wake at 01:55. If it slept anyway, the job runs nothing during the day and the session start says so. A closed lid puts the Mac to sleep, whatever the run’s wake lock asks. The scheduled wake then wakes it for only a few seconds at a time, so the run crawls and its tests time out.
- A red suite that the Mac slept through for more than two minutes is reported as not run, with how long it slept. It runs again the next night.
- A suite is also stopped when the Mac wakes after the suite’s time limit, so it does not run on into the working day.
The weekly flaky report
The lead runs python3 .claude/lib/flaky-report.py once a week. It collects every flaky test that the gates, the records and the nightly log published over 14 days, and matches each to .forge/bugs.md (closed bugs included). It says which need a bug: a test flaky twice in 14 days, or once on an idle machine. A single flaky failure on a busy machine is only watched.
Why
Measured in October 2026: the CRM’s full browser gate had grown to about 5,600 tests. That took about 50 minutes on a Mac and 2.6 to 3 hours on Windows, and it almost never passed in one attempt on a busy laptop. So PRs waited overnight, and two heavy runs on one machine failed each other. The new setup keeps every check while giving the time back: the full run moves to the night, a PR or a work item runs only what it needs, heavy runs queue, and flaky tests are counted instead of failing the run. On 2 November the lead took the browser gate off PRs into main altogether, and made the records run only what a change needs, without waiting for the planned week of green nights. Problems are found by the next morning and fixed together. The full evidence and every measurement are in the design’s § Evidence and § Measurements.
What the trade-off asks of us
- The nightly run has to happen. The lead’s Mac stays on, plugged in and open overnight; three nights were lost to a closed lid from 31 October to 2 November, and one to a flat battery on 3 November. A night missed is a night nothing checks
main, since CI runs no suite there. - A
pnpm devleft running no longer stops the night. Since 4 November each browser gate serves the app on a port of its own (the CRM’s on 15173, the capture app’s on 15174), so apnpm devor a hand-run Playwright spec on 5173 can stay up. Only a gate’s own server left running (from a gate stopped by hand) still blocks that gate. - A red night is fixed first. Several PRs may have merged since the last green night, so whoever picks it up reads the commits the issue lists and finds which one broke it.
- Run the selected gate for a screen you changed. It is the cheapest way to keep the nightly run green.
Sources
.forge/design/harness/test-run-cost.md— the decisions (N-1 to N-4, P-1, P-2, P-8, P-9, F-1 to F-3, M-2, M-5, S-1 to S-4, H-1, H-2, W-1, B-1, B-4, B-6, V-1, V-3) and the evidence; PRD Rev 81..claude/CLAUDE.md § Quality Gates › Browser QA;.claude/rules/git-conventions.md § The test suites run locally, and the PR records them;.forge/test-strategy.md § When Tests Run,§ Browsers & Viewports,§ Flaky Tests.forge-et-fee2e/README.md § The browser gate; each app repo’sCLAUDE.md § Common commands.