Fewer red nights: catch broken browser tests before the merge — Design
Status: approved by the lead (Draft 2, 2026-11-12): recommendations 1–3 are adopted as R-1 to R-3 (§ The rules adopted). In force when the implementation PR, reviewed by a fresh-context reviewer, merges (
.claude/CLAUDE.md § Reference Map, harness design records). Author: Tharinda Rajapaksha, from the triage of forge-et-fe#484 (2026-11-10) Date: 2026-11-11 Scope, if accepted:.claude/CLAUDE.md § Quality Gates(the browser-QA paragraph),.claude/rules/git-conventions.md § Pull Requests,.claude/rules/test-record.md,.claude/agents/frontend-reviewer.md,.forge/test-strategy.md § When Tests Run, andforge-et-fe’s PR template andCLAUDE.md. Nothing is cloud-gated, and no AWS cost changes. Recommendation 1 adds local run time to a frontend UI PR; it adds no GitHub Actions minutes.
Problem
Since PRD Rev 81 (2026-11-02), a PR into main runs no browser gate, and the nightly full run on
the lead’s Mac is the only run of the browser suites (.claude/CLAUDE.md § Quality Gates, Browser
QA). A red night stops new work until it is fixed (git-conventions.md § Pull Requests), so every
cause the nightly run finds after a merge costs the team a morning. The question is which of those
causes could have been caught before the merge, and at what cost.
Evidence (as read on 2026-11-10 and 2026-11-11)
Sources: the nightly log (the harness issue labelled nightly-log), the nightly-red issues in the
three app repos, .forge/bugs.md and .forge/lessons.md. Counts are point-in-time.
| # | Cause | Examples | Could a pre-merge step have caught it? |
|---|---|---|---|
| C-1 | The nightly machine itself: the Mac slept or its lid was closed, a gate’s port was held, a config key or file was missing, a suite hit its time limit | 2026-10-28 (every suite at its limit), 2026-11-02 (the night’s Compose ports file was missing; then a 31-minute sleep), 2026-11-11 (both browser gates not run: the Mac slept for 177 minutes) | No. It is not a code defect. About half of the red or missed nights in the log are this kind |
| C-2 | A change merged without running the browser tests it affects | forge-et-fe#442 (feature/ui-rev100, 709 files) updated the specs for its changed strings but ran none of them, as its PR body says; the first failures of forge-et-fe#484 (BUG-390–393, BUG-397) start at its merge |
Yes: a selected browser run on the PR |
| C-3 | Two PRs that each pass, merged one after the other, break each other | forge-et-fe#470 (F32-d, the Country list became a multi-select) and #472 (tests written against the old single-select), BUG-398; BUG-324 (browser specs against AD #41’s keys); BUG-326 (a spec importing a helper another PR removed) | Partly: a selected run again when main has moved in the same area since the PR’s last run |
| C-4 | Flaky tests | 84 bugs titled Flaky in bugs.md on 2026-11-11, 77 of which mention machine load and 63 Windows; lessons L-101 (wait for the page to settle, not for an element), L-125 (no assertion on what a time-bounded run got done), L-162 (reproduce under a throttled CPU before calling a test flaky) |
Partly: the lessons are known but not enforced by any check |
| C-5 | A size or accessibility failure that only the size matrix finds | the size-matrix and WebKit failures in forge-et-fe#484 (pricing-import-create-unknown at 768 and 1024 wide) |
Yes, when the selected run includes the size projects for the changed screens |
Recommendations
Recommended now (they target C-2 and C-3, the causes a pre-merge step can catch):
- A selected browser run before a frontend UI PR is marked ready. A
forge-et-feorforge-et-pwaPR that changes a screen, a string, a test id or a token runspnpm e2e:gate --selectedon its final head beforegh pr ready, and its PR body carries theBrowser gate:line (skipped — <reason>allowed, as forrelease/*). Whether theFast lanechecks the line, as it does forrelease/*, is a decision for the lead, because it changes a workflow file. - A selected run again when
mainhas moved in the same area. Whenmainhas gained a merge that touches the same feature folder or the samee2e/specs since the PR’s last browser run, the PR mergesmainin and runs the selected gate again. This is one of the reasonsgit-conventions.md § Branchingalready accepts for mergingmainin. - A
frontend-reviewercheck that every string, test id and token the diff changes or removes is searched for ine2e/, and that each hit is either updated in the diff or explained.
Considered for later:
- Split a UI change the size of forge-et-fe#442 into PRs per screen area, each with its own selected run.
- Lint rules for L-101 and L-125 in
e2e/(nowaitForon an element where the lesson asks for a settled page; no assertion on a count reached within a time limit). - Move the nightly full run to a machine that does not sleep, which would remove most of C-1. This needs a price and the lead’s decision.
Open questions
- Q-1: Does the lead accept recommendations 1–3, and should the
Fast lanecheck theBrowser gate:line on frontend UI PRs intomain? (A workflow change needs the lead’s approval.) Answered 2026-11-12 (the lead): recommendations 1, 2 and 3 are accepted. TheFast lanecheck is not part of this change: the rules below are checked by the review (R-1), and a CI check stays a later decision, because it changes a workflow file and the browser gate’s--pushrefuses a selected run by design. - Q-2: What counts as a “UI change” for recommendation 1: any file under
src/featuresorsrc/ui, or the same rulepnpm e2e:gate --selectedalready uses to choose specs? Answered (this draft): the gate’s own selection. A PR whose change makespnpm e2e:gate --selectedchoose at least one spec or size-matrix state needs the run; one that selects nothing needs none (e2e/feature-map.json,scripts/browser-gate/select.mjs). - Q-3: Is the local run time acceptable on Windows, where the full gate took two to three hours
(
test-run-cost.md), given that a selected run is usually a few minutes? Answered (this draft): yes for a selected run, which on an ordinary branch also runs one size per layout band, and Chromium only on Windows (scripts/browser-gate.mjs). A change to shared code selects every spec and so takes as long as a full run; R-1 lets the developer record a skip with a reason instead, as a review skip is recorded.
The rules adopted (Draft 2, 2026-11-12)
- R-1 — A selected browser run before a frontend PR into
mainis marked ready. Inforge-et-feandforge-et-pwa, a PR intomain, a wave PR included, runspnpm e2e:gate --selectedon its final pushed head, after the test record and beforegh pr ready, when its change selects at least one spec or size-matrix state. The PR description’s test evidence carries the line the run prints, under the label Selected browser run (Selected browser run: Browser gate: passed · <sha> · … · selected: <features>), orSelected browser run: none selected, orSelected browser run: skipped — <reason>, the developer’s decision with a stated reason. A failing test is fixed before the PR is ready, or, when it is a defect outside the PR, filed as a bug and named on the line. Nothing else changes: a PR intomainstill carries noBrowser gate:line for theFast lane, needs no full gate result, and the nightly full run still runs every spec.frontend-reviewerand/forge-review-prtreat a missing or failed line on a ready PR as an Important finding, so the review record cannot beapprovewithout it. - R-2 — A selected run again when
mainhas moved in the same area. Before such a PR merges, whenmainhas gained a merge since the PR’s selected run that changes a file under the samesrc/features/<feature>folder, the samee2e/spec or the same page state, the PR mergesmainin and runs R-1 again on the new head. This is a reasongit-conventions.md § Branchingaccepts for mergingmainin (mainchanged something the PR’s tests rely on). - R-3 — The review searches the browser tests for what the diff changes.
frontend-reviewersearchese2e/and the unit tests for every user-visible string,data-testidand design token the diff changes or removes. Each hit must be updated in the diff, or the finding explains why it stays; an old value still asserted by a test is an Important finding.
Where they are written (the implementation PR changes each): .claude/CLAUDE.md § Quality Gates › Browser QA; .claude/rules/git-conventions.md § Branching and § Pull Requests; .claude/rules/test-record.md;
.claude/rules/review-record.md; .claude/agents/frontend-reviewer.md; .claude/commands/forge-review-pr.md;
.claude/commands/forge-deliver.md, .claude/lib/forge-deliver/agent-prompt-template.md,
.claude/skills/forge-pr-open/SKILL.md and .claude/skills/forge-pre-pr-review/SKILL.md (where they say a
PR into main runs no browser gate); .forge/test-strategy.md § When Tests Run;
.forge/checklists/quality-checklist.md; .forge/guides/test-runs-explained.md and the guides index line;
.forge/project-prd.md §Non-Functional Requirements › Accessibility, Supported sizes, with a PRD
revision; test-run-cost.md (a pointer to this record from S-4); and, in each frontend repo, CLAUDE.md
and the PR template. Plans and records that quote the old rule are point-in-time and stay as written.
Revisions
- Draft 1 (2026-11-11): written from the analysis made during the triage of forge-et-fe#484.
- Draft 2 (2026-11-12): the lead accepted recommendations 1, 2 and 3 (Q-1); Q-2 and Q-3 answered; the rules R-1 to R-3 and the files they change added. Status: approved by the lead, before its fresh-context review.