source: .forge/design/harness/fewer-nightly-reds.md · generated — edit the markdown, not this file

Fewer red nights: catch broken browser tests before the merge — Design

Status: approved by the lead (Draft 2, 2026-11-12): recommendations 1–3 are adopted as R-1 to R-3 (§ The rules adopted). In force when the implementation PR, reviewed by a fresh-context reviewer, merges (.claude/CLAUDE.md § Reference Map, harness design records). Author: Tharinda Rajapaksha, from the triage of forge-et-fe#484 (2026-11-10) Date: 2026-11-11 Scope, if accepted: .claude/CLAUDE.md § Quality Gates (the browser-QA paragraph), .claude/rules/git-conventions.md § Pull Requests, .claude/rules/test-record.md, .claude/agents/frontend-reviewer.md, .forge/test-strategy.md § When Tests Run, and forge-et-fe’s PR template and CLAUDE.md. Nothing is cloud-gated, and no AWS cost changes. Recommendation 1 adds local run time to a frontend UI PR; it adds no GitHub Actions minutes.

Problem

Since PRD Rev 81 (2026-11-02), a PR into main runs no browser gate, and the nightly full run on the lead’s Mac is the only run of the browser suites (.claude/CLAUDE.md § Quality Gates, Browser QA). A red night stops new work until it is fixed (git-conventions.md § Pull Requests), so every cause the nightly run finds after a merge costs the team a morning. The question is which of those causes could have been caught before the merge, and at what cost.

Evidence (as read on 2026-11-10 and 2026-11-11)

Sources: the nightly log (the harness issue labelled nightly-log), the nightly-red issues in the three app repos, .forge/bugs.md and .forge/lessons.md. Counts are point-in-time.

# Cause Examples Could a pre-merge step have caught it?
C-1 The nightly machine itself: the Mac slept or its lid was closed, a gate’s port was held, a config key or file was missing, a suite hit its time limit 2026-10-28 (every suite at its limit), 2026-11-02 (the night’s Compose ports file was missing; then a 31-minute sleep), 2026-11-11 (both browser gates not run: the Mac slept for 177 minutes) No. It is not a code defect. About half of the red or missed nights in the log are this kind
C-2 A change merged without running the browser tests it affects forge-et-fe#442 (feature/ui-rev100, 709 files) updated the specs for its changed strings but ran none of them, as its PR body says; the first failures of forge-et-fe#484 (BUG-390–393, BUG-397) start at its merge Yes: a selected browser run on the PR
C-3 Two PRs that each pass, merged one after the other, break each other forge-et-fe#470 (F32-d, the Country list became a multi-select) and #472 (tests written against the old single-select), BUG-398; BUG-324 (browser specs against AD #41’s keys); BUG-326 (a spec importing a helper another PR removed) Partly: a selected run again when main has moved in the same area since the PR’s last run
C-4 Flaky tests 84 bugs titled Flaky in bugs.md on 2026-11-11, 77 of which mention machine load and 63 Windows; lessons L-101 (wait for the page to settle, not for an element), L-125 (no assertion on what a time-bounded run got done), L-162 (reproduce under a throttled CPU before calling a test flaky) Partly: the lessons are known but not enforced by any check
C-5 A size or accessibility failure that only the size matrix finds the size-matrix and WebKit failures in forge-et-fe#484 (pricing-import-create-unknown at 768 and 1024 wide) Yes, when the selected run includes the size projects for the changed screens

Recommendations

Recommended now (they target C-2 and C-3, the causes a pre-merge step can catch):

  1. A selected browser run before a frontend UI PR is marked ready. A forge-et-fe or forge-et-pwa PR that changes a screen, a string, a test id or a token runs pnpm e2e:gate --selected on its final head before gh pr ready, and its PR body carries the Browser gate: line (skipped — <reason> allowed, as for release/*). Whether the Fast lane checks the line, as it does for release/*, is a decision for the lead, because it changes a workflow file.
  2. A selected run again when main has moved in the same area. When main has gained a merge that touches the same feature folder or the same e2e/ specs since the PR’s last browser run, the PR merges main in and runs the selected gate again. This is one of the reasons git-conventions.md § Branching already accepts for merging main in.
  3. A frontend-reviewer check that every string, test id and token the diff changes or removes is searched for in e2e/, and that each hit is either updated in the diff or explained.

Considered for later:

  1. Split a UI change the size of forge-et-fe#442 into PRs per screen area, each with its own selected run.
  2. Lint rules for L-101 and L-125 in e2e/ (no waitFor on an element where the lesson asks for a settled page; no assertion on a count reached within a time limit).
  3. Move the nightly full run to a machine that does not sleep, which would remove most of C-1. This needs a price and the lead’s decision.

Open questions

  • Q-1: Does the lead accept recommendations 1–3, and should the Fast lane check the Browser gate: line on frontend UI PRs into main? (A workflow change needs the lead’s approval.) Answered 2026-11-12 (the lead): recommendations 1, 2 and 3 are accepted. The Fast lane check is not part of this change: the rules below are checked by the review (R-1), and a CI check stays a later decision, because it changes a workflow file and the browser gate’s --push refuses a selected run by design.
  • Q-2: What counts as a “UI change” for recommendation 1: any file under src/features or src/ui, or the same rule pnpm e2e:gate --selected already uses to choose specs? Answered (this draft): the gate’s own selection. A PR whose change makes pnpm e2e:gate --selected choose at least one spec or size-matrix state needs the run; one that selects nothing needs none (e2e/feature-map.json, scripts/browser-gate/select.mjs).
  • Q-3: Is the local run time acceptable on Windows, where the full gate took two to three hours (test-run-cost.md), given that a selected run is usually a few minutes? Answered (this draft): yes for a selected run, which on an ordinary branch also runs one size per layout band, and Chromium only on Windows (scripts/browser-gate.mjs). A change to shared code selects every spec and so takes as long as a full run; R-1 lets the developer record a skip with a reason instead, as a review skip is recorded.

The rules adopted (Draft 2, 2026-11-12)

  • R-1 — A selected browser run before a frontend PR into main is marked ready. In forge-et-fe and forge-et-pwa, a PR into main, a wave PR included, runs pnpm e2e:gate --selected on its final pushed head, after the test record and before gh pr ready, when its change selects at least one spec or size-matrix state. The PR description’s test evidence carries the line the run prints, under the label Selected browser run (Selected browser run: Browser gate: passed · <sha> · … · selected: <features>), or Selected browser run: none selected, or Selected browser run: skipped — <reason>, the developer’s decision with a stated reason. A failing test is fixed before the PR is ready, or, when it is a defect outside the PR, filed as a bug and named on the line. Nothing else changes: a PR into main still carries no Browser gate: line for the Fast lane, needs no full gate result, and the nightly full run still runs every spec. frontend-reviewer and /forge-review-pr treat a missing or failed line on a ready PR as an Important finding, so the review record cannot be approve without it.
  • R-2 — A selected run again when main has moved in the same area. Before such a PR merges, when main has gained a merge since the PR’s selected run that changes a file under the same src/features/<feature> folder, the same e2e/ spec or the same page state, the PR merges main in and runs R-1 again on the new head. This is a reason git-conventions.md § Branching accepts for merging main in (main changed something the PR’s tests rely on).
  • R-3 — The review searches the browser tests for what the diff changes. frontend-reviewer searches e2e/ and the unit tests for every user-visible string, data-testid and design token the diff changes or removes. Each hit must be updated in the diff, or the finding explains why it stays; an old value still asserted by a test is an Important finding.

Where they are written (the implementation PR changes each): .claude/CLAUDE.md § Quality Gates › Browser QA; .claude/rules/git-conventions.md § Branching and § Pull Requests; .claude/rules/test-record.md; .claude/rules/review-record.md; .claude/agents/frontend-reviewer.md; .claude/commands/forge-review-pr.md; .claude/commands/forge-deliver.md, .claude/lib/forge-deliver/agent-prompt-template.md, .claude/skills/forge-pr-open/SKILL.md and .claude/skills/forge-pre-pr-review/SKILL.md (where they say a PR into main runs no browser gate); .forge/test-strategy.md § When Tests Run; .forge/checklists/quality-checklist.md; .forge/guides/test-runs-explained.md and the guides index line; .forge/project-prd.md §Non-Functional Requirements › Accessibility, Supported sizes, with a PRD revision; test-run-cost.md (a pointer to this record from S-4); and, in each frontend repo, CLAUDE.md and the PR template. Plans and records that quote the old rule are point-in-time and stay as written.

Revisions

  • Draft 1 (2026-11-11): written from the analysis made during the triage of forge-et-fe#484.
  • Draft 2 (2026-11-12): the lead accepted recommendations 1, 2 and 3 (Q-1); Q-2 and Q-3 answered; the rules R-1 to R-3 and the files they change added. Status: approved by the lead, before its fresh-context review.

Generated from .forge/design/harness/fewer-nightly-reds.md by .claude/lib/render-docs-html.js. Markdown is canonical.