source: .forge/design/harness/load-check-and-flaky-tests.md · generated — edit the markdown, not this file

Test runs on a loaded machine: a load check, and how flaky-test bugs are handled — Design

Status: approved (Draft 2) Author: Tharinda Rajapaksha Reviewed by: one fresh-context review pass, 2026-10-15 (Draft 1 “Ready after Importants”; Draft 2 applies every finding — see ## Revisions). Approval: Tharinda Rajapaksha (lead), 2026-10-15 Date: 2026-10-15 · approved 2026-10-15 Scope: the delivery harness and the three app repos.

  • Harness: .forge/test-strategy.md § Flaky Tests (and a revision entry), .claude/rules/tracker.md § Bug Tracking (the paragraph on flaky failures filed per operating system), .claude/rules/git-conventions.md § Pull Requests (the scripts’ refusals, and who may pass --ignore-load), .forge/team-guide.md § The Team, and a revision to the F40 spec (.forge/specs/F40-accessibility-responsive-gate-spec.md, FR-7: the gate’s refusals and its result fields).
  • forge-et-be: scripts/local-build-evidence.py (the build record) and CLAUDE.md § Common commands.
  • forge-et-fe: scripts/browser-gate.mjs (its options and usage text), scripts/browser-gate/run.mjs (the browser gate’s run command), a new measuring module under scripts/browser-gate/, scripts/local-test-evidence.mjs (the test record), their tests under src/test/scripts/, and CLAUDE.md § Common commands.
  • forge-et-pwa: the same files as byte copies of forge-et-fe’s (its scripts/shared-with-fe.json), and its CLAUDE.md.

No CI workflow changes: the browser gate’s run command and the two record scripts run only on a developer’s machine. The Browser gate: line and the Test record status keep their format, and the gate’s covers and ci-check commands do not change. Nothing here is cloud-gated, so .forge/cloud-gated-validation.md gains no row.

Context

CI does not run the test suites on an ordinary pull request (.claude/rules/git-conventions.md § Pull Requests). The developer runs them on their own laptop — the backend build record (python3 scripts/local-build-evidence.py), the frontends’ test record (pnpm test:record) and the browser gate (pnpm e2e:gate) — and a pull request merges on that local result. The team works on macOS and Windows (.forge/team-guide.md § The Team), and a flaky failure is filed once per operating system (.claude/rules/tracker.md § Bug Tracking).

A full browser gate already uses the whole machine on its own: BUG-151’s run, at Playwright’s default 4 workers on an 8-core Mac with nothing else running, showed a load average of 13 to 15. These laptops are often running much more than that: several Claude sessions work in the workspace at once, each may have a backend, an isolated compose stack, Testcontainers or a browser gate running, and on macOS the Docker VM’s CPU counts against the host. Two bugs filed on 2026-10-14 show the result:

  • BUG-194 (macOS): 12 WebKit size-matrix states ran 1.8 to 9 minutes each against a 30-second test timeout, at a load average of 20 to 30 on 8 cores. Alone, the same 83 states passed in 1.5 minutes.
  • BUG-195 (macOS): a 60-second PollingConditions wait ended after 539 seconds and 6 polls. The test’s own thread barely ran. Alone, the spec passed 14 of 14.

A test that runs many times longer than its own timeout was not slow; its process was not getting the CPU. Nothing in the gate or the records notices this, and nothing records how loaded the machine was, so each such failure is filed and investigated as if it might be a defect. BUG-149 was closed wontfix on 2026-10-12 with the instruction that a recurrence be filed “with the gate’s load”, which no tool reports today.

Filing every flaky failure is still worth it. Failures that looked like load noise have had real causes. F11-f’s finding F-001 was a product defect that users on slow machines would have hit: typed text lost in a combobox (.forge/lessons.md L-162). BUG-110 was a react-aria behaviour the tests had to respect (L-155). BUG-150 and BUG-151 were timing faults in the tests. BUG-077, BUG-081 and BUG-082 were tests that measured real elapsed time against a fixed bound (L-125). And because the local result is the only test gate, a team that re-runs failures until they pass will eventually re-run a real failure away. So the bug entry stays. What changes is that overload is prevented before a run, recorded during it, and recognised when a bug is triaged.

Terms

  • CPU busy share: the fraction of all cores’ time that was not idle over a sample, from 0 to 1. Measured over five seconds: Node from two os.cpus() readings; Python with GetSystemTimes through ctypes on Windows and /proc/stat on Linux, and on macOS as Q-2 decides. It reacts within seconds, so it is what the check before a run uses.
  • Load average: on macOS and Linux, the 1-minute load average divided by the number of logical cores (Node os.loadavg()[0] / os.availableParallelism(), Python os.getloadavg()[0] / os.cpu_count()). Above 1.0, more work is waiting to run than the machine has cores. It is an average over about a minute, so it lags behind the machine’s real state and keeps reading high for a minute after work stops. Windows has none (Node returns zeros; Python raises). It is recorded during a run (D-3), because unlike the busy share it shows how far demand exceeds the cores, but the check before a run does not use it.
  • A stall: a test that ran for more than three times its own timeout, or a polling wait that polled fewer than a third of the times its interval allows. A stall means the test’s process was not scheduled, not that a page or a query was slow.
  • Forced load: busy loops started beside the test, one per logical core (yes > /dev/null on macOS and Linux, a PowerShell busy loop on Windows). It is used to reproduce a load-dependent failure on purpose, in every engine, including WebKit, which Chrome’s CDP throttle cannot reach.

Decisions

# Decision Why
D-1 The browser gate’s run and both record scripts check the CPU busy share before they start work, and refuse when it is above the limit. The limit is 0.65 (Q-1, decided 2026-10-15; Draft 1 and 2 said 0.5). The limit is one named constant in each script, which ET_LOAD_LIMIT (a number between 0 and 1) overrides. The refusal uses each tool’s existing refusal path and names the busy share, the core count and the five processes using the most CPU now (ps -Ao pid,%cpu,comm -r on macOS and Linux; Get-Counter '\Process(*)\% Processor Time' on Windows, since Get-Process’s CPU value is total seconds since each process started). Where the check runs: in the gate, after its existing refusals (run.mjs step 4) and before the previous result is deleted, so a refusal leaves the last result.json in place for covers; in both record scripts, only on the path that runs the build or the tests. There is no check on the path that carries an earlier record forward, on --self-test, or once a script has already refused. A full gate alone saturates the machine (BUG-151’s load), so any substantial other work oversubscribes it. Refusing costs a few seconds and names what to stop. The busy share responds within seconds; the load average would refuse a run started right after another one finished.
D-2 --ignore-load runs anyway, and the run’s result says it started over the limit. There is no automatic wait and no automatic change to the number of workers. Claude passes --ignore-load only when the developer says so in that session (added to git-conventions.md § Pull Requests). A person may know the load is about to drop, or accept the risk. An agent that is refused would otherwise add the flag and repeat the problem the check exists to prevent.
D-3 The load is recorded during the run. Every 30 seconds the tools sample the CPU busy share, the load average where it exists, and the five busiest processes. They keep the starting value, the highest and the average of each, and the process list at the highest sample. The browser gate writes them to .browser-gate/result.json as a load object, passed or failed, with ignored: true when --ignore-load was used; it also lists any test that ran more than three times its timeout, from Playwright’s JSON report, as stalls. covers and ci-check do not read either. The test record and the build record add one line to their PR comment: Machine load: busy 0.35 at start, 1.00 peak; load average 1.9 per core peak (8 cores, macOS), with no load average on Windows and started over the limit (--ignore-load) when that applies. A failed run prints the same line and the process list to the terminal. A failure can then be triaged from what was recorded, which BUG-149 asked for, and the recorded values show whether the limit in D-1 is right (Q-1).
D-4 A flaky failure is triaged in this order (written into .forge/test-strategy.md § Flaky Tests): (1) Read the failed run’s record: was it a stall, did it start over the limit, and which processes were busiest at the peak? (2) Reproduce it on a machine otherwise within the limit: under forced load in every engine, with the CDP CPU throttle as well for Chromium, and, as L-162 says, by holding back the one step that may land late (a route loader, a held response) when neither reproduces it. (3) If any of these reproduces it, it is a timing defect in the product or the test, fixed as one, as today. (4) If none does, and the record shows the machine was overloaded by work other than the run — it started with --ignore-load, or the busiest processes at the peak include another gate, build, backend or container — it is closed wontfix because the machine was overloaded. The Resolved: line gives the recorded load, the busiest processes and the reproduction attempts, and names the lead as the person who decided, as BUG-149’s does; the bug gets a Lesson: line as tracker step 6 requires. A system-wide measure cannot separate the run’s own share of the load from other work’s, so the process list at the peak is the evidence of other work. If the record shows no other work, the failure stays open as a defect to find. A recurrence on a machine within the limit is filed as a new bug that links the first. This is an exception to the policy’s deletion rule (“if a flake cannot be fixed quickly, delete the test and record why”): an overload-only failure is not a flake in the test, so the test is kept, and the load check is what stops the failure recurring. The exception needs the reproduction attempts and the record of other work, so it cannot be used to close a real defect.
D-5 Flaky failures are still filed per operating system, and one cause is investigated once. A failure with the same symptom as an open bug on the other operating system is filed as its own bug whose spec_ref names the first, and its notes say it waits on the first bug’s cause. Once that cause is known, the second bug proves the fix, or the D-4 outcome, on its own operating system. Each fix still needs proof on the kind of machine where it failed, but two people investigating the same cause in parallel is wasted work (BUG-129 on Windows and BUG-194 on macOS are likely one stall).
D-6 Unchanged: zero retries locally; a sleep or a longer timeout is not a fix; no quarantining; a bug entry for every flaky failure, with the operating system and what else was running in its Repro: line, now also with the run’s recorded load. These rules caught real defects (§ Context) and do not depend on the load.
D-7 The open flaky bugs are triaged under D-4 once the load check is on main, by their owners: BUG-128 and BUG-129 (Windows, Nanndish), BUG-194 and BUG-195 (macOS, Tharinda). The triage replaces the “give the worker a budget that names why” option in the Fix: lines of BUG-129, BUG-194 and BUG-195, which D-6 does not allow; each Fix: line is updated when its bug is triaged. Fixed and wontfix bugs are not reopened. They were filed without a recorded load, and D-4 needs one.

Rejected alternatives

  • Wait until the load drops instead of refusing. A wait can last indefinitely and hides why a run has not started.
  • Cut the number of workers automatically. It lengthens a full gate a great deal without saying so. ET_E2E_WORKERS stays the manual control it is today.
  • A machine-wide lock so only one gate or build runs at a time. It would stop parallel sessions from overloading the machine together, but it blocks a light run behind a heavy one and needs stale-lock handling on both operating systems. Rejected for now; see the accepted gap below.
  • Refuse on low free memory. Free memory is hard to read reliably on macOS, where memory the system can reclaim is not counted as free (Q-3).

Accepted gap: two sessions that start a gate or build within seconds of each other both pass the check and then overload the machine together. The recorded load (D-3) shows this when it happens; if it shows up often, the lock above is reconsidered.

Requirements

forge-et-fe (and its byte copies in forge-et-pwa)

  • FE-1. A module under scripts/browser-gate/ (with its .d.mts file) that measures the CPU busy share, the load average where it exists and the five busiest processes, on macOS, Linux and Windows (§ Terms, D-1). It imports only Node built-ins, like every other module there, so the gate’s commands still run before pnpm install.
  • FE-2. run checks the busy share after its step 4 refusal and before it deletes the previous result (D-1). --ignore-load (D-2) is added to the run options and the usage text in scripts/browser-gate.mjs. The sampler (D-3) starts with Playwright and stops when it exits; its values and the stalls list go into result.json.
  • FE-3. local-test-evidence.mjs does the same around pnpm test, only on the path that runs the tests: the check before, the sampler during, the line in the comment.
  • FE-4. Unit tests with the measurement and the process list injected: a refusal above the limit on each operating system; a run at the limit; ET_LOAD_LIMIT; --ignore-load; the previous result.json unchanged after a refusal; no check on the carried-record path; the recorded values and stalls in result.json and the line in the comment; a failed run printing the line.
  • FE-5. Each new file is added to forge-et-pwa’s scripts/shared-with-fe.json, the files are copied there, and the hashes are recorded (node scripts/check-shared-drift.mjs --update --from=<forge-et-fe commit>).

forge-et-be

  • BE-1. local-build-evidence.py checks the busy share before ./gradlew build, only on the path that runs the build (D-1); accepts --ignore-load (D-2) and ET_LOAD_LIMIT; samples during the build in a background thread (D-3); and adds the line to its comment. Python standard library only, as the script is today.
  • BE-2. --self-test gains the same cases as FE-4 that apply to it.

The harness

  • H-1. .forge/test-strategy.md § Flaky Tests: the D-4 triage order, the D-4 wontfix outcome stated as an exception to the deletion rule, and cause 3 (machine load) pointing at the load check; a revision entry, because the policy was accepted at Rev 5.
  • H-2. .claude/rules/tracker.md § Bug Tracking: the per-operating-system paragraph gains D-5’s “waits on the first bug’s cause”, and a flaky bug’s Repro: line carries the run’s recorded load.
  • H-3. .claude/rules/git-conventions.md § Pull Requests: the record scripts’ refusals include the load check, and Claude passes --ignore-load only when the developer says so (D-2).
  • H-4. .forge/team-guide.md § The Team: one sentence that the gate and the records refuse to start on an overloaded machine, and what to do (stop other runs, or --ignore-load).
  • H-5. A revision to the F40 spec: FR-7’s list of the gate’s refusals gains the load check, and its result fields gain load and stalls.
  • H-6. A row in each app repo’s CLAUDE.md § Common commands for --ignore-load and ET_LOAD_LIMIT.

Where each rule lives

Rule File
The check, its limit and --ignore-load the three scripts (FE-2, FE-3, BE-1); described in git-conventions.md § Pull Requests and F40 FR-7
Who may pass --ignore-load git-conventions.md § Pull Requests
The triage order and the wontfix exception test-strategy.md § Flaky Tests
Per-operating-system filing and twin bugs tracker.md § Bug Tracking

Rollout

  1. This design is approved.
  2. forge-et-fe first (FE-1 to FE-4), then forge-et-pwa (FE-5), then forge-et-be (BE-1, BE-2): each a normal app-repo PR with its review record.
  3. The harness changes (H-1 to H-6) in one harness PR, merged once the app-repo PRs have merged, so no rule describes a check that does not exist yet.
  4. D-7: the open flaky bugs are triaged.
  5. After two weeks of recorded loads, Q-1’s limit is reviewed.

Acceptance checks

  • AC-1. On a machine within the limit, pnpm e2e:gate, pnpm test:record and the build record start and finish as today, and result.json and the record comment carry the load.
  • AC-2. With one busy loop per core running for at least 10 seconds before the command, each of the three refuses to start, names the busy share, the cores and the busy loops among the busiest processes, and records nothing; the previous result.json is unchanged.
  • AC-3. With the same busy loops, --ignore-load runs, and the record says it started over the limit.
  • AC-4. A record script whose run carries an earlier record forward does not check the load, even with the busy loops running.
  • AC-5. AC-1 to AC-4 are run on macOS and on Windows.
  • AC-6. BUG-194 and BUG-195 are triaged under D-4 with a recorded load, and each is closed fixed or wontfix with its evidence.

Open questions

  • Q-1. The limit — decided 2026-10-15: 0.65 (Tharinda, lead). The first guess, 0.5, refused ordinary working days. The readings of 2026-10-15, as the CPU busy share over the check’s five seconds, were:

    • quiet: 0.02 to 0.06;
    • a normal working day (the Claude app, editors, idle containers): 0.24 to 0.36;
    • the same with another session’s backend test worker: 0.39 and 0.53, two samples 20 seconds apart, so one sample swings about 0.15;
    • another session’s browser gate: 0.74, 0.97 and 1.00.

    0.65 leaves about 0.12 above the highest normal reading and still refuses a concurrent gate. ET_LOAD_LIMIT sets a machine’s own limit. Change it again only on recorded evidence: refusals on a machine doing ordinary work, or failures whose record shows a start just under the limit.

  • Q-2. The CPU busy share in Python on macOS. The standard library has no system CPU times there. The options are the Mach host_statistics call through ctypes, or summing ps -A -o %cpu; the plan picks whichever reads correctly on the team’s Macs.

  • Q-3. Memory. BUG-129 also saw free memory fall to about 2 GB. This design refuses on CPU only (§ Rejected alternatives). Add memory if a recorded failure shows it was the cause.

Revisions

  • Draft 1 (2026-10-15): first draft.
  • Draft 2 (2026-10-15): applies the fresh-context review of Draft 1 (“Ready after Importants”). Corrected § Context: BUG-151 was a test timing fault, and the product defect was F11-f’s F-001 (L-162); a full gate alone reaches a load average of 13 to 15 on 8 cores (BUG-151). The check before a run now uses the CPU busy share, because the load average lags by about a minute; the load average is recorded during the run only. The Windows process list now uses Get-Counter, and Python on Windows uses GetSystemTimes. The check sits before the previous result is deleted, and never on a carried-record path. D-4 defines the wontfix case by the record of other work, states it as an exception to the deletion rule, and adds the CDP throttle (Chromium only) and holding back a late step to the reproduction. Claude may pass --ignore-load only when the developer says so. Added: the stalls list, the result format, ET_LOAD_LIMIT, who decides wontfix and the Lesson: line, the bugs’ owners, the F40 FR-7 and git-conventions.md requirements, the pwa shared-with-fe.json entries, and the sections Rejected alternatives, Where each rule lives and Rollout.
  • After approval (2026-10-15): Q-1 decided by Tharinda (lead): the default limit is 0.65, not 0.5, from the day’s recorded readings (§ Open questions, Q-1); D-1 updated. Built in forge-et-fe#279, forge-et-pwa#81 and forge-et-be#264. Q-2 was settled in forge-et-be#261: Python reads macOS CPU time through Mach host_statistics in ctypes.
  • Revised by test-run-cost.md (2026-10-27): D-6’s “zero retries locally” no longer holds for the browser gate, which retries a failed test once and counts a pass on retry as flaky, failing above five (that design’s F-1, the lead’s answer to its Q-2), nor, since 2026-10-27, for the two records (its F-3); a developer’s own runs still retry nothing. The rejected machine-wide lock is adopted there as M-5 (its Q-7) and was built on 2026-10-27: a second heavy run on one machine waits for the first, because the accepted gap of two runs starting together is what the records showed. The load check still runs once a run holds the machine.

Generated from .forge/design/harness/load-check-and-flaky-tests.md by .claude/lib/render-docs-html.js. Markdown is canonical.