Skip to content

Fix e2e readiness races and cancelled CDP writes closing the connection - #421

Merged
tnsardesai merged 4 commits into
mainfrom
hypeship/stabilize-e2e-races
Sep 29, 2026
Merged

tnsardesai merged 4 commits into
mainfrom
hypeship/stabilize-e2e-races

Conversation

@tnsardesai

@tnsardesai tnsardesai commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

These tests fail intermittently when the CI host is under load. In each case the test either acts before the thing it depends on is ready, or reads state left by an earlier step.

  • TestContainer.WaitBrowser (new, container.go): polls /json/version through the DevTools proxy until Chromium answers. WaitReady covers only the API server and WaitDevTools only the proxy's listening port; both are ready before Chromium is. TestPlaywrightExecuteAPI, TestPlaywrightDaemonRecovery, TestPlaywrightExecuteTimeoutReturnsPromptlyAndRecovers and TestBrowserReplAPI now call it before their first browser-dependent request. Their first call goes through the Playwright daemon, which gives up after one 1s reconnect retry and returns success=false. For the timeout test, that is the likely cause of its setup ... Should be true failures. The assertion there had no message, so the daemon's error was never recorded. It now prints the response body.

  • TestBrowserReplAPI/*/chromium restart preserves repl_id and bindings: restartChromium waited on WaitDevTools, which returned in under 400ms because the proxy keeps listening while Chromium restarts. The next REPL cell's Runtime.evaluate then hit a connection that closed ("outcome is unknown because the connection closed before its response"). The helper now records the browser WebSocket URL before the restart and waits until /json/version reports a new one.

  • TestDisplayResizeChromiumWindow/headful_*, TestDisplayResizeOddWidthHonoursLibxcvtRounding: with ENABLE_WEBRTC=true, PATCH /display resizes through neko on 127.0.0.1:8080. Neko starts after the API server and Chromium, and the tests sent the resize before it was listening, which fails with "failed to call login API: ... connection refused". A new waitForNeko helper polls neko's port from inside the container, and these tests call it before the first resize.

  • TestNetworkCaptureFromWorkers, TestTelemetryConnectionOwnershipAndReconnect: the wait for page load plus interaction.js injection into a cold Chrome goes from 5s to 15s, the same as the existing interaction test in chrome_e2e_test.go.

  • assertNone (cdpmonitor test helper) and TestTabOpened/iframe_target_no_tab_opened:

    • The helper re-scanned the whole event history whenever any notify token arrived. TestTabOpened shares one collector across its subtests, so the iframe subtest failed whenever a token woke it during its 200ms window: the earlier subtest's page_tab_opened was always in history.
    • The same design also meant an event published before assertNone was called went unnoticed if its token had already been consumed.
    • assertNone now takes a checkpoint index, like waitForNew, and scans on entry, on each notify, and once at the deadline. TestTabOpened passes a checkpoint taken before the iframe attach. The other callers each use a fresh collector per subtest and pass 0, which keeps their meaning while also catching events published before the call.
  • cdpclient.Send (production change): TestInjectionOutcomeSurvivesDetach and TestLateInjectionOutcomeAfterTargetRemoval intermittently got ErrOutcomeUnknown where they expected context.Canceled. Send wrote each command with the caller's context, and coder/websocket closes the connection when a write's context ends while the frame is being written. Cancelling one command at that moment dropped the shared CDP connection and failed every other pending command on it. The same thing can happen in production to the monitor's and WebMCP's shared connections. Send now returns the caller's error without writing if the caller is already done. Otherwise it writes under the client's own context with a 10s bound, so a stuck peer still can't hang the write and Close still unblocks it. A new test, TestClientCancelledCommandKeepsConnectionOpen, sends with a cancelled context and checks the connection stays usable.

Testing

  • TestTabOpened with the old helper under CPU contention (GOMAXPROCS=2, all cores busy, -race): 8 failures in about 400 runs. With this change, 400 of 400 passed under the same conditions.
  • All assertNone callers (TestNetworkIdle, TestLayoutSettled, TestNavigationSettled, TestStopSuppressesTimers, TestTabOpened, TestBindingAndTimeline, TestPerTargetStateMachines, TestNavDataMetadata) passed 12 runs under the same contention.
  • A temporary test confirmed the helper now fails on a matching event published after the checkpoint but before the call, and ignores one from before the checkpoint.
  • cdpclient: under the same contention, TestInjectionOutcomeSurvivesDetach failed 9 of 400 runs before the fix and passed 400 of 400 after. TestLateInjectionOutcomeAfterTargetRemoval passed 150 of 150 after. The new regression test fails 6 of 20 runs against the old Send and passes 20 of 20 with the fix. It is not deterministic against the old code, because that depends on whether the websocket library closes the connection before the write finishes.
  • go test -race over all non-e2e packages passes, except lib/devtoolsproxy. That package needs a local Chromium and fails the same way without this change (a TempDir cleanup race), and it doesn't import cdpclient.
  • KERNEL_CDPMONITOR_CHROME_E2E=1 go test ./lib/cdpmonitor -run '^(TestNetworkCaptureFromWorkers|TestTelemetryConnection)' passes.
  • Against onkernel/chromium-headless:e018f7b, these pass: TestPlaywrightExecuteAPI, TestPlaywrightDaemonRecovery, TestPlaywrightExecuteTimeoutReturnsPromptlyAndRecovers and TestBrowserReplAPI/Headless. wait_browser took 240–470ms on an idle host.
  • Against onkernel/chromium-headful:e018f7b, TestDisplayResizeChromiumWindow (all four scenarios) and TestDisplayResizeOddWidthHonoursLibxcvtRounding pass.
  • Headful variants of the Playwright and REPL tests were not run locally.

Note

Medium Risk
The cdpclient.Send change affects all shared CDP connections (monitor, WebMCP); behavior is intentional but warrants careful review. E2E-only changes are low risk.

Overview
Reduces flaky CI by waiting for real browser/neko readiness before tests act, and fixes a production bug where cancelling one CDP command could tear down a shared WebSocket.

E2E readiness: Adds TestContainer.WaitBrowser (polls /json/version through the DevTools proxy) and uses it in Playwright and Browser REPL tests—API and proxy-port readiness alone is not enough for Chromium. After a supervisor Chromium restart, REPL tests now wait for a new browser WebSocket URL instead of WaitDevTools, which returns while the proxy still listens during restart. WebRTC display tests call new waitForNeko before PATCH /display so neko’s login API is up. Chrome CDP monitor e2e waits for script injection go from 5s to 15s under load; the Playwright timeout test logs the response body on setup failure.

cdpclient (production): Send no longer writes with the caller’s context (websocket could close the whole connection when that context ends mid-frame). It skips the write if the caller is already cancelled; otherwise writes under the client context with a 10s cap. New test TestClientCancelledCommandKeepsConnectionOpen locks this in.

cdpmonitor tests: assertNone takes a checkpoint index (like waitForNew) so shared event collectors and events published before the assertion window are handled correctly.

Reviewed by Cursor Bugbot for commit 81bcd0b. Bugbot is set up for automated code reviews on this repo. Configure here.

- restartChromium waits for the new browser instead of the proxy port
- wait for the browser before the first Playwright execute in the timeout test
- give page injection 15s in the worker and lifecycle Chrome tests
- isolate TestTabOpened subtests so assertNone never sees a prior event
- WaitBrowser polls /json/version through the DevTools proxy; the Playwright
  and REPL tests call it before their first browser-dependent request
- assertNone takes a checkpoint and scans on entry, so a stale event from an
  earlier step no longer fails it and one published before the call still does
- TestTabOpened goes back to a shared monitor using that checkpoint
@tnsardesai tnsardesai changed the title Fix races in e2e and cdpmonitor tests Fix readiness races in e2e and cdpmonitor tests Sep 29, 2026
With ENABLE_WEBRTC, PATCH /display resizes through neko, which starts after
the API server and Chromium. Resizing before neko listens fails with
"failed to call login API: connection refused".
coder/websocket closes the connection when a write's context ends while the
frame is being written. Send used the caller's context, so cancelling one
command could close the shared connection and fail every other pending
command with ErrOutcomeUnknown. Send now skips the write if the caller is
already done, and otherwise writes under the client context with its own
10s bound.
@tnsardesai tnsardesai changed the title Fix readiness races in e2e and cdpmonitor tests Fix e2e readiness races and cancelled CDP writes closing the connection Sep 29, 2026
@tnsardesai
tnsardesai marked this pull request as ready for review September 29, 2026 18:54
@tnsardesai
tnsardesai requested a review from Sayan- September 29, 2026 18:54
@tnsardesai
tnsardesai merged commit 6f92f2c into main Sep 29, 2026
12 checks passed
@tnsardesai
tnsardesai deleted the hypeship/stabilize-e2e-races branch September 29, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants