Stabilising Flaky Extension Tests

Extension test suites are flakier than web-app suites for structural reasons: every test depends on a service worker that starts asynchronously and can be evicted, on storage that persists across page loads within a profile, and often on real third-party pages that change under you. A retry makes a flaky test green without making it right — and some of the flakes are real bugs that users hit too. Fixing the cause is almost always cheaper than living with the retries. This guide is part of end-to-end testing automation.

Where extension flakiness comes from

Step-by-step

1. Measure the flake rate before changing anything

1npx playwright test --repeat-each=20 --workers=4 --retries=0 --reporter=json > flake.json
2jq '[.suites[].specs[] | {title, fails: ([.tests[].results[] | select(.status != "passed")] | length)}]
3    | map(select(.fails > 0)) | sort_by(-.fails)' flake.json

Execution context: your shell or a dedicated CI job. Running each test twenty times with retries off turns “occasionally red” into a number per test. Fix the worst first, and re-run the same command to prove the fix.

2. Wait for the worker, and for its readiness

The extension’s worker registering is not the same as it being ready. If initialisation reads storage asynchronously, a test that sends a message immediately can arrive before the handler’s dependencies are loaded.

1// Worker: publish readiness once initialisation completes.
2const ready = init().then(() => { globalThis.__ready = true; });
3
4// Test: wait for it explicitly.
5await sw.evaluate(() => new Promise((r) => {
6  const check = () => (globalThis.__ready ? r() : setTimeout(check, 20));
7  check();
8}));

Execution context: the service worker in a test build, and the test’s evaluate into it. If this wait makes a flake disappear, the product has the same race for real users — the fix belongs in the worker, as described in messages sent while the worker is starting. The readiness flag is compiled out of release builds like any test hook.

3. Replace sleeps with conditions

1// Flaky: assumes the sync finishes within 500 ms on every machine.
2await page.waitForTimeout(500);
3expect(await badgeText()).toBe("3");
4
5// Stable: waits for the condition, however long it takes, up to a limit.
6await expect.poll(() => sw.evaluate(() => chrome.action.getBadgeText({})), { timeout: 10_000 }).toBe("3");

Execution context: a Playwright test. expect.poll retries the probe until it matches or the timeout expires, so a fast machine finishes quickly and a slow CI runner still passes. Every waitForTimeout in an extension suite is a flake waiting for a busy runner.

4. Isolate state between tests

Storage, alarms, registrations and dynamic rules all persist within a browser profile. A test that leaves an alarm behind changes the next test’s world.

1test.beforeEach(async ({ sw }) => {
2  await sw.evaluate(async () => {
3    await chrome.storage.local.clear();
4    await chrome.storage.session.clear();
5    await chrome.alarms.clearAll();
6    const rules = await chrome.declarativeNetRequest.getDynamicRules();
7    await chrome.declarativeNetRequest.updateDynamicRules({ removeRuleIds: rules.map((r) => r.id) });
8  });
9});

Execution context: the service worker, before each test. Clearing every persistent surface the extension uses makes tests order-independent. Where a whole fresh profile per test is affordable, it is the stronger guarantee; where it is not, an explicit reset is the next best thing.

5. Stop depending on the internet

A test that loads a real news site fails when the site changes its markup, shows a cookie banner, or is slow. Serve fixtures instead.

1test.beforeEach(async ({ context }) => {
2  await context.route("https://news.example.com/**", (route) =>
3    route.fulfill({ path: "e2e/fixtures/article.html", contentType: "text/html" }));
4});

Execution context: Playwright’s network layer. Content scripts match on the URL, so they still inject into the fixture as if it were the real site — the test keeps its realism about matching and loses only the network’s unpredictability. Keep one or two smoke tests against real sites in a separate, non-blocking job to catch genuine site changes.

6. Make eviction deterministic

Chrome may evict the worker mid-test on a slow runner, turning a test that assumes a warm worker into a flake. Either test cold starts deliberately, or keep the worker warm for tests that are not about the lifecycle.

1// Deliberately exercise eviction where it matters:
2await stopWorker(context, sw);
3// …then the action that should survive it…
4
5// Or, for unrelated tests, keep it warm with an open port:
6const keep = await page.evaluate(() => { globalThis.__p = chrome.runtime.connect({ name: "test-keepalive" }); });

Execution context: Node and a page in the test. The stop helper is described in driving service worker state from a test. An open port keeps the worker alive, removing eviction as a variable from tests about something else — never use the keep-alive in the product itself.

Retries, quarantine and keeping the suite trusted

A suite people do not trust is worse than a smaller one they do. Two policies protect that trust.

Allow one retry in CI, and report it. A single retry absorbs genuine infrastructure noise — a runner hiccup, a slow disk. But a test that passes only on retry is still flaky, and the report should say so. Playwright marks these as “flaky” in its output; surface that count in the pipeline summary rather than letting it disappear into a green tick.

Quarantine rather than delete. A test that flakes above a threshold and cannot be fixed immediately moves to a quarantined project that runs but does not block merges, with an owner and a date. That keeps the signal without blocking everyone.

1// playwright.config.js
2export default {
3  retries: process.env.CI ? 1 : 0,
4  projects: [
5    { name: "main", grepInvert: /@quarantine/ },
6    { name: "quarantine", grep: /@quarantine/, retries: 0 },
7  ],
8};

Execution context: the Playwright configuration. Tagging a test @quarantine in its title moves it; the CI workflow runs both projects but only fails on main. The pipeline side is covered in running extension tests in headless CI.

Cross-browser variation

• Chrome / Edge: worker startup and eviction are the dominant flake sources; CDP gives you control over both, so they can be made deterministic.
• Firefox: the event-page background is evicted less aggressively, so eviction flakes are rarer — and tests may pass on Firefox that fail on Chrome for a real lifecycle reason.
• Safari: manual or safaridriver-based runs are the least deterministic; keep Safari automation to a small smoke set.
• All three: state persistence within a profile is the same everywhere. Explicit resets make a suite portable between engines.

Verification

1. Re-run the repeat command after each fix and confirm the fail count for that test drops to zero across 20 runs.
2. Shuffle test order and confirm the suite still passes — order dependence is state bleed:

1npx playwright test --repeat-each=5 --workers=1 --shard=1/1 --reporter=list -- --fully-parallel

Execution context: your shell. Running with full parallelism across a single worker set exposes tests that relied on a previous test’s leftovers.

3. Grep the suite for waitForTimeout and confirm none remain outside deliberately justified cases.
4. Confirm the pipeline summary shows the flaky-on-retry count.

FAQ

Is one retry acceptable?

In CI, yes — as long as flaky-on-retry results are reported. Locally, run with zero retries so flakes are visible while you work.

The flake only happens in CI. How do I reproduce it locally?

CI runners are slower and more contended. Throttle the CPU with CDP (Emulation.setCPUThrottlingRate) and run with several workers; most CI-only flakes are timing assumptions that surface under load.

Should every test get a fresh browser profile?

It is the strongest isolation and the slowest. A fresh profile per file plus explicit resets per test is usually the right balance.

Related

• Driving service worker state from a test — controlling the worker deliberately.
• Running extension tests in headless CI — the pipeline these run in.
• Registering listeners at the top level — the product bug behind many startup flakes.
• End-to-end testing automation — the parent guide.