Running Extension Tests in Headless CI

End-to-end tests that pass on a laptop fail on a CI runner for a reason that has nothing to do with the tests: the browser there has no display, and for years headless Chrome could not load extensions at all. That has changed — Chrome’s new headless mode runs the full browser, extensions included — but the configuration is still easy to get wrong, and the failure mode is a test that times out waiting for a service worker that was never started. This guide is part of CI and release automation.

Headless modes and extensions

Step-by-step

1. Launch Chromium with the extension, in new headless mode

 1// e2e/fixtures.js
 2import { test as base, chromium } from "@playwright/test";
 3import path from "node:path";
 4
 5export const test = base.extend({
 6  context: async ({}, use) => {
 7    const ext = path.resolve("dist/chrome");
 8    const context = await chromium.launchPersistentContext("", {
 9      channel: "chromium",                 // bundled Chromium supports new headless with extensions
10      headless: true,
11      args: [`--disable-extensions-except=${ext}`, `--load-extension=${ext}`],
12    });
13    await use(context);
14    await context.close();
15  },
16  extensionId: async ({ context }, use) => {
17    let [sw] = context.serviceWorkers();
18    if (!sw) sw = await context.waitForEvent("serviceworker");
19    await use(new URL(sw.url()).host);
20  },
21});

Execution context: Node, under Playwright. Extensions require a persistent context — launchPersistentContext with an empty user-data dir creates a fresh temporary profile. Using Playwright’s bundled Chromium rather than a branded Chrome channel matters: branded Chrome builds have at times refused --load-extension in some configurations, while the bundled Chromium consistently supports it. The fixture itself is expanded in loading an unpacked extension in Playwright.

2. Wait for the service worker, not a timeout

1test("worker starts", async ({ context }) => {
2  const sw = context.serviceWorkers()[0] ?? await context.waitForEvent("serviceworker", { timeout: 15_000 });
3  expect(sw.url()).toMatch(/^chrome-extension:\/\/[a-p]{32}\/service-worker\.js$/);
4});

Execution context: a Playwright test. If this times out, the extension never loaded — most often because the path in --load-extension does not point at a directory containing a valid manifest.json, or because the runner’s browser is running in a mode that silently ignores the flag. It is worth a dedicated first test, so every other failure can assume the extension is present.

3. Build once, test the artifact

 1# .github/workflows/ci.yml (excerpt)
 2  e2e:
 3    needs: build
 4    runs-on: ubuntu-latest
 5    steps:
 6      - uses: actions/checkout@v4
 7      - uses: actions/setup-node@v4
 8        with: { node-version: 20, cache: npm }
 9      - run: npm ci
10      - uses: actions/download-artifact@v4
11        with: { name: extension-chrome, path: dist/chrome }
12      - name: Cache Playwright browsers
13        uses: actions/cache@v4
14        with:
15          path: ~/.cache/ms-playwright
16          key: pw-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
17      - run: npx playwright install --with-deps chromium
18      - run: npx playwright test

Execution context: GitHub Actions. Testing the downloaded build artifact — rather than rebuilding — means the bytes tested are the bytes that will be uploaded. Caching the Playwright browsers keyed on the lockfile saves a minute or more per run.

4. Fall back to xvfb when something needs a display

A few extension features behave differently without a display even in new headless mode — screen capture, some clipboard paths, GPU-dependent canvas work.

1      - run: sudo apt-get install -y xvfb
2      - run: xvfb-run --auto-servernum npx playwright test --project=headed

Execution context: the CI runner. A separate Playwright project with headless: false runs only the tests tagged as needing a display, under a virtual framebuffer. Keeping those few tests in their own project stops them slowing down the main suite.

5. Keep failures diagnosable

 1// playwright.config.js
 2export default {
 3  retries: process.env.CI ? 1 : 0,
 4  use: {
 5    trace: "retain-on-failure",
 6    screenshot: "only-on-failure",
 7    video: "retain-on-failure",
 8  },
 9  reporter: [["list"], ["html", { open: "never" }]],
10};

1      - if: failure()
2        uses: actions/upload-artifact@v4
3        with: { name: playwright-report, path: playwright-report }

Execution context: the Playwright config and the CI workflow. A trace captures DOM snapshots, network and console for every step, which is usually enough to diagnose an extension failure without re-running. One retry in CI absorbs genuine infrastructure blips; more retries hide real flakiness, which is covered in stabilising flaky extension tests.

Making CI runs reflect real conditions

A CI runner is a very particular environment: a fresh profile every time, no other extensions, fast local disk, often more CPU than a user’s laptop. Tests that pass there can still miss problems that appear on real machines, and a few adjustments narrow the gap.

Exercise cold starts deliberately. On a fresh profile the worker starts once and stays warm for the whole suite, so tests never meet the eviction behaviour users do. Add a helper that stops the worker between steps where it matters, as in driving service worker state from a test.

Throttle where timing matters. Chrome DevTools Protocol lets a test slow the CPU; a 4× throttle on the tests that check popup open time or content-script overhead catches regressions a fast runner would hide.

Test an upgrade, not just an install. A fresh-profile suite only ever exercises reason: "install". One test that loads the previous release, populates storage, then loads the new build and asserts the migration ran is the most valuable addition to most extension suites — the process in testing an update before you publish it.

Cross-browser variation

• Chrome / Edge: new headless mode (headless: true with bundled Chromium in current Playwright) loads extensions. Branded Chrome and Edge channels may differ; prefer the bundled build in CI.
• Firefox: Playwright’s Firefox does not load WebExtensions through a launch flag in the same way; use web-ext run --target=firefox-desktop with --browser-console or a Selenium/geckodriver setup with a temporary add-on. web-ext supports headless via MOZ_HEADLESS=1.
• Safari: no headless mode; automation requires macOS runners with safaridriver and the extension enabled in Safari’s settings, which cannot be fully automated. Most teams run Safari tests manually or on a dedicated macOS machine.
• All three: CI profiles are fresh on every run. Nothing persists between jobs unless you explicitly cache or seed it.

Verification

1. Run the suite locally with CI=1 and headless: true before pushing; the first test should confirm the service worker appeared.
2. Break the extension path deliberately and confirm the job fails at the worker gate with a clear message, not thirty minutes later.
3. Check timing on the runner:

1npx playwright test --reporter=list 2>&1 | tail -3
2#   24 passed (1.8m)

Execution context: the CI log. Compare against local timings; a runner that is dramatically slower is often missing the browser cache.

4. Force a failure and confirm the trace artifact is uploaded and opens in npx playwright show-trace.

FAQ

Why does headless: true ignore my extension?

Either the browser build does not support extensions in headless mode, or --load-extension points at the wrong directory. Use Playwright’s bundled Chromium and log the resolved path.

Do I need xvfb at all?

Usually not with new headless mode. Keep it for the few tests that genuinely need a display, in their own project.

Can the popup be tested in headless mode?

The popup surface itself cannot be opened by automation, but its HTML can be loaded as a page at chrome-extension://<id>/popup.html, which covers nearly everything — as described in testing a popup and options page with Playwright.

Related

• Loading an unpacked extension in Playwright — the launch fixture in detail.
• Building a GitHub Actions pipeline for extensions — the pipeline this job sits in.
• Stabilising flaky extension tests — when retries are not the answer.
• CI and release automation — the parent guide.