Visual Regression Testing for OSS: Mac-Independent Baselines Updated Only in CI

When operating visual regression tests in an OSS project, taking the screenshots was not the difficult part. The hard question was, “Whose environment produces the correct baseline?”

For an Ionic theme I maintain, baseline images updated on a Mac did not match screenshots captured on Linux in GitHub Actions exactly. Even with the same code and Chromium version, small differences appeared in areas such as text edges.

Instead of increasing the allowed difference, I decided that every baseline committed to the repository would be generated in one shared Linux container.

Detecting an intentional change as a visual diff

A real example is PR #131 for the iOS 26 theme.

This PR added elements to the demo that correctly display a Radio Group spanning multiple Inset Lists. The code change was intentional, but those elements were absent from the saved baseline, so the visual regression test correctly failed.

The previous Expected image is on the left, and the PR's Actual image is on the right.

Expected	Actual

In the Diff generated by Playwright, the added area appears in red.

Nothing is broken here; the diff is the result of intentionally changing the demo and theme. A visual regression test shows the difference, but the people reviewing the PR decide whether to adopt it as the new baseline.

When a maintainer comments /update-screenshots on the PR, CI regenerates the images on Linux and adds them to that PR in a bot commit. Tests run again on that commit and verify that both Ionic 8 and Ionic 9 match the same new baseline.

This PR initially detected visual differences in both Ionic 8 and 9. After the /update-screenshots update, both test suites passed.

Start with Playwright's toHaveScreenshot

The image comparison itself uses Playwright's toHaveScreenshot.

await page.goto(route.path, { waitUntil: 'networkidle' });
await prepareScreenShot(page, route.name);

await expect(page).toHaveScreenshot(`${route.name}.png`, {
  fullPage: true,
  animations: 'disabled',
  mask: [page.locator('ion-spinner')],
});

The tests cover more than Buttons and Inputs. All demo routes and states—including Alerts, Action Sheets, Modals, Popovers, Toasts, and Tabs—are defined in an array. Each route is captured in both light and dark mode, producing more than 90 baseline images today.

Before capturing dark mode, the test enables Ionic's class-based palette.

await page.evaluate(() => {
  document.documentElement.classList.add('ion-palette-dark');
});

await expect(page).toHaveScreenshot(`${route.name}-dark.png`, {
  fullPage: true,
  animations: 'disabled',
  mask: [page.locator('ion-spinner')],
});

Visual regression tests need to minimize anything that changes from one capture to the next. These tests keep the following conditions consistent.

• Use Chromium only
• Capture full-page screenshots
• Disable Ionic and Playwright animations
• Wait for networkidle and visible ion-content
• Mask spinners that keep moving
• Match the viewport to the content height
• Use the same Playwright container image in CI

The test code can be short; creating a stable screen is the more important part. If every capture includes changing dates, random values, network responses, or animations, the image comparison quickly stops being trustworthy.

Do not turn OS differences into a debate over whose image is correct

Playwright's official documentation also states that browser rendering varies with the OS, hardware, headless mode, and other factors, and recommends running tests in the same environment that generated the baselines. See Visual comparisons.

Running --update-snapshots on a local Mac generates baselines with Chromium on macOS. Taking those images to GitHub Actions means comparing them with Chromium on Linux.

Even when they look identical, font rendering, antialiasing, pixel rounding, and other details do not match perfectly. If fine-grained diffs appear even in unchanged regions, every PR must determine whether each difference is a real regression or merely an OS difference.

Playwright provides maxDiffPixels, maxDiffPixelRatio, and threshold.

await expect(page).toHaveScreenshot('button.png', {
  maxDiffPixelRatio: 0.01,
});

This could allow a test to pass when, for example, up to 1% of all pixels differ. A pixel count, however, cannot distinguish a small broken area at the edge of an image from a small OS-induced difference.

What I need to protect in this CSS theme includes spacing of just a few pixels, borders, and text positions. I therefore left maxDiffPixels and maxDiffPixelRatio unset instead of widening the tolerance merely to make the tests pass.

This choice is not about being strict for its own sake; it gives an OSS project a shared standard for making decisions. Instead of folding differences that occur only on one person's Mac into the allowed tolerance, comparing every PR in the same Linux environment lets contributors and maintainers inspect the same Diff.

I standardized the comparison environment instead.

jobs:
  test:
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.58.2-noble

Both PR tests and baseline updates run in the same Playwright Linux container. Pinning the container tag also limits the impact of browser updates on the runner.

Manage permission to change baselines within the PR

playwright test --update-snapshots can be run locally, but contributors no longer commit images generated on their own computers as the baseline for this OSS project.

When the baseline needs to change, a maintainer posts the following comment on a PR in the same repository.

/update-screenshots

GitHub Actions receives the issue_comment event and checks the commenter's repository permission. If they have admin, maintain, or write, the workflow adds an 👀 reaction to the comment. Updating the baseline requires authorization just as pushing code does.

on:
  issue_comment:
    types: [created]

jobs:
  pr-metadata:
    if: >-
      github.event.issue.pull_request &&
      contains(github.event.comment.body, '/update-screenshots')

The actual update runs in a read-only job inside the pinned Playwright container.

1. Check out the PR head SHA that was current when the command was posted
2. Run --update-snapshots with Ionic 9
3. Switch to Ionic 8 and run the normal tests against the same images
4. Save the updated images as an artifact

I do not maintain separate baselines for Ionic 8 and 9. Ionic 9 updates the baseline, and the tests verify that Ionic 8 produces the same appearance.

- name: Update screenshots with Ionic 9
  run: npm run test:e2e:update
  env:
    IONIC_MAJOR: 9

- name: Verify Ionic 8 against the same screenshots
  run: npm run test:e2e
  env:
    IONIC_MAJOR: 8

Keeping separate images for the two versions would make both test suites easier to pass. The guarantee I want from this theme, however, is that “the same theme version looks the same on Ionic 8 and Ionic 9.” Sharing one set of baselines makes CI enforce that compatibility.

Keep externally supplied code and write permission out of the same job

A new commit might be added to the PR while the screenshots are being generated. Images created from an old SHA must not be committed to a new head.

I therefore separated the job that generates the images from the job that commits them. The job that checks out and builds PR code has no write access to the repository.

read-only job
  check out PR SHA
  update images on Linux
  save as artifact
          │
          ▼
write job
  fetch current PR head SHA
  verify it matches the SHA at comment time
  create bot commit through GitHub API

Before committing, the workflow confirms that the PR's repository, branch, and head SHA are unchanged from when image generation began. If any value has changed, it fails and requires another /update-screenshots command.

It also validates filenames so the artifact cannot introduce unexpected files. The commit uses GitHub's Git Data API and updates the branch with force: false.

The workflow never uses permissions from the upstream repository to commit automatically to a fork PR. Automatic image updates work only for branches in the same repository. For a fork, the contributor must update images using the same Playwright Linux container, or a maintainer must bring the change onto a branch in the upstream repository. This limitation is somewhat inconvenient, but I prioritized avoiding a path that writes to a third party's branch with upstream permissions.

After a successful update, the workflow posts the following comment on the PR.

✅ Screenshots have been updated successfully!

The new screenshots have been committed to this PR.

The PR now contains the entire process: inspect the difference, update the baseline, and revalidate the result with Ionic 8 and 9.

Let contributors and maintainers inspect the same Diff on the PR

When a visual regression test fails, a long Actions log does not tell a contributor what changed. Review is not possible if only a maintainer's local environment can display the Diff.

On failure, the workflow therefore saves Playwright's Expected, Actual, Diff, and trace files as artifacts. A summary of the Ionic 9 JSON report is posted as a PR comment, and the HTML report is published to a per-PR GitHub Pages directory. That report directory is removed after the PR is closed.

CI uses a matrix for Ionic 8 and 9.

strategy:
  fail-fast: false
  matrix:
    include:
      - ionic-major: 8
        ionic-version: 8.8.19
      - ionic-major: 9
        ionic-version: 9.0.0

With fail-fast: false, one failure does not prevent the other result from being recorded. The PR can show whether the issue affects only Ionic 8, only Ionic 9, or whether both versions correctly detected the difference.

OSS baselines are decided by review, not implementation

/update-screenshots is useful, but it is not a command to run automatically whenever a test fails.

A visual regression test tells us only one fact: the result differs from before. Contributors and maintainers must inspect Expected, Actual, and Diff to decide whether that difference is a fix or a regression.

The correct order is:

1. The visual regression test detects a difference
2. A person inspects Actual and Diff
3. If the change is intentional, run /update-screenshots
4. CI updates the baseline on Linux
5. Ionic 8 and 9 are revalidated against the same images

Increasing the tolerance makes CI quieter. Quiet CI and correct visuals are not the same thing.

An OSS visual regression test must share not only its images, but also the process used to decide what is correct.

The differences between Mac and Linux were not a reason to make comparisons more permissive; they were a reason to choose one place where baselines are created. Update baselines only in CI. Share the Diff on the PR and have people review it. Keep externally supplied code separate from permission to write to the repository.

With these boundaries in place, no particular maintainer's computer defines correctness. Whoever opens a PR can inspect the same images and protect differences measured in just a few pixels.

See you next time.