Visual QA and Regression Evidence

Modules 1 and 2 covered review of individual artifacts and screens: critique loops with named dimensions, and heuristic evaluations and walkthroughs at scale. This module changes the question. It is no longer is this screen good? but has anything that was already approved quietly changed? That is a different kind of review, and the only honest way to answer it is comparison against stored evidence.

Name the failure mode this module replaces: it looked fine last time I checked. Most teams' visual regression process is one designer's memory of how the product looked a few weeks ago, applied unevenly to whichever pages someone happens to open before a release. Memory does not store spacing values, breakpoint behaviour, or empty states, and it degrades exactly when the team is busiest.

Set the boundary early as well: this is a review workflow, not a licence for the agent to change the UI. The agent captures, compares, and classifies; a human reads the ranked report and approves what gets fixed and which baselines get updated. That gate structure is the same one the previous two modules established, applied to a new kind of evidence.

The examples on this slide are drawn from real sweep case studies in the school's regression-sweep workflow: a design-system version bump that loosened card padding on eleven of eighteen pages, a CMS theme update that swapped a heading font fallback and broke rhythm on three long-form pages, and a cookie banner that overlapped the primary call to action at 390 pixels on every page — found by a sweep after two days of manual checking had missed it. None of these broke a functional test, because nothing stopped working; the product simply stopped matching the design.

The deeper point is about the shape of the problem versus the shape of the usual response. Regressions caused by shared tokens, components, or templates are systemic — they appear on many pages at once, often subtly. Memory-based review is the opposite shape: one person, a handful of pages, at desktop width, on whatever day they had time. The mismatch is structural, not a matter of diligence.

This is also why the work suits agents. Capturing the same routes at the same widths, comparing them against the same baseline, and writing up the differences is repetitive, evidence-heavy work that humans do badly on the fourth iteration and an agent does identically on the fortieth. The judgment about what the differences mean stays human — that comes later in the module.

The baseline is two things at once: a folder of approved screenshots, and the manifest that says which routes, states, and viewports those screenshots represent. The screenshots come from the last approved release or, early on, from design exports. The manifest is the part that needs design judgment, because it encodes a priority call: which surfaces matter enough to protect.

Walk the three dimensions. Surfaces should start with the pages where a regression has real cost — the core task flows, anything involving money or sign-up, and the screens that exercise the design system most heavily. States are where manual review is weakest: nobody re-checks the empty state, the error state, or the expanded panel before every release, which is exactly why they belong in the baseline. Breakpoints must be fixed and reused; this school's own tooling uses 390, 768, and 1440 pixels, and the specific numbers matter less than never changing them, because comparability across runs is the entire point.

The intent line per page is the cheapest and most neglected item. Dense triage view; the queue must stay above summaries on mobile gives the compare agent something to judge differences against. Without it, every difference is just pixels. And keep the manifest in the repository so additions and removals go through review — a sweep cannot catch a regression on a page nobody listed.

Keep the discussion of this file practical. Each page entry has four parts: an id used in file names, the route, the states that matter, and the intent line. The viewports are declared once at the top so every page is captured at the same widths. The capture script loops over pages and viewports; the compare agents receive the captures plus the intent line for their page and nothing else.

The intent line deserves the most attention because it is what turns a diff into a finding. Pricing figures must stay aligned tells the compare agent that a one-pixel shift in the invoice table matters more on this page than it would on a marketing page. Without intent, the agent either reports everything — and the report becomes a diff log nobody reads — or applies its own generic judgment, which is exactly what Module 1 warned against.

States need honesty about cost. Some states are reachable from a URL; others need setup — a seeded account in a past-due state, a feature flag, a mocked error. Start with the states you can reach cheaply and add the expensive ones as the sweep proves its value. A manifest that names a state nobody can actually capture just produces a coverage gap that looks covered, which is worse than an honest omission.

The capture layer is deliberately boring: a short script that launches Chromium through Playwright, walks the manifest, sets each viewport, navigates to each route, waits for network idle, and saves a full-page PNG with the page id and viewport name in the file name. This school keeps such a script in its own repository and uses it to produce the captures referenced in the visual QA article — the pattern is not hypothetical. The agent can also drive a browser interactively over Playwright MCP or Chrome DevTools MCP, which is useful for poking at one state, but the scripted path wins for sweeps because it produces an identical evidence set every run.

Determinism is the recurring battle. Animations frozen mid-state, lazy-loaded images that never fired, sticky headers repeated through a long full-page stitch, and fonts captured before they swapped all produce differences that look like regressions and are not. Wait for network idle, disable animations where the build allows it, and treat any finding that only appears in a long full-page capture with suspicion until a viewport-sized capture confirms it.

The token caution is practical and current as of mid-2026: practitioner reports have measured a single full-page screenshot returned into a conversation as base64 consuming a six-figure token count. Save captures to disk, pass paths, and let each compare agent read only its own pair of files. That is also what makes the fan-out pattern on the next slides affordable.

Pixel diffing is the oldest tool in this space and it remains useful precisely because it is ruthless: overlay two images and any differing pixel is flagged. Its weakness is the same ruthlessness — anti-aliasing, font rendering differences between environments, and one-pixel sub-layout shifts all light up the diff, and teams that rely on pixel diffs alone spend their time approving false positives until they stop looking at the output. Structural comparison works on the accessibility tree or DOM snapshot instead of pixels: it sees that a heading level changed, a label disappeared, or content reordered, and it is far cheaper in tokens than images, but it is blind to anything purely visual.

The agent comparison layer sits on top of both. A compare agent that reads the baseline capture, the current capture, and the page's intent line can report differences in design language — the queue now sits below the summary cards on mobile, which contradicts the stated intent — rather than as coordinates of changed pixels. That is what makes the report actionable for a designer.

Be honest about the trade-off. The agent layer costs an inference call per page, and it can miss small shifts a pixel diff catches mechanically. The practical pattern is to let the cheap mechanical diffs run on everything as a first filter, and spend the agent's attention on the pages where something changed or where the intent is most sensitive. As of June 2026, dedicated visual regression services exist that handle the pixel layer well; the agent layer is what they generally lack.

Walk the board left to right and keep returning to who does what. Column one is capture: the same script, the same routes and states from the manifest, the same three widths, producing a baseline set and a current set. Column two is the fan-out: one compare agent per page, reading only its own pair of captures and its intent line, reporting observable differences with the evidence file named on every finding. Keeping each agent narrow keeps it honest — an agent looking at one page does not average its judgment across the whole product, and forty pages of raw findings never flood one context window.

Column three is the classification this module turns on: every difference is proposed as a regression, an intended change, or noise, with evidence attached. The agent proposes the classification; it does not get to act on it. Column four is the human review — and the design goal of the entire pipeline is visible in what reaches that column: ranked blockers with their evidence, systemic findings grouped to one likely cause, baseline updates awaiting approval, and judgment calls. Noise never reaches the room.

If participants take one thing from the diagram, it should be the filtering. The value of the sweep is not that it finds more differences than a human would — it is that the human only spends attention on the differences that need a decision, each one arriving with its evidence already attached.

This classification is what separates a regression sweep from a diff dump. A raw comparison tool reports that 212 things changed; the report a designer can act on says which of those are regressions to fix, which are the design moving forward and therefore baseline updates, and which are artifacts of the capture itself. Each bucket has a different response, and mixing them up has a different cost in each direction: treating an intended change as a regression wastes a fix cycle reverting good work, treating a regression as intended quietly ratchets the product away from its design, and chasing noise erodes the team's trust in the whole report.

Within the regression bucket, the same P0 to P3 severity rubric from the earlier modules applies, assigned by user impact rather than by how easy the fix looks. P0 blocks the release — a save action below the fold at phone width. P1 changes hierarchy or breaks a key responsive state. P2 weakens polish or consistency and gets logged and batched. P3 is a judgment call and goes to a human, never to an automatic fix.

The most valuable signal in a merged report is repetition. When eleven pages report the same loosened card padding, the cause is almost certainly one shared token or component, and one fix clears eleven findings. A good report groups repeated findings by likely shared cause rather than listing them per page — that is the difference between a report that reads in five minutes and one that reads like a log.

This is the governance problem every visual regression practice eventually hits, regardless of tooling. If the baseline never updates, every sweep re-reports the same known differences, the report gets longer and less read, and the one genuine regression hides in the pile. If the baseline updates automatically whenever a sweep completes — or whenever a build merges — the tool stops measuring drift and starts ratifying it: whatever shipped becomes, by definition, correct.

The resolution is to treat baseline updates exactly like the intended-change bucket from the previous slide: each one is an explicit approval, made by someone accountable for the product's look, ideally with a one-line note about which design decision it reflects. Per-page or per-finding approval matters because bulk approve all is where discipline goes to die — it feels efficient and it converts every unnoticed regression into the new standard in one click.

There is also a useful diagnostic in the update history. A page whose baseline needs updating every sweep is telling you its design is still in motion; that page belongs in the critique loop from Module 1, not in the regression set, until it settles. And when an intended change is systemic — a new spacing scale, a typography refresh — update the affected baselines in one reviewed batch tied to that decision, rather than letting them trickle through as individual approvals over weeks.

The sweep trades depth per page for coverage across the product, so it earns its keep when the change is broad and the risk is diffuse: release branches that bundle weeks of merged work, design-system version bumps, framework migrations, theme updates, and refactors that touch shared layout components. Those are the moments when a single underlying change can move dozens of pages at once and no individual code review sees the whole effect.

The inverse matters just as much. A single new feature page does not need a sweep — it has no baseline yet, and what it needs is the deeper single-screen review from Module 1: critique against named dimensions, with the heuristic and walkthrough methods from Module 2 where the flow is task-critical. The sweep protects what is already approved; the earlier methods judge what is new. Teams that confuse the two either run sweeps that report nothing useful on new work, or skip critique because the sweep passed.

On cadence: a full sweep is in the range of thirty to sixty minutes, most of it agent time, once the manifest and capture script exist. That is cheap enough to run before every release and on a recurring schedule as a health check. The recurring run matters because drift accumulates between releases too — dependency bumps, content changes, and small fixes all move pixels nobody is watching. Where a per-pull-request check fits into this picture is the subject of Module 5.

Walk the table as a story rather than a set of numbers, and be clear about the source: this case study comes from the school's published regression-sweep workflow, describing a product team's run, not a controlled benchmark. The setting is the worst case for memory-based review — nine weeks of merged work, fourteen pages, two days of runway — exactly the situation where someone clicks through a few pages, sees nothing alarming at desktop width, and signs off.

The two P0s repay attention because of why they were invisible. The save button below an overflowing panel at 390 pixels is a responsive-state failure: it only exists at phone width, on one settings page, in one panel state. The white-on-white chart legend came from a theme variable rename — the chart still rendered, the data was still correct, every functional test passed, and the legend was simply unreadable. Neither is the kind of thing a code reviewer sees in a diff.

The outcome is the part to emphasise for sceptics. The team did not slip the release. Blockers and the four systemic P1s were fixed in one afternoon, the affected pages were recaptured to prove the fixes, and the release shipped on schedule. The sixteen P2s did not get fixed that week — they became a polish backlog with named owners, which is a far better fate than a vague feeling that the release looked rough. The sweep's output was a set of decisions, not just a longer to-do list.

The exercise is deliberately paper-only, and the constraint to five to eight routes is the point: an unconstrained list grows until it covers everything and protects nothing in particular, because nobody maintains it. Choosing what to leave out is the design judgment the sweep cannot make for you.

The intent lines are where most participants slow down, and that slowdown is the lesson. Writing dense triage view; the queue must stay above summaries on mobile forces you to articulate what the page is actually for — which is the same articulation a critique brief needs, and the reason this exercise feeds forward rather than being busywork. The state column usually surfaces an honest gap: the states that matter most, like error and past-due, are often the hardest to reach without seeded data, and a manifest should record that cost rather than pretend it away.

The last bullet is the self-test. Asking which recent change this set would have caught makes the value concrete; asking which it would have missed keeps the limits honest — a sweep cannot protect a page that is not listed, cannot judge a brand-new page with no baseline, and cannot prove accessibility from screenshots alone. If running this live, compare manifests in pairs: two people covering the same product area almost never choose the same routes, and the differences are a fast way to surface what each person believes matters most.

Recap by connecting the bullets back to the single idea: the question has anything approved quietly changed? can only be answered by comparison against stored evidence, and agents make that comparison cheap enough to run at every release rather than as an occasional heroic effort. The human contribution is concentrated at three points — choosing what the baseline protects, classifying what the differences mean, and approving what changes, including the baseline itself.

Restate the boundaries so the module does not oversell. The sweep cannot judge new pages with no baseline, cannot prove accessibility from screenshots alone, cannot detect issues on pages missing from the manifest, and cannot decide taste questions — it can only put them in front of a person with evidence attached. Those limits are also the bridge points: critique and evaluation handle the new work, and the next module handles the checks screenshots cannot carry.

Preview Module 4 concretely. It covers the two reviews teams defer longest: accessibility review that goes beyond automated contrast checks into structure, keyboard flow, and screen-reader sense, and content review that holds interface copy to a written voice standard. Both reuse this module's discipline — evidence per finding, severity by user impact, and a clear split between what an agent can fix and what needs design or legal judgment. Participants should bring their exercise manifest forward; the same routes are a natural starting set for the accessibility pass.