Machine-Graded Visual Evals for Design Agents

Design agents are excellent at producing a first screen that looks intentional in a share link. They are much worse at surviving the second week, when a spacing regression, a missing empty state, or a mobile hierarchy break ships because nobody re-checked the same evidence twice.

Human visual QA still matters. It does not scale to every agent iteration on every branch. Machine-graded visual evals close that gap by making visual quality falsifiable: capture the page, compare it to a baseline or brief, score the differences, and fail closed when the score or the checklist says the work is not review-ready.

This article is not about chasing a single magic number. It is about a layered eval: deterministic checks first, calibrated model judgment second, human authority last. Teams that invert that order get expensive theater — an LLM praising a broken checkout because the screenshot looks "clean."

A visual eval has four jobs. Capture must be repeatable. Deterministic checks must catch the cheap failures. A judge — human or model — must interpret the rest against a rubric. A gate must turn the result into a pass or fail that CI and agents both respect.

Fail closed means the default is no. Missing screenshots fail. Unreadable baselines fail. A judge that cannot cite evidence fails. Soft warnings are allowed only for polish bands you explicitly configure. If the pipeline can be talked into green, it is not an eval; it is a mood.

Useful visual evals measure observables: presence of required regions, order of headings, contrast failures, overflow, clipped CTAs, spacing deltas against a baseline, and parity between desktop and mobile task order when the brief requires it. They do not measure "premium feel" or "modern layout."

Parity scores work when the baseline is intentional. Compare candidate screenshots to an approved reference, or compare implementation to a Figma export that the team has already accepted. Do not compare against the agent's previous attempt and call improvement a grade. Improvement without a fixed target is just motion.

Separate structural scores from aesthetic scores. Structural failures are blockers. Aesthetic disagreements are questions for a designer. Mixing them into one 0–100 "beauty" score teaches the pipeline to average away real bugs.

• Structural: required landmarks, CTA visibility, form labels, error text present.
• Layout: overflow, overlap, collapsed spacing, sticky header covering content.
• Parity: pixel or region diff vs approved baseline within tolerance.
• Accessibility: axe violations at the configured impact threshold.
• Out of scope for machine grade alone: brand taste, novel art direction, legal copy.

A design agent refactored a three-step checkout. Desktop looked fine. On a 390px capture, the sticky order summary covered the primary pay button by eight pixels. The PR description said "mobile polish." Without evals, that might have merged after a quick desktop glance.

The team's visual eval job captured /checkout/payment at 1280 and 390, ran axe, and compared region masks for summary, form, and primary CTA. The deterministic layer failed on "primary CTA fully visible." The LLM judge never ran — correctly — because fail-closed rules short-circuit on structural blockers. The agent received the finding, fixed the sticky offset, and the second run passed structural checks before a human reviewed polish findings.

The useful lesson is sequencing. If the judge had run first, it might have praised the denser summary and missed the occlusion. Cheap deterministic checks protect expensive judgment.

Unstable captures poison every later layer. Freeze fonts, disable animations, wait for network quiet or a known selector, and store images on disk rather than stuffing base64 into the model context. Name files by route, viewport, and git SHA so CI and local runs remain comparable.

// scripts/visual-capture.ts
import { chromium } from "playwright"

const viewports = [
  { name: "desktop", width: 1280, height: 800 },
  { name: "mobile", width: 390, height: 844 },
]

const routes = ["/checkout/payment"]

for (const route of routes) {
  for (const vp of viewports) {
    const browser = await chromium.launch()
    const page = await browser.newPage({
      viewport: { width: vp.width, height: vp.height },
      reducedMotion: "reduce",
    })
    await page.goto(`http://127.0.0.1:3000${route}`, {
      waitUntil: "networkidle",
    })
    await page.addStyleTag({
      content: "*, *::before, *::after { cursor: none !important; }",
    })
    await page.screenshot({
      path: `evals/captures/${sha}/${routeSlug(route)}-${vp.name}.png`,
      fullPage: true,
    })
    await browser.close()
  }
}

An uncalibrated judge agrees with whoever sounds confident. Calibration means a fixed rubric, mandatory evidence citations, a small gold set of human-labeled screenshots, and periodic accuracy checks against that gold set. If the judge disagrees with humans on blockers more than your tolerance, it does not belong in CI.

Ask the judge for findings, not vibes. Require severity, evidence (file name + region or quote from accessibility snapshot), user impact, and a smallest fix. Ban redesign proposals during eval. Ban praise. Ban scores without findings.

Use the model after deterministic checks pass or for dimensions those checks cannot see, such as whether the price summary answers the buyer's question. Never let the model override an axe critical or a missing CTA.

You are a visual eval judge for a checkout payment step.

Inputs:
- brief.md (user job + must-show elements)
- baseline screenshots
- candidate screenshots
- axe.json
- layout-diff.json

Rules:
- Cite evidence for every finding (filename + region or DOM text).
- Do not redesign.
- Do not praise.
- If evidence is missing, return status: FAIL_EVIDENCE.

Severities:
- blocker: task impossible or high-risk mistake
- important: task harder, trust or comprehension harmed
- polish: consistency only

Return JSON:
{
  "status": "pass" | "fail",
  "parityScore": 0-100,
  "findings": [{ "severity", "evidence", "issue", "impact", "fix" }]
}

Fail if any blocker exists or required elements from brief.md are absent.

A parity score is a compression of diffs, not a substitute for findings. Publish how it is computed: for example, 40% structural checklist, 30% layout diff under tolerance, 20% axe clean at serious+, 10% judge important-count penalty. Make the formula boring and versioned in the repo.

Do not hide a blocker behind a high average. A page can score 91 and still fail CI if the CTA is clipped. Scores communicate trend; gates enforce safety.

The difference between a real gate and decorative CI is visible in what can go green.

Agents should run the same command humans run. Put capture + check + judge behind npm run eval:visual and call it from AGENTS.md as a required pre-review step. In CI, fail the job on non-zero exit. Store captures as artifacts so reviewers can see what the machine saw.

For local agent loops, keep the judge optional until structural checks pass. That saves tokens and prevents the model from negotiating with itself about whether a clipped button is "still mostly visible."

{
  "scripts": {
    "eval:capture": "tsx scripts/visual-capture.ts",
    "eval:check": "tsx scripts/visual-check.ts",
    "eval:judge": "tsx scripts/visual-judge.ts",
    "eval:visual": "npm run eval:capture && npm run eval:check && npm run eval:judge"
  }
}

# GitHub Actions (sketch)
- run: npm run build && npm run start &
- run: npm run eval:visual
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: visual-evals
    path: evals/

Pixel diffs flake when fonts, anti-aliasing, or lazy media shift. Prefer region masks and structural asserts for blockers; use tight pixel tolerances only on stable chrome. LLM judges drift as models change — pin versions where you can and re-run the gold set after upgrades.

Evals do not replace accessibility with assistive technology, brand critique, or legal review. They catch repeatable visual and structural failures early. Say that out loud in your team docs so nobody treats a green badge as a ship certificate.

Cost is real. Full-page screenshots into a judge are expensive. Disk files, accessibility snapshots, and shortlist-only image crops keep the loop affordable. FinOps for critique fleets belongs beside this practice, not as an afterthought.

Use this whenever an agent changes UI that users will see. Same steps locally and in CI.

1. Freeze intent
   - brief.md with user job + must-show elements
   - approved baseline captures (or explicit first-baseline mode)

2. Capture
   - named routes × viewports
   - reduced motion, settled network, disk output

3. Deterministic checks
   - required selectors / CTA visibility
   - axe threshold
   - layout or region diff

4. Judge (only if step 3 did not fail closed)
   - rubric JSON findings with evidence
   - derive parityScore from published formula

5. Gate
   - fail on blockers, missing evidence, or score policy
   - upload artifacts

6. Fix + recapture
   - one scoped pass
   - never "fix" by loosening the gate silently