Heuristic Evaluation and Cognitive Walkthroughs at Scale

Module 1 built the critique loop: named dimensions, the agent as first reviewer, findings with evidence, and the human deciding what matters. This module takes that loop and points it at two formal inspection methods that pre-date all of this by decades — heuristic evaluation and the cognitive walkthrough. They were never discredited; they were rationed. Most teams run them once, on the flow everyone already worries about, and call the rest of the product covered.

The argument of the module is narrow and worth stating up front: agents change the cost of running these methods properly, and nothing else. The heuristics are still Nielsen's plus whatever the team has committed to in writing. The four walkthrough questions are still the four questions. The agents supply patient, consistent application across every flow and every task-persona combination — the part humans do unevenly by the fifth flow of the afternoon — and the humans still confirm severity and decide what gets fixed.

Flag the structure: heuristic evaluation first, walkthroughs second, then the parts they share — evidence rules, triage, and the human validation pass — a worked example, and an exercise small enough to run this week.

Keep the refresher conservative. Heuristic evaluation, as defined by Nielsen and Molich, is a small group of evaluators inspecting an interface against a short list of usability principles and recording violations. It is cheap, it needs no participants, and it reliably surfaces a worthwhile share of real problems — particularly the procedural ones: inconsistent patterns, missing feedback, errors with no recovery path. It does not measure how often a problem occurs or how much it costs, and it cannot stand in for testing with real users. Those limits are part of the method, not a flaw agents introduce.

The practical weakness has always been labour. Walking every step of every flow, holding each screen against each heuristic, and writing findings precise enough that someone else can locate the problem is patient, repetitive work. A two-person team facing seven core flows either spends a week or evaluates the three flows everyone already worries about. That rationing — not any doubt about the method — is why most products carry years of unexamined usability debt.

If the audience includes people who have never run one by hand, it is worth saying that doing one screen manually at least once is the fastest way to develop a feel for what a good finding looks like. The agent-scaled version is the same activity with the tedium removed, and you review it better if you have felt the tedium.

The scaling move is structural, not clever prompting. Each key flow gets its own evaluator agent, and every agent works from the same three inputs: the heuristics file, a folder of ordered screenshots for its flow, and access to the live or staging build for the states the screenshots missed — errors, empty states, what happens after a wrong input. Because each flow gets a fresh agent, the seventh flow is evaluated with the same attention as the first, which is precisely where human evaluators degrade.

The heuristics file deserves the most discussion time. Nielsen's ten are the stable base; the project heuristics are the rules that are true for this product but not products in general — a banking app's rule that the balance is never hidden behind promotional content, a B2B tool's rule that admin actions are reversible. Each needs an ID, a one-sentence definition, and one example of a violation, because every finding will reference an ID and the merge stage dedupes partly by it. Keep the project list to five to eight; a long list dilutes everything.

The other discipline is evidence. Capture the screenshots before the run with a small script, name them by flow and step, and require every finding to cite a filename or a described action in the build. The merge agent then combines all per-flow findings, dedupes problems that recur across flows into one finding with multiple locations, and ranks by candidate severity and frequency. The word candidate is doing real work there — the agent proposes, the human confirms, and that division is the design of the workflow rather than a disclaimer bolted on.

Walk the diagram left to right and name who owns each column. The two left cards are human work and they are where the quality is decided: choosing which flows or tasks count as the product surface, and writing the shared contract — the heuristics file for an evaluation, or the task list, personas, and four questions for a walkthrough, plus the evidence rules and the candidate severity scale that both methods use.

The middle two columns are agent-run. The fan-out is one evaluator per flow, or one walkthrough per task-and-persona cell, each returning evidence-backed candidate findings. The merge agent is what makes the output usable: without it you have several hundred scattered notes; with it you have recurring problems collapsed into single findings with multiple locations, grouped, ranked, and accompanied by a list of what was checked and found clean — which matters, because an evaluation that only reports problems gives no sense of coverage.

The right side is the human gate and the outputs. The severity pass is a sample-and-confirm activity, not a re-run of the evaluation; the next slides cover how to do it without it becoming a second audit. The confirmed set then feeds one of three things — a fix backlog with owners, a redesign brief, or an archived baseline the next run is compared against. If the team cannot say which of those three the run feeds, the run is not ready to start.

The cognitive walkthrough is the second classic method, and it answers a different question to heuristic evaluation. Heuristics ask whether the interface follows good principles; the walkthrough asks whether a specific person, with a specific level of knowledge, could learn their way through a specific task. It is the method for first-run experiences, onboarding, admin consoles where new staff must self-serve — anywhere learnability is the question.

The discipline is the four questions, asked at every step, in order, without skipping the ones that feel obvious. Each answer is a yes or no with a one-sentence justification grounded in what is visible on the screen, plus a screenshot reference. Any no makes the step a failure point with a severity guess: blocks the task, causes hesitation or error, or minor friction. Write the questions into a per-step template the agents must follow, so every step of every task gets the same treatment and the merge stage can compare like with like.

State the caveat plainly and early, because it is the one this method most needs: the agent is simulating an evaluator applying an inspection method, not simulating a user's mind. The verdicts are predictions under the method. The way to confirm the ones that matter is usability testing with real participants — and the walkthrough's most useful output is often a sharper test plan, because it tells you exactly which screens and steps to watch.

Two human-authored inputs decide the quality of a walkthrough run, and neither can be delegated. The first is the task list. Each task needs a starting point and the correct action sequence — the answer key the walkthrough checks the interface against. Writing that sequence is itself diagnostic: if the team cannot agree what the correct sequence is, or cannot describe a task in a sentence, that is a finding before any agent runs. Choose tasks where learnability matters: the first-run path, the things new staff must do unaided, the tasks that generate support tickets.

The second input is the personas, and for a walkthrough a persona is almost entirely a knowledge profile: what they know about the domain, what they know about this product, what conventions they can be assumed to recognise, and what they have already done — installed the CLI, received the invite email, hold an account. Resist demographic decoration; the four questions never ask how old the persona is. Two or three personas is usually enough, and more multiplies cost faster than insight, because the run size is the product of tasks and personas.

The payoff of running multiple personas is differential findings. A step that fails for the novice and passes for the expert is precisely located as a learnability gap rather than a flaw in the flow itself, which changes the fix — often a label or one line of explanation rather than a redesign. Keep the task list and persona files in the repository and reuse them; rerunning the same grid after each significant release turns the walkthrough into a cheap learnability regression check.

This table is the quality bar for both methods, and it is the same bar Module 1 set for critique findings: criterion, evidence, location, candidate severity, scoped suggestion. The difference at scale is volume — a seven-flow evaluation or a twenty-seven-cell walkthrough produces hundreds of candidate findings, and the only way the team can review that many honestly is if every one of them is checkable without re-running the work. That is what the evidence rule buys.

Walk a couple of the contrasts. "The send-money flow is confusing" is an opinion; "NN-09, send-money step 4, screenshot 04-transfer-failed.png: a failed transfer shows error code TRX-409 with no explanation or retry path" is a finding someone can open, verify, and fix. "The app is inconsistent" is a vibe; "destructive actions use three different confirmation patterns, recurring in four flows, locations listed" is one deduplicated finding with a scoped fix. Severity stated as fact is the most dangerous weak pattern, because agents will assign severity with unearned confidence if the prompt allows it — which is why the scale is agreed in advance, every rating is labelled candidate, and confirmation is structurally a human step.

Confidence is worth a sentence too: agents over-report near-duplicates and occasionally stretch a heuristic to cover an aesthetic preference. Requiring the criterion ID makes the stretch visible — if the finding does not genuinely fit the heuristic it cites, it is either a judgment to be labelled as such or a guess to be rejected at the severity pass.

Scale creates a new failure mode the manual versions of these methods never had: the team receives hundreds of candidate findings, feels obliged to process all of them, and either burns a sprint on severity-1 friction or shelves the whole report. Triage is the answer, and most of it is mechanical enough for the merge agent to do before any human reads a line.

The sequence: dedupe, group, rank. Deduplication collapses the same unlabelled icon button found in four flows into one finding with four locations — and frequency then becomes part of the ranking signal. Grouping by heuristic or by screen is what surfaces the patterns worth more than any individual finding: nineteen consistency findings traceable to three competing button patterns is a consolidation workstream, not nineteen tickets; twenty-two failure points clustered on one permissions screen is one redesign, not nine task-level fixes. Ranking by candidate severity then frequency tells the human reviewer where to spend the confirmation effort.

Two more habits keep the queue workable. First, the report should list what was checked and found clean — heuristics with zero findings, tasks that passed for every persona — because coverage is half the value and it is invisible if only violations are reported. Second, decide before the run what the output feeds: a fix backlog with owners, a redesign brief, a usability test plan, or an archived baseline for the next comparable run. Findings without a destination become a document nobody opens twice.

The severity pass is where the module's recurring theme — agents propose, humans decide — becomes a concrete working session, and it is worth being precise about its size. For a seven-flow evaluation the agent run itself typically finishes well inside the hour; budget most of the one-to-two-hour total for this pass. It is not a re-run of the evaluation. The reviewer opens the evidence behind every high-severity candidate, samples the middle and low bands, confirms or adjusts severities, marks duplicates the merge missed, and rejects findings whose evidence does not support the claim when the screenshot is actually opened.

The rejection rate deserves emphasis because teams misread it. Rejecting some findings is not the workflow failing — it is the review doing its job, and it is information about how the agents are misfiring: too many near-duplicates suggests the merge instructions need tightening, heuristics stretched to cover aesthetic preferences suggests the heuristics file needs sharper definitions or the evaluator agent needs a firmer rule. A pass that confirms every candidate has not reviewed anything; it has rubber-stamped the agent's judgment and attached a human name to it.

The pass is also where accountability transfers. Once the design lead has confirmed severities, the report stops being agent output and becomes the team's assessment — which is exactly what you want before it reaches a backlog, an executive, or a client. Archive the confirmed set together with the heuristics file and the evidence folder, so the next run is compared against this one rather than against memory.

This run is documented in the heuristic-evaluation workflow on the school's site; treat the numbers as one traced run, not a benchmark. The team was about to start a major redesign and wanted a baseline of usability debt before any new design work began — one of the classic moments to reach for the method, alongside before/after comparisons of the same flow and an agency's structured first read of a prospect's product.

Walk the funnel: 96 candidate findings from the fan-out, 61 after the merge deduplicated problems recurring across flows, 54 confirmed at the severity pass with 7 rejected because the evidence did not support the claim. The confirmed set included three severity-4 task blockers — two in the send-money flow, including fees that appeared only on the receipt screen in violation of a project heuristic, and a failed transfer that showed a raw error code with no recovery path — plus an onboarding step that silently timed out and discarded entered data.

The most useful output was not the blockers, which the team half-knew about. It was the heuristic-level pattern: nineteen of the fifty-four confirmed findings referenced consistency, and almost all traced back to three different button and confirmation patterns that had accumulated over four years. That cluster gave the redesign brief a consolidation workstream it would not otherwise have had — and it is exactly the kind of cross-flow pattern a manual evaluation of three flows would never reveal. The archived run became the baseline the redesigned flows were later measured against, which is what makes a one-off audit compound over time.

Close the methods section with the limits, because the credibility of everything else depends on stating them unprompted. Heuristic evaluation and cognitive walkthroughs are inspection methods: they predict problems from principles and structured questions. They cannot tell you how often a problem occurs, how much it costs, or whether real users actually stumble where the method says they should. Usability testing and analytics are how predictions get confirmed, and the workflows are explicit that the walkthrough in particular pairs naturally with a usability test booked before the report is written.

The agent-specific failure modes are worth naming because the severity pass is designed around them: over-reporting near-duplicates, stretching a heuristic to cover an aesthetic preference, and assigning severity with unearned confidence when the prompt does not forbid it. None of these is fatal; all of them are why severity is structurally a human decision and why the rejection rate at the gate is information.

The last bullet is the one that protects teams politically. Which flows count as key, whether a project heuristic reflects a genuine commitment or an aspiration, and what gets shown to a client or an executive are judgment calls the evaluation cannot make. The agency case in the workflow is instructive: the page that explained the method, what was and was not evaluated, and whose judgment the severities reflect did more work in the pitch than the findings did. Honesty about limits is not hedging — it is what makes the rest of the report believable.

The exercise deliberately stays manual. Everything produced here — the task list, the persona, the project heuristics, the destination decision — is the human-authored contract from the left side of the diagram, and it transfers directly into a scaled run when the participant is ready. Steer people towards tasks that are real but bounded: things new users must do unaided, things that generate support tickets, not redesign our app.

The one-screen manual walkthrough is the most important step. Answering the four questions by hand for a single screen takes about ten minutes and does two things: it surfaces how much discipline the questions actually demand — most people discover they want to skip Q1 because it feels obvious — and it calibrates the participant for reviewing agent output later. Someone who has produced one careful per-step record can tell at a glance whether an agent's record is grounded in the screen or generated from vibes.

The project heuristics step is where the exercise connects back to Module 1's named dimensions: the rules that are true for this product specifically, written down with an ID and an example so they can be cited and checked. Common discoveries are that the team has strong opinions it has never written anywhere, and that some "rules" turn out to be aspirations the product violates everywhere — both useful findings before any agent runs. Ask participants to keep the page; it becomes the inputs file for the first scaled evaluation, and Module 5 will assume those criteria exist when wiring review into pull requests.

Recap by walking the pipeline rather than the bullets in isolation: the human-authored contract on the left, the fan-out and merge in the middle, the severity gate on the right, and a named destination for the confirmed findings. The two methods differ in what they ask — principles per screen versus four questions per task step — but they share the same shape, the same evidence rules, and the same division of labour, which is why one diagram covered both.

If participants did the exercise, the page they produced is the left-hand column of that diagram for their own product. The natural next step is to run the smallest scaled version: three flows or three tasks against one persona, with the severity pass done properly, before scaling to the whole surface. The workflows referenced through this module — heuristic evaluation at scale and cognitive walkthrough at scale — contain the prompts, agent definitions, and traced case studies to copy from.

Preview Module 3 concretely. This module evaluated the product against principles and tasks; the next one evaluates it against itself over time. Screenshot baselines for the surfaces that matter, agent-run capture sweeps across states and breakpoints, and diff reports that distinguish genuine regressions from intended change — so the answer to "did anything break visually" stops depending on whoever last looked at the screen and starts depending on evidence.