Surveys, Experiments, and Funnel Diagnosis

This module moves the course from qualitative to quantitative evidence, and the framing has to be set in the first minute: agents make quantitative work faster, but they do not make it more rigorous. Speed and rigour are different properties. An agent will happily draft a leading survey question, estimate a percentage it never computed, or describe a correlational drop-off as a cause — quickly and fluently. The discipline in this module is about preventing exactly that.

The three workflows behind the module are the school's survey design and analysis workflow, the experiment design and results readout workflow, and the funnel and drop-off diagnosis workflow. They share one rule that recurs on almost every slide: numbers come from code the agent writes and runs against an export, never from the model's recollection or estimate. The model is good at writing the script and bad at being the calculator.

Set expectations on depth too. This is not a statistics course. Where the work needs significance testing, weighting, or contested methodological calls, the honest answer is to involve someone who knows the methods. The module teaches designers and researchers to use agents for the parts they can own — instrument critique, scripted analysis, evidence-linked hypotheses — and to recognise the boundary where statistical expertise takes over.

This slide establishes the division of labour the rest of the module assumes. On the agent side: drafting survey questions mapped to constructs, critiquing drafts against named bias patterns, computing sample sizes with a script, writing and running analysis scripts against exports, coding open-text answers against a codebook, and attacking draft readouts as a skeptic. All of that is coverage and computation, and agents are genuinely good at it.

On the human side: the research question and the decision it informs, the ethics of what gets asked and tested, the launch decision, the interpretation of caveats, and the final call. None of that transfers. The middle ground — significance tests, weighting, margin-of-error claims, choosing between analytical framings — is not agent territory and not necessarily designer territory either; it belongs to someone who knows the methods, and the honest move is to say so in the report rather than improvise.

The highlight line is the operational rule that makes the rest enforceable. Models smooth numbers when allowed to estimate: 38 per cent becomes roughly 40 per cent, a cell of 51 becomes about 50. The fix is structural, not motivational — the workflow only reports what its scripts printed, and the human spot-checks a few numbers by re-running the script.

Surveys fail in two places, and most teams only watch the second. The first failure happens before launch: a leading question that smuggles the desired answer into the wording, a double-barreled item that asks two things and allows one answer, a scale with no honest middle or no not-applicable escape, or an ordering effect where an early question primes everything after it. Once people have answered a flawed question, the data is flawed permanently — there is no analysis pass that recovers it.

The discipline that prevents most of this is mundane: before drafting a single question, write down the research questions and the constructs they imply. A question like why are trial users not converting decomposes into constructs — perceived value, price sensitivity, setup blockers, comparison with alternatives — and each construct earns one or two survey items. Anything that does not map to a construct is a question someone wanted to ask, not a question the study needs. Surveys that skip this step tend to grow to forty questions, and long surveys buy worse data twice: through drop-off and through the fatigue of whoever finishes.

The bias patterns named on the slide are standard survey methodology, not agent-specific. The next slide is where the agent earns its place: as a critic that checks the draft against these patterns systematically, before anyone answers it.

The survey workflow's first stage is a critique loop. A critic subagent — defined once and reused across studies — reads the draft the way a hostile methodologist would: it checks every question for leading or loaded wording, double-barreled construction, missing or unbalanced answer options, scale problems such as unlabelled endpoints or stacked agree/disagree items, and ordering effects. It also checks every question against the construct map and flags anything unmapped. Each finding comes back with the offending wording quoted verbatim, the bias named, and a rewrite proposed. The loop runs until the critic returns only judgment calls, and those go to the researcher rather than back into the loop — the researcher owns the revision.

A second cheap pass before launch is the persona read-aloud: an agent answers the survey as a handful of named personas — a rushed respondent on a phone, a non-native English speaker, a power user, someone who hit the exact problem the survey is about — and narrates where each one hesitates, misreads, or cannot find an honest answer among the options. This is a simulation, not a substitute for piloting with real people, and the report should say so. But it reliably catches the question that makes sense to the team and to nobody else.

The caveat on the slide matters: the critique loop reduces wording bias, it does not certify the instrument. When the stakes justify it, a human methodologist reviewing the final draft is still the stronger gate, and a pilot with five real respondents is always worth the hour it costs.

The analysis stage of the survey workflow has one inviolable rule: every number in the report was computed by a script the workflow wrote and ran against the export. The model never estimates a number, because the difference between 38 per cent and roughly 40 per cent is exactly the kind of thing it will smooth over if allowed to. For most product surveys the scripts are not statistics packages — counting, percentages with their bases, and cross-tabs cover the bulk of the work, and a plain script keeps everything inspectable. When a stakeholder later asks where a number came from, the answer is a file and a command, not a conversation.

The structure is a fan-out: each segment defined in the brief — plan tier, role, tenure — gets its own analysis agent running the same script with a different filter, and a separate agent codes the open-text answers against a codebook with the same discipline as interview coding from Module 3: verbatim excerpts, response IDs, counts per code. Open text is where conclusions change; in the workflow's pricing case study, 64 per cent of closed-question answers looked like acceptance, but coding 814 free-text answers showed much of that acceptance was explicitly conditional on a discount or a missing feature.

The last gate before the report is a challenge pass: an agent that reads every draft claim and checks it against sample size per cell, response rate, and who never saw the invitation. Read the challenge file before the findings file, every time. Then do the human spot-check: re-run a few numbers with the same script and arguments, and check a few quoted excerpts against the raw export.

Walk the loop and name the owner of each step. Framing the question is human: the decision at stake, the constructs or metrics, the segments, and the ethics of asking or testing at all. Design support is agent-run: drafting the instrument or the experiment plan, critiquing it against bias patterns, computing the sample size with a script, and assembling a pre-registration for a human to sign. Data collection is owned by the team and its real tooling — the agent does not launch surveys and does not touch production traffic; it designs and reads out. The analysis is agent-run but scripted: every number from code, bases and segments and intervals reported, assumptions stated, and a skeptic or challenge pass attacking the draft before any human reads it.

The readout review gate is the human step most teams skip when they are in a hurry, and it is where laundering gets caught: spot-check numbers against the export, compare what the readout claims against what the pre-registration said would be reported, and strip anything the sample cannot carry. Only then does the decision happen — and the decision is recorded separately from the evidence, so next year nobody has to reconstruct whether the data made the call or the team did.

The dashed line matters for how research functions in a team: the decision raises the next question, which is how this connects back to the research packet structure from Module 1 and forward to the insight-to-brief work in Module 6.

Experiment programs rarely fail at running tests; they fail at being honest about them. The failure modes have names: peeking — checking significance daily and stopping on a good day; HARKing — hypothesising after the results are known; segment fishing — slicing until something clears the bar; and novelty effects mistaken for durable lift. None of these require bad intent, only a deadline and an unguarded readout.

The design stage forces the discipline to exist before launch. The hypothesis must carry a because clause that points at real evidence — a coded ticket theme, a usability finding, a drop-off the funnel work surfaced. There is one primary metric and a small set of guardrails with their harm directions stated, because guardrails are where redesigns quietly do damage and also where unexpected wins hide. The minimum detectable effect and the sample size come from a script with its inputs written down; if the script says nine weeks of traffic and the team has three, that is a design conversation now, not a surprise after an inconclusive readout. All of it goes into a one-page pre-registration that a human signs.

After the run reaches its pre-registered length, the analysis agent computes per-arm results, intervals, guardrails, and only the pre-registered segments — with code. Then a skeptic agent receives the draft, the pre-registration, and the script outputs, and is prompted to assume the conclusion is wrong: it checks the stopping rule, hypothesis drift, the number of segments examined versus pre-registered, novelty and seasonality in the trend, guardrail damage, and practical significance. Its objections are published in the readout with their dispositions, whether or not they change the conclusion. The readout's final section — what the team decides — is left for the humans.

Laundering is the right word for what happens when weak data goes in and confident claims come out, and it rarely involves anyone lying. It involves rounding a percentage upward in conversation, dropping the base from a quote, promoting a single segment's result into the headline, and letting roughly and most carry weight the numbers never earned. Agents make this worse if unguarded, because they generate confident summary prose by default — and better if guarded, because the challenge and skeptic passes are tireless about exactly these failures.

Walk a couple of rows. The pricing row is from the survey workflow's case study: 64 per cent acceptable or better looked like broad acceptance until the open-text coding showed much of that acceptance was explicitly conditional — and the challenge pass added that the sample skewed towards engaged customers, so the figure could not be read as a rate across the whole base. The navigation row is the classic segment over-claim: no overall effect, one pre-registered segment moved, and the smallest cell at that — fragile evidence under any reasonable multiple-comparisons view. The honest version, an underpowered signal worth a dedicated follow-up, led to a smaller but real confirmed effect two months later.

The test in the highlight is mechanical enough to apply in review: base, segment, source. Train yourself and the team to ask those three of every quoted number, and most laundering dies in the readout review gate rather than in front of leadership.

The structural problem funnel diagnosis solves is that the number and the interface that produced it are rarely on the same desk. The analytics say 60 per cent drop at email verification, so the team shortens the email copy — but nobody opened the verification screen on a phone, saw the link expire, or noticed the resend button fail silently. The number locates the wound; only the screen explains it.

The workflow runs in two connected stages. First, an analytics agent computes step conversion from a flat export — one row per user, one column per step, plus segment columns — overall and cut by device, plan, and source. The step definitions, time window, and whether steps can be skipped are decisions the researcher writes into the brief before any agent runs, because they change every downstream number. Second, the workflow dispatches walkthrough agents only to the two or three steps where the drop-off or the segment gap is largest. Those agents drive the actual product in a browser, at desktop and phone widths, in a test environment: capture the screen, read the verbatim copy, attempt the step, trigger the plausible error states, note what loads slowly, and check whether progress survives leaving and returning. Everything is recorded as an observation with evidence — a screenshot path, quoted copy — and explicitly not yet as an explanation.

Two caveats belong on this slide. Walkthrough agents work in test environments, so they miss what only happens with real payments, real email deliverability, and real accounts. And sometimes the most important hypothesis is about the instrumentation: a step event firing twice, or not firing on mobile. Broken tracking masquerades as user behaviour, and the workflow should be allowed to say so.

The merge stage is where the discipline about causation is enforced. Funnel data is correlational. A drop-off plus a broken-looking error state is a strong hypothesis, not a demonstrated cause — and the output format makes that impossible to forget, because every hypothesis must cite at least one number and at least one walkthrough observation, and must name the method that would verify or falsify it. Hypotheses that restate the chart in words, or leap to a confirmed cause, or prescribe a redesign with no evidence link, get rejected in the merge.

The verification routes are worth spelling out because they are also a cost ladder. Session replays are the fastest check for behavioural hypotheses — do users actually hit the error state, scroll past the field, abandon at the load stall. A five-user usability test is the right tool for comprehension and expectation hypotheses, where the question is whether the copy means what the team thinks it means. An experiment is reserved for hypotheses that survived a cheaper check and justify a build — and that is the connection to the previous slides: this workflow produces the evidence-backed because clause the experiment pre-registration asks for. The fourth route is an instrumentation fix, for the cases where the walkthrough suggests the number itself is wrong.

The prioritisation at the end — which hypothesis is worth a five-user test versus an engineering quarter — is a business judgment about cost and risk that belongs to the team, with the evidence board in front of it rather than in place of it. One more habit worth keeping: walk the worst step yourself once, on your own phone. Ten minutes of first-hand experience is the cheapest calibration available for judging which hypotheses ring true.

This traced run comes from the funnel diagnosis workflow's case study, and it is worth walking slowly because it shows every discipline from the module in one place. The team's standing theory was motivational: people did not want the product enough to open an email. The segment cut undermined that immediately — 47 per cent drop on desktop versus 71 per cent on mobile is not a pattern motivation explains — and that single cut redirected the whole investigation.

The walkthrough agents supplied the candidate explanations: a verification email that took up to four minutes to arrive in the test runs, an interstitial that said check your email with no mention of delay, a resend control below the fold on a phone, a link that expired after ten minutes — shorter than the delivery delay plus a normal distraction — and an expired link landing on a generic error page with no path back. Note what the merge stage did with this: it produced two ranked hypotheses, each tied to a number and an observation, each tagged with a verification method matched to its nature. The dead-end hypothesis was behavioural, so it went to session replays — the cheapest check — and was confirmed within a week. The expectation-setting hypothesis was about copy, so it was tested as a change.

The outcome — verification completion up 19 points over the following month — belongs to the team's fixes and their verification, not to the diagnosis itself, and the readout said so. That is the standard to hold: the workflow ranks and routes hypotheses; humans and their cheaper checks establish what is actually true.

The exercise is deliberately scoped to the cheapest, highest-leverage part of the module: instrument critique. It needs no export, no analysis scripts, and no statistics — just an existing survey and an hour. Steer participants towards a survey with real stakes if they have one: a pricing or satisfaction survey that informed an actual decision is far more instructive than a throwaway pulse check, because the findings land differently when the data has already been quoted to leadership.

The construct map comes first for a reason: without it, the critique becomes a wording polish, and the most common real defect — questions that serve no construct at all, and constructs with no question — stays invisible. Insist that each agent finding quotes the verbatim wording and names the bias; vague feedback like this could be clearer is not a finding and teaches nothing. The sorting step is where the human judgment sits, and it mirrors the workflow's own boundary: defects loop back into revision, judgment calls go to the researcher.

The last bullet is the uncomfortable one and the most valuable. If the survey already ran, some findings will imply that specific numbers in the report rest on flawed questions. The point is not to relitigate the old report; it is to feel, once, how expensive a pre-launch hour would have been compared to the alternative — and to make the critique loop a standard step before the next launch.

Recap by returning to the opening claim and showing it has now been made operational. Agents do not supply rigour, but the loop does: a critique pass before any respondent answers, sample sizes and analysis from scripts rather than estimates, a challenge or skeptic pass before any human reads a conclusion, a readout review gate where claims are compared to the pre-registration, and a decision recorded separately from the evidence. None of those steps is statistically sophisticated; they are habits, and they are the habits that stop weak data being laundered into confident claims.

It is worth restating the boundaries one last time, because they are what keep the module honest. Surveys describe the people who answered them, and selection bias survives every script. Stated preference is not behaviour. Experiments answer narrow causal questions about the populations and periods they ran in. Funnel data is correlational, and walkthroughs in test environments miss what only happens in production. Where significance, weighting, or contested methods decide something that matters, bring in someone who knows the methods.

The bridge to Module 5 is direct: the same event exports, session paths, and ticket data this module analysed in funnels become the raw material for journey maps and service blueprints — generated from what the product actually records rather than workshop recollection, with the gaps the data cannot see marked honestly, and kept current by re-running the generation rather than redrawing the poster.