Research Synthesis with Agents

The framing sentence for the whole module is on the slide: synthesis is two jobs. The first job is sorting — reading every transcript, ticket, and review carefully, tagging what each person actually said, and keeping the link between a claim and its source. The second job is meaning — deciding which patterns matter, what they imply for the product, and what the team should do differently. Most synthesis sessions blur the two jobs together in one tired afternoon of sticky notes, which is why the loudest interview and the most recent one carry more weight than they should.

Agents change the economics of the first job dramatically and barely touch the second. A folder of fifteen transcripts that would take a week to code carefully can be coded in an hour, with every quote attributed and every theme counted. But the agent has no idea whether a theme justifies changing the roadmap, whether the sample can support the claim, or which finding the team is politically unable to hear yet. That is the researcher's work, and this module keeps it visibly separate.

If participants did Modules 1 and 2, connect back: the research packet structure from Module 1 is where the synthesis output lands, and the observation-versus-inference discipline from Module 2 reappears here as the line between coded evidence and interpreted meaning.

This slide comes before any technique on purpose. The temptation with agentic synthesis is to point the tool at the raw export and start, because the raw export is right there. The discipline is that nothing enters the agent's working folder until names have become participant IDs, contact details and employers are stripped, and any health or financial detail that is not the subject of the study is removed. The mapping between IDs and identities is kept in a separate file that the agent never sees — a small mechanical habit that prevents a large class of accidents.

Consent is the harder question because it is a judgment, not a script. Most consent forms say something like recordings will be analysed by the research team. Whether that covers automated analysis by an agent is a decision the researcher makes explicitly, documents, and is prepared to defend — not something to discover the answer to after the run. The same goes for the tooling side: confirm that the agreement covering your agent platform keeps the data out of model training before the first transcript is uploaded, and date-stamp that check, because terms change.

The last bullet is practical as much as ethical: agents work from text. Transcripts and notes are what they can read carefully and quote verbatim from; recordings are not. Keeping agents away from video also keeps the most identifying material out of the pipeline entirely.

The codebook is the single highest-leverage artifact in this workflow, because it is the contract between the researcher and every coding agent. Each code gets a short name, a one-sentence definition, and an example quote, so that fifteen agents coding fifteen transcripts apply the same standard. Codes come first from the research questions — what would change a decision — and the codebook explicitly invites emergent codes, conventionally prefixed NEW-, so the framework does not blind the run to what the data actually contains.

Length matters more than people expect. Eight to fifteen codes is workable. Thirty codes means every quote matches something, the coding looks thorough, and the synthesis learns nothing, because the categories are too fine to accumulate weight. The other failure direction is vagueness: a code defined as user frustration will absorb half the corpus and produce a theme that says users are frustrated, which the team already believed walking in.

The non-delegable step is reading. The lead researcher reads at least three full transcripts — or a meaningful sample of tickets — before drafting the codebook, and spot-reads after the run. This is not ceremony. The codebook encodes what the researcher noticed, and a researcher who has not read the material has nothing to encode. The agent accelerates coding; it does not replace familiarity with the data.

Walk the funnel left to right and name the owner of each stage. Prepare is the researcher: anonymisation, consent, reading a sample, and drafting the codebook — everything covered on the previous two slides. Code is agent work, fanned out one agent per transcript or per batch of tickets, so the fifteenth source gets the same quality of attention as the first; the rules are verbatim quotes only, participant ID and line reference attached, and never a paraphrase presented as a quote. Cluster is the merge: themes with a one-sentence claim, the codes they draw on, the participants who support them, and an honest coverage count. Challenge is a separate agent acting as a sceptical second researcher, flagging thin evidence, quotes that do not appear verbatim in the sources, over-generalisation, and disconfirming evidence the synthesis ignored. Interpret is the researcher again: spot-check quotes, decide meaning, set confidence, park what is thin.

Point at the green band and stay on it. The evidence trail is the property that distinguishes this from a very fast opinion generator. At every stage, the link from claim back to verbatim quote, participant ID, and line reference is preserved, which is what makes the output auditable by someone who was not in the run.

In practice this runs as an orchestrated workflow — in Claude Code, a dynamic workflow where intermediate codings stay in script variables rather than the agent's own context, which is what keeps later transcripts from being read more carelessly than early ones. The mechanics matter less than the shape: human bookends, agent middle, evidence trail throughout.

Traceability is the single test that separates trustworthy synthesis from plausible fiction, so this slide sets the standard precisely. A theme is a one-sentence claim plus its supporting machinery: the codes it draws on, the participant IDs behind it, two to four verbatim quotes with line or timestamp references, and a coverage count. The evidence table sits alongside the themes and lists every quote with its source, so a reviewer can go from any claim to the exact words of the exact person in under a minute.

The verbatim rule deserves its own emphasis because it targets the specific way language models fail here. An agent under pressure to be helpful will smooth several similar comments into one tidy quote that nobody actually said, or will paraphrase a sentiment and present it inside quotation marks. The rule is mechanical: a quote must appear word for word in the named source, and if it cannot be found verbatim it is not reported. The instruction goes into the coding prompt, the challenge agent re-checks it, and the researcher checks it again by hand.

The spot-check is a fixed step, not an optional one: pick five quotes at random from the evidence table and confirm each exists verbatim in the named transcript or ticket. The sampling is small because the consequence is large — a single fabricated quote means the entire evidence table is treated as suspect, and the coding stage reruns with a stricter prompt. That sounds harsh and it is meant to; the credibility of every other quote rests on it.

Loudness bias is the quiet failure of manual synthesis: the most articulate participant, the angriest ticket, and the interview that happened most recently all punch above their weight, because human memory rewards vividness. Coverage counts are the antidote, and agents make them cheap. Every theme states how many participants or what share of coded tickets support it, which turns the team conversation from competing anecdotes into a discussion of which patterns are actually load-bearing.

The second half of the slide is the discipline that stops counts becoming a different kind of dishonesty. Eight of twelve participants is a description of the sample, not an estimate of prevalence in the user base, and qualitative synthesis should never be laundered into percentage claims for a board deck without quantitative backing. The same caution applies to corpora that select themselves: support tickets and app reviews only represent people motivated enough to write, so a number like 31 percent of coded onboarding tickets is true of the tickets and unknowable about new users in general. The honest output names that bias rather than hoping nobody asks.

The practical pattern from teams running this well is pairing: the synthesis produces a counted, evidenced hypothesis, and product analytics or a follow-up survey establishes the prevalence before anything is sized or prioritised. That pairing is also the bridge to Module 4, where the quantitative side gets its own treatment.

Agents share a bias with tired researchers: they prefer tidy narratives. Codes that are merely similar get over-merged, the theme that matches the team's existing strategy gets stated a little more confidently than the evidence supports, and the two participants who flatly contradict the pattern quietly fall out of the summary. The challenge pass is the structural answer — a separate agent, prompted to act as a sceptical second researcher, reviewing every theme against the full set of coded excerpts.

Its job is specific, not vaguely critical: flag themes supported by fewer than three sources, flag quotes that do not appear verbatim in the named transcript, flag claims whose language generalises beyond the sample, and surface disconfirming evidence the synthesis did not mention. In the traced runs behind the workflows this course draws on, the challenge pass routinely demotes around a third of the proposed themes — and the demotions are often the most useful output, because they are the claims the team was most ready to believe. One frequently cited example: a synthesis initially claimed customers wanted a specific CRM integration; the challenge pass showed only two participants mentioned it and one was speculating about a colleague's needs. That became a question for the next study rather than a roadmap item.

Two habits make the pass effective. First, the challenge agent is genuinely separate — a different prompt and a fresh context, not the synthesis agent marking its own homework. Second, the researcher reads the challenge file before the findings file. Reading order matters more than it should: once the tidy story is in your head, the caveats read as footnotes.

These rows are drawn from the user-research-synthesis workflow's quality comparison and from its churn-study case, so they are grounded in traced runs rather than invented for contrast. Walk one or two rows rather than all four. The first row is the most common failure: a claim that is probably true, reads well, and cannot be checked, defended, or acted on with any precision. Its good counterpart names the participants, counts the coverage, and points at quotes — which means a designer can read the original wording, and a sceptical stakeholder can audit the claim instead of arguing with it.

The pricing row is worth dwelling on because it shows synthesis changing a conclusion rather than just decorating one. Sales believed churn was about price; the coding showed price was named in seven interviews but in five of them only after a value gap was described, and the side-by-side codes with line references made that sequence visible. That is the kind of finding that only survives because the evidence trail exists — stated without it, it is just the researcher's opinion against the sales team's.

The quote row is the integrity case: a vivid quote with no participant ID that appears in no transcript is not a strong finding badly cited, it is fiction. The honest version is less dramatic and more useful. Close on the highlight: the bad column genuinely does present better, which is exactly why the discipline has to be structural rather than relying on good intentions in the moment.

This is the second of the two jobs from the title slide, and it does not compress just because the first job got faster. The interpretation pass starts from the challenged, counted themes and asks the questions no agent can answer: which of these matter for the decision the team actually faces, what does the pattern mean given everything the researcher knows about the product, the market, and the history that never appears in the transcripts, and what should change as a result. A theme can be well-evidenced and still not matter; a thin theme can be the early signal of something important. Telling those apart is the job.

Confidence levels are the practical tool for keeping that judgment honest. Each theme gets an explicit rating — strong, suggestive, or thin — with a one-line justification that references the coverage count, the challenge flags, and the researcher's own read of the sources. Thin themes are parked as open questions rather than deleted, and the parking is recorded, because the next study's questions should come from this study's loose ends.

There is also an ethics-of-representation point that belongs to the researcher alone: how findings are framed to stakeholders, which caveats travel with the headline, and whether a count is presented in a way that invites over-reading. The agent produced the material; the researcher signs the interpretation. That signature is not a formality — it is the accountability that Module 1 of the Fundamentals course says never transfers, applied to research.

The reporting format question sounds cosmetic and is not. Most research dies in the deck: a finding becomes a bullet, the bullet becomes an adjective, and six weeks later nobody can say which participants it came from or whether it was strong or thin. The packet structure from Module 1 is the alternative — a folder of plain files where the themes, the evidence table, the challenge file, the opportunity map, and the open questions live together, with the codebook and the run prompt archived alongside them.

Walk the files briefly. The themes file carries the claims with their full machinery, including the confidence levels from the interpretation pass. The evidence table is the audit surface — anyone can go from claim to quote to source. The challenges file stays in the packet rather than being tidied away, because the demoted themes and named caveats are part of the finding, not an embarrassment to hide. The opportunities file has one rule: nothing appears in it that is not linked to at least one supported theme, which is what stops the roadmap wishlist sneaking in dressed as research.

The deck still gets made — stakeholders need a readout — but it is generated from the packet and points back into it, rather than being the only place the findings exist. Keeping the packet in the project repository also pays forward: the next study can re-code against an updated codebook and compare across time, and Module 6 builds on exactly this structure when findings get converted into design briefs.

This run follows the same shape as the ticket-mining case in the user-research-synthesis workflow, scaled to a corpus a single team can hold in its head: forty support conversations tagged onboarding, exported as text. Conversations are shorter than interviews, so they are batched — here ten per coding agent — which keeps the run fast without giving any single agent so much material that the later conversations get skimmed.

Walk the table by stage and call out where the human time went. Preparation was the researcher's reading and the codebook draft — about forty minutes. The coding and clustering were agent time, returning 163 coded excerpts and six proposed themes with coverage counts and quotes attached. The challenge pass demoted two themes, and the demotion reasons are the teaching point: one rested on three conversations, and one used users-in-general language for a pattern that, by construction, only describes people frustrated enough to contact support. The interpretation pass spot-checked quotes, set confidence levels, and — because ticket counts cannot establish prevalence — paired the leading theme with product analytics before the team committed to a fix.

Be explicit that the numbers are from a traced run of this pattern, not a benchmark, and that forty conversations is deliberately small. The same structure scales to hundreds of tickets or fifteen hour-long interviews; what changes is batch size and run time, not the shape, and not the location of the human gates.

The exercise is sized so the human audit is genuinely possible: a corpus small enough that fabrication has nowhere to hide. Steer participants towards material they already have rights to and familiarity with — their own product's support tickets, app-store reviews, or transcripts from a past study where consent was broad enough to cover this use. The anonymisation and consent step is part of the exercise, not preamble; for some participants the honest outcome is discovering that their existing consent language does not cover automated analysis, which is itself the lesson.

The codebook step is where most of the design thinking happens, and where most participants will feel the pull to skip the reading. Encourage them to notice that pull, because it is the same pull that produces codebook-shaped themes in real studies. The run itself can use whatever agent setup they have available — the structure matters more than the tooling, and the workflow prompts in the school's user-research-synthesis workflow are a working starting point.

The audit and the hand-written interpretation are the two steps to insist on. The quote audit either builds justified trust or surfaces a fabrication early, in a low-stakes setting; both outcomes are valuable. The interpretation forces the participant to do the second job rather than presenting the agent's clustering as the finding. If running this live, compare confidence levels across the room — two people theming the same corpus with the same codebook will still rate themes differently, which is a useful demonstration that interpretation was never mechanical.

Recap by walking the funnel rather than the bullets: the researcher prepares and interprets, agents code, cluster, and challenge in between, and the evidence trail runs the full length. The two-jobs framing from the title slide is the thing to leave in people's heads — the coding got cheap, the meaning did not, and a team that confuses the two will produce fast, confident, unaccountable findings.

The bridge to Module 4 is already planted in this module twice: the prevalence-versus-loudness slide ended with pairing qualitative themes with analytics, and the worked example did exactly that before the team sized a fix. Module 4 takes up that quantitative side properly — survey instruments that do not lead, experiment readouts that do not invent significance, and funnel diagnosis that produces ranked, testable hypotheses — with the same underlying posture: agents accelerate the work, and the rigour still has to come from somewhere human.

If participants did the exercise, suggest they keep the packet folder. Module 6 returns to it when findings get converted into design briefs, and the open questions they parked here are candidates for the studies they will scope there.