Every vendor demo of AI open-end coding looks the same: a thousand verbatims go in, a tidy codeframe comes out, and the whole thing takes ninety seconds. The demo is real. What it leaves out is the part that decides whether you can put the numbers in a client deck.

This is a description of what actually happens between the upload and the codeframe, and of the five places it breaks in ways a spot check will not catch.

What AI coding actually does

Automated coding of open-ended responses is not one operation. It is four, and they fail differently.

It reads the question, not just the answers. The same verbatim means different things under "What do you like least about the brand?" and "What would make you recommend us?" A system that codes answers without the question stem will merge them.

It proposes a codeframe. Either derived from the responses themselves, or matched against a codebook you already have. This is the step where a tracking study lives or dies: wave 12 must use wave 11's frame, or the comparison is meaningless.

It assigns codes to each verbatim — usually several, because real answers carry several ideas. "Cheap but the app crashes" is two codes, not one, and any system that forces a single code per response silently discards half your data.

It reports what it could not place. The residual bucket is the most informative output of the whole process, and the one most often ignored.

The five failure modes

1. Themes that are categories, not findings

An AI asked for themes will happily return "Price", "Quality" and "Service". Those are the sections of a questionnaire, not findings. A codeframe earns its place when it distinguishes "price is high for what you get" from "price is unpredictable" — two complaints with opposite remedies that collapse into one useless code.

2. The residual bucket that grows quietly

The "Other" code is a measurement, not a leftover. If it holds three percent of responses, the frame fits. At fifteen percent, the frame is describing a study that is not the one you ran. Most reporting never surfaces this number, so nobody notices the drift until a client asks what the bar chart excludes.

3. Sentiment mistaken for topic

"The delivery was fast" and "The delivery took a week" are the same topic and opposite sentiment. Systems that conflate the two produce a codeframe where you cannot count how many people mentioned delivery at all — which is usually the number the client wanted.

4. Codebook drift between waves

In tracking work, the codeframe is the instrument. If wave 12 invents "shipping speed" while wave 11 called it "delivery time", the trend line breaks and no chart will tell you. This is the most expensive failure in the list, because it is discovered months later and retroactively.

5. Rules derived from structure instead of meaning

The subtlest failure, and the one that survives every demo. A system that decides what belongs in a group by position — the highest code, the first three options, the longest response — will be right until it meets a questionnaire built differently. A "don't know" option placed last is the classic case: a positional rule sweeps it in with the favourable answers, and the resulting number is wrong in a way that looks entirely plausible on the page. The fix is not a better rule. It is deriving groups from what the labels mean.

What to check before you trust a codeframe

  • The residual rate. Ask for it as a number. If the tool cannot report it, that is the answer.
  • Multi-coding. Take ten verbatims that clearly say two things. Count the codes assigned.
  • The frame against last wave. Not the output — the frame. Codes that changed name changed the trend.
  • Disagreement, on purpose. Have a human code two hundred responses blind and compare. You are not looking for a match. You are looking for where the disagreement clusters, which tells you which codes are ambiguous.
  • The long tail. Sort codes by frequency and read the bottom. Genuine low-frequency themes look different from noise, and only a human can tell them apart.

Where the human belongs

The useful question is not whether AI can code open ends. It can, and the speed difference is not marginal — work that took a team a fortnight takes an afternoon. The question is where a person has to stand in the process for the output to be defensible.

The answer, consistently, is at the frame and at the residual. Reviewing every verbatim defeats the purpose. Reviewing the codeframe before it is applied, and the unplaced responses after, catches most of what matters at a fraction of the cost — because both are small, and both are where the systematic errors live.

Errors in individual assignments are random and average out across a thousand responses. Errors in the frame are systematic and do not.

The method behind all of this — codebook design, multi-coding rules and the quality checks end to end — is covered in our survey coding guide. For how the discipline differs once the work is commercial rather than academic, see open-end coding in market research.