How to analyze survey responses with ChatGPT

Begin with the question and the decision

The same text can support different analyses. A comment about a slow reply may be relevant to a support staffing decision. A comment about confusing assembly instructions may guide a product page update. Tell ChatGPT what you want to learn, but do not ask it to find only evidence that supports your preferred action.

Include the survey question because it shapes the answers. 'What should we improve?' invites problems and suggestions. 'What did you like?' invites positive feedback. A summary of either question should not be described as a balanced measure of overall satisfaction. You need to know what respondents were invited to talk about.

Our invented sample asks: 'What worked well, and what should we improve about your recent order?' The decision is which customer-information and support issues to investigate first. The sample is small on purpose, so you can inspect every row and see how a defensible summary is built. It is not an actual customer survey or a reported business result.

Prepare one row per response

Use stable IDs such as R01 and R02 rather than names. Keep the original answer text beside any cleaned version. Removing an accidental extra space is different from rewriting 'the instructions were fine' as 'instructions need work.' Do not improve the wording before analysis if that changes the evidence.

Decide what counts as a response. Two submissions by the same person may be duplicates or separate experiences, depending on the survey design. Do not automatically delete matching sentences: two people can write 'no issues.' Set a rule using the actual collection method, apply it consistently, and keep a record of excluded rows.

In this sample, all 12 submissions are retained as distinct survey responses. R11 and R12 contain blank text. We have 10 nonblank comments for theme analysis. A blank is not the same as 'no issues,' which is a real answer expressing something about the experience. The count of invitations, submissions, and usable text should stay separate throughout the report.

Create a codebook with clear boundaries

A codebook is a short set of rules for labeling comments. Without one, ChatGPT may call the same issue 'unclear guide' in one batch and 'setup difficulty' in another. You then cannot tell whether a difference in counts reflects the comments or the wording of the labels.

Use themes that describe what the text supports. 'Instruction usability issue' includes confusing wording, a hard-to-follow guide, and a request for larger diagrams. It does not include every mention of setup. R09 asks a setup question but does not say the instructions were unclear. R05 explicitly says the instructions were fine, so it should not receive the issue label.

A request for an assembly video is a separate theme. R02 could not find one; R07 asks for one. That supports 'video requested or not found,' but not 'the company has no video.' R06 says a video helped, showing why the stronger claim would be wrong. Your label should preserve the difference between what a respondent experienced and what is true of the whole service.

Positive themes also need definitions. 'Product positive' covers an explicit positive assessment of the product or a stated absence of product problems. 'Helpful support' requires a positive support experience, not any mention of support. Keep those labels separate from slow or absent replies so mixed comments retain both parts.

Allow more than one label on a comment

R01 reports late delivery and helpful support. If you force it into a single negative category, you lose evidence that the support response worked. R04 mentions both late delivery and unchanged tracking. Those are different problems even if they happened in the same order. Multi-label analysis means one response can receive several relevant labels.

Keep a row-level table before making the summary. This is the bridge between the original words and the theme counts. You can inspect a wrong label, change the rule if needed, and recalculate without guessing how the paragraph was produced. For a small sample, check every row. For a larger sample, use the same rules in manageable batches and examine uncertain or consequential cases closely.

Negation needs particular care. R08 says the parcel arrived on time, so it should receive tracking issue but not delivery late. R05 says instructions were fine, so it should receive support slow or absent but not instruction usability issue. A keyword match that finds 'delivery' or 'instructions' would miss those distinctions.

Count distinct IDs and name the denominator

Count responses, not occurrences of a word. If one respondent says 'late' three times, that is still one response mentioning late delivery. For each theme, gather its distinct IDs and count them. A spreadsheet can do this from one true-or-false column per theme; a script can do the same for larger files. The IDs make the result easy to verify.

Our eight theme counts total 16 labels across 10 nonblank comments. That is expected because several comments have two labels. The percentages below sum to 160%, which is also expected for multi-label themes. Do not turn the themes into a pie chart that implies they divide the respondents into separate, non-overlapping groups.

The table uses 10 nonblank comments as its denominator. Instruction usability has three matching IDs, so 3 ÷ 10 × 100 = 30%. If you instead report among all 12 submissions, it is 3 ÷ 12 × 100 = 25%. Both calculations can be correct, but they answer different questions and must be labeled.

The submission rate is 12 ÷ 40 × 100 = 30% of invited people. Ten of the 12 submissions contained text, about 83.3%. Neither number is a satisfaction score. Do not call people who skipped the text question satisfied, and do not project these voluntary replies onto every customer.

Repair a misleading first summary

Suppose ChatGPT writes, 'Most customers complained about delivery, and there is no useful assembly video.' The sample supports neither statement. Two of 10 nonblank comments mention late delivery, which is not most. R06 explicitly says the video helped. R02 and R07 suggest a finding or access problem, not proof that no video exists.

Ask for a precise repair: 'Use the checked theme table and source IDs. Replace claims about all customers with claims about this sample. Delivery late has R01 and R04 only. Preserve R06's helpful-video comment. Describe R02 and R07 as video requested or not found. Name the denominator for every percentage and avoid most unless the count is more than half.'

Do not solve a disputed label by asking for more confidence. Return to the words and the rule. If a comment is genuinely unclear, mark it uncertain and explain the choice. You can count it separately or show how the totals would change under two reasonable interpretations. That is more honest than hiding an ambiguous sentence inside a firm total.

Write a result someone can use

A useful summary links patterns to limited, practical actions. Instruction usability appears in three comments, but the examples differ: confusing wording, a hard-to-follow guide, and smaller-than-desired diagrams. Read those comments before deciding that a complete rewrite is required. You might test clearer steps and larger diagrams first.

Video access deserves a separate check because two people requested or could not find a video, while another found it helpful. An obvious next action is to inspect where the video link appears and whether customers can find it. The survey does not prove the video content is bad. Tracking and delivery should also stay separate because R08 reports unchanged tracking despite on-time arrival.

Do not rank solely by frequency. A rare report can matter greatly if it describes a serious harm or a blocked essential task. Investigate the actual case using an appropriate process. With ordinary improvements, compare frequency, severity, effort, and the decision you can realistically make from the available evidence.

Worked summary: We received 12 submissions from 40 invited people; 10 contained text. Among those 10 comments, three mentioned instruction usability issues. Two mentioned late delivery, two tracking issues, two slow or absent support replies, and two a requested or hard-to-find assembly video. Comments could receive several themes, so counts overlap. Three comments were positive about the product, one described helpful support, and one said the video helped. Suggested next steps: inspect the guide wording and diagram sizes, check how customers find the assembly video, and review the two support-response cases. These are patterns in a small voluntary sample, not measured rates for all customers.

Scale the method while protecting the original evidence

For a larger survey, finalize the codebook on a varied initial group, then apply it consistently to the remaining rows. If a genuinely new theme appears, define it and check earlier batches for the same issue. Otherwise the later responses get a label the earlier ones never had a chance to receive, which distorts comparisons.

Use stable IDs in every batch and join results back to the original table. Check that all intended IDs appear exactly once in the labeled output. Count themes with ordinary spreadsheet formulas or a script after labeling. This keeps repeated calculations cheap and reproducible, while ChatGPT handles the harder task of interpreting the language.

Share only what the analysis needs. De-identification means removing or replacing details that identify someone, but unusual stories can still identify a person in a small team. Public quotations deserve another check for that reason. You can paraphrase a sensitive point in a clearly labeled summary rather than publish a distinctive quote.

Save the codebook, labeled rows, denominator definitions, and final checked table. The next survey can then use the same method. A change in themes becomes easier to interpret because you know which differences came from customer comments and which came from your own rules.

Your next useful step

Turn the strongest supported themes into small actions or follow-up questions. Keep the codebook and ID table so the next batch can be compared using the same rules.