Writing Survey Questions That Survive Contact With a Respondent

In short

A questionnaire is not a list of things you want to know. It is a script a stranger will read once, under time pressure, possibly in their second language, and answer in a few seconds. Most bad data comes from questions that were clear to the person who wrote them and ambiguous to everyone else.

The faults that do the real damage are predictable: two ideas in one question, a word the respondent reads differently from you, response options that do not cover every case, a sensitive question placed before any trust has been built, and a leading phrase that supplies the answer. Every one of them survives internal review because the people reviewing already know what was meant.

The only reliable defence is to pilot. A questionnaire that has not been read aloud to real respondents in every language it will be used in is a draft, not an instrument.

Start with the decision, not the questions

The first draft of every questionnaire is too long, because it is written as a wish list.

Work backwards instead. Write down the decision the research will inform, then the handful of findings that would change that decision, then the minimum set of questions that produce those findings. Anything that survives that filter earns its place. Anything that does not is interesting rather than necessary, and interesting is what pushes a twelve-minute interview to twenty-five and collapses your completion rate.

A useful discipline: for every question, say out loud what you will do differently depending on the answer. If there is no answer, drop it.

Writing a single question that works

One idea per question

The double-barrelled question is the most common fault in survey research and the easiest to spot once you are looking for it.

"Are you satisfied with the price and quality of the service?" cannot be answered by someone who thinks it is cheap and poor. Whatever they pick, you cannot interpret it. Split it in two.

Watch for the hidden ones too. "Do you support the new tariff policy?" bundles the policy, the government that introduced it, and whatever the respondent thinks the tariff does.

Use the words respondents use

Every organisation has internal vocabulary, and it leaks into questionnaires. "Primary household decision maker", "formal financial inclusion", "brand consideration set". None of these mean anything to a respondent.

Write how people speak. Instead of "What is your primary mode of transportation to work", ask "How do you usually get to work?"

Define the terms that could mean two things

Common words carry different meanings for different people, and the differences are invisible in the data.

"Household" is the classic. In much of Nigeria a household includes people who sleep elsewhere and relatives who contribute to a shared pot, and whether a respondent counts them changes every per-household figure you report. Define it in the question: "By household I mean everyone who normally eats from the same pot."

The same applies to "income", which some people report as salary and others as take-home after deductions and others as household total, and to "regularly", which could mean daily or monthly depending on who is asked.

Attach a time frame to every behaviour question

"Do you buy this brand?" is unanswerable. Bought once, three years ago? Buys weekly? Both say yes.

"In the last seven days, how many times did you buy this brand?" is answerable, and recall over seven days is reasonably accurate. Push the window out to a year and you are measuring memory rather than behaviour. Shorter windows cost you rare events, so match the window to how often the behaviour actually happens.

Do not lead

A leading question supplies its own answer, and respondents, especially where an interviewer is present and politeness matters, will take it.

"Don't you agree that the new policy has hurt small businesses?" is not a question. "What effect, if any, has the new policy had on your business?" is.

Loaded words do the same work more quietly. "Reform" and "change" describe the same thing and pull opposite ways. Where a topic is contested, test alternative wordings and report which one you used.

Avoid negatives

"Which of these services do you not use?" takes an extra step of processing and produces errors even among careful respondents. Ask which they do use and invert it yourself.

Double negatives are worse. "Do you disagree that the policy should not continue?" has no clean answer.

Assume nothing about the respondent

"How satisfied are you with your bank?" assumes they have one. "When did you last vote?" assumes they have voted and have been eligible to.

Screen first, then ask. A filter question costs five seconds and saves a column of meaningless data.

Writing the response options

The options are half the question, and they get a fraction of the attention.

Cover every case, and do not overlap

Age bands written as 18 to 25, 25 to 35, 35 to 50 have two problems: anyone who is exactly 25 belongs in two bands, and nobody over 50 has anywhere to go. Write 18 to 24, 25 to 34, 35 to 49, 50 and over.

Balance the scale

A satisfaction scale running "excellent, very good, good, fair, poor" is not balanced. Three of the five options are positive, and results drift upward before any respondent has formed a view. Use equal numbers of positive and negative options with a defined middle.

Separate "no opinion" from the middle

A midpoint collects people who genuinely sit in the middle, people who have no view, and people who do not want to say. If that distinction matters, and in political and sensitive research it always does, provide "Don't know" and "Prefer not to say" explicitly rather than letting all three groups hide in the middle.

Rotate long lists

Options near the top of a long list get chosen more often on screen, options near the end get chosen more often when read aloud. Randomise the order per respondent and the bias averages out instead of accumulating. Keep "Other", "None of these" and "Don't know" anchored at the end.

Offer "Other, please specify" when the list might be incomplete

And read what comes back. Open text in the Other box is the cheapest way to discover that your list missed something important, which is why the pilot matters.

Ordering: the part that gets skipped

Order changes answers. The same question produces different results depending on what came before it, and that effect is as large as wording in many studies.

Open the easy way. Start with something straightforward and non-threatening. The first question establishes whether this is going to be an interrogation or a conversation.

Go general before specific. Ask about overall satisfaction before asking about individual features. The reverse order makes respondents aggregate the features you just reminded them of, rather than reporting the view they actually hold.

Keep sensitive questions late. Income, voting intention, health, anything a respondent might not want to disclose belongs after rapport exists. A survey that opens with "What is your monthly income?" loses people at question one.

Group related questions. Jumping between topics raises cognitive load and drops data quality.

Demographics at the end, with the exception of anything needed to screen or quota.

Mind priming. A block of questions about insecurity followed by "How do you rate the government's performance?" produces a different number than the reverse order. Where this matters, rotate blocks between respondents and report that you did.

Length

Every extra minute costs completions, and the people who drop out are not a random slice of those who finish.

As a working guide: a telephone interview should aim for 12 to 15 minutes, an in-person interview can run 30 to 45 where the respondent is compensated for their time, and a self-completion survey on a phone should stay under 10. Beyond those, straight-lining rises and open-ended answers shrink to a word.

If the brief will not fit, the right move is to drop objectives rather than squeeze them. Two questions answered properly beat six answered carelessly.

Multilingual questionnaires

This is where Nigerian fieldwork differs from the textbook, and where most imported instruments fail.

A questionnaire administered in English, Pidgin, Hausa, Yoruba and Igbo is five instruments, not one. Word-for-word translation does not survive, because scale labels in particular do not map evenly. The gradations between "strongly agree" and "agree" exist in English and have to be rebuilt rather than translated.

What works:

Translate, then back-translate. A second translator renders it back into English without seeing the original, and the differences show you where meaning moved.

Have interviewers who will administer it read it aloud. Written translation that works on paper often stumbles spoken.

Pilot in every language. A question with a 4% don't-know rate in English and 30% in one translation is a translation problem, and you can only see it by running it.

Keep the scale consistent across languages. If one translation effectively offers four points where the others offer five, the results are not comparable.

Pilot, and treat it as real

Twenty to fifty interviews with people like your real respondents, administered the way the real fieldwork will be.

What to look at:

• Where interviewers had to explain a question. Every explanation is a question that failed.
• Don't-know rates. High rates signal ambiguity more often than genuine ignorance.
• Questions with almost no variation. If 97% give the same answer, either the question is leading or it is not worth asking.
• Length. Time it, and remember the pilot is usually faster than the field.
• Open text in "Other" boxes. These tell you what your options missed.
• Drop-off points in self-completion. Where people leave is where a question is too long, too intrusive, or too confusing.

Then change the questionnaire. A pilot that changes nothing was not a pilot.

The pre-field checklist

1. Every question maps to a decision.
2. No question contains two ideas.
3. Terms that could mean two things are defined in the question.
4. Every behaviour question has a time frame.
5. No leading or loaded wording, and contested topics have had alternatives tested.
6. No negatives, and certainly no double negatives.
7. Response options are exhaustive and non-overlapping.
8. Scales are balanced, with "Don't know" separate from the midpoint.
9. Long lists rotate.
10. Sensitive questions sit late; demographics sit last.
11. Length fits the mode.
12. Every language version has been back-translated and piloted.
13. Logic and skip patterns have been tested on the actual instrument, not on paper.

Related reading

• Likert scales explained
• How many people you need to survey
• Our fieldwork and languages
• Our disclosure standards

Frequently asked questions

What is questionnaire design?

Questionnaire design is the process of turning research objectives into a set of questions that respondents understand the same way the researcher intended, in the order and language they will be asked. It covers the wording of each question, the response options offered, the sequence, the length, and the translation and piloting needed before fieldwork begins. Most data quality problems originate here rather than in sampling or analysis.

What is a double-barrelled question?

A double-barrelled question asks about two things at once, such as "Are you satisfied with the price and quality of the service?" A respondent who feels differently about each has no honest answer available, and whichever option they choose cannot be interpreted. It is the most common fault in survey questions and the fix is simply to split it into two.

How long should a survey be?

It depends on the mode. A telephone interview should aim for 12 to 15 minutes, an in-person interview can run 30 to 45 minutes where respondents are compensated for their time, and a self-completion survey taken on a phone should stay under 10. Beyond those thresholds completion rates fall and answer quality degrades, with straight-lining rising and open-ended responses shrinking to single words.

Does question order affect survey results?

Yes, and the effect can be as large as wording. Asking about specific features before overall satisfaction changes the overall figure, because respondents aggregate what they were just reminded of. A block of questions on one topic also primes answers to what follows. Good practice is to move from general to specific, keep sensitive questions late, group related items, and rotate blocks between respondents where priming is a genuine risk.

Should a survey scale include a neutral or don't-know option?

A midpoint and a "Don't know" are different things and should be offered separately. A single middle option collects people who genuinely sit in the middle, people with no view, and people unwilling to say, and those are three different findings. In political and sensitive research the distinction matters enough that "Don't know" and "Prefer not to say" should always be available explicitly.

How do you design a questionnaire for multiple languages?

Treat each language version as a separate instrument rather than a translation. Translate, then have a second translator back-translate into the original language without seeing it, and investigate wherever the meaning has shifted. Have the interviewers who will administer it read it aloud, keep scales consistent across versions so results remain comparable, and pilot in every language. A question with a much higher don't-know rate in one language has a translation problem.

Why do surveys need to be piloted?

Because a questionnaire that is clear to its author is routinely ambiguous to respondents, and only real respondents reveal that. A pilot of twenty to fifty interviews, run the way the real fieldwork will be run, surfaces questions interviewers had to explain, unexpectedly high don't-know rates, questions with no variation in the answers, the true length, and the points where people abandon a self-completion survey. A pilot that leads to no changes was not a pilot.

What is a leading question in a survey?

A leading question signals the expected answer, for example "Don't you agree that the new policy has hurt small businesses?" Respondents, particularly in interviewer-administered surveys where politeness norms apply, tend to accept the suggestion. Loaded vocabulary does the same thing more subtly, since words like "reform" and "change" describe the same thing while pulling in opposite directions. Neutral phrasing and tested alternatives on contested topics are the defence.