The goal of solid survey design is to limit the answers as much as possible so that your data is clean and reliable without limiting the truth or skewing reality. Before you start designing your survey, you should’ve covered the following steps in your data journey:
- Clearly define your research questions
- Identify the characteristics/parameters you want to collect information about
- Complete the secondary data review and identify data gaps
- Identify the target sample and the geographical location/s for your survey
Once those steps have been completed, you can follow this guide to designing a survey to help you get the most our of your data.
Step one: Outline the design of your survey
After you’ve defined your sample, which you can read about in this blog, you can decide how you’ll collect the data from the respondents. Your survey design will include a survey format with a list of questions which correspond to your data needs and the frequency at which the data will be collected. The frequency of data collection depends on the type of data being collected. Baseline or mapping studies are one time surveys while tracking/monitoring surveys are conducted at time intervals. The periodicity of data collection is determined by the project goals and objectives and the related set of indicators listed in your Theory of Change (download our design eBook for more information).
Step two: Adopt good practices while designing your questionnaire
A questionnaire is likely to be most effective if you KISS: Keep it short and simple. If you don’t have a satisfactory answer to what you will do with the answer to a question, leave it out. Avoid the temptation to add a few more questions just because you are doing a questionnaire anyway. If necessary, place your questions into three groups: must know, useful to know and nice to know. Discard the last group, unless the previous two groups are very short.
Start your questionnaire with an introduction or welcome message clearly stating who you are and why you want the information in the survey. A good introduction or welcome message will encourage respondents to cooperate and participate. In case of sensitive or private information, reassure your respondent that their responses will not be revealed. In some cases e.g. child/underage surveys, it may be mandatory to seek the consent of the respondent or guardian. In any case, it’s important that you gain the informed consent of the participants interviewed. This may require a signature or verbal consent, so think about how you can incorporate this into your survey.
While designing a questionnaire, it’s important to reflect on how the order of the questions can impact the results of your survey. Ideally, you should:
- Place the easiest and most pleasant to answer questions at the beginning of your survey.
- Group together questions on the same topic.
- Leave difficult or sensitive questions until near the end of your survey.
- Address the data collector observations, validation issues, GPS readings, photographs, testing (e.g. water quality testing) at the end of the survey, and avoid breaks while interviewing the respondent.
- Use a logical or natural order to answer choices, presenting positive to negative or excellent to poor scales and agree-disagree choices in that order.
Step three: Design your questionnaire according to the data type
Broadly speaking, there are two types of data: quantitative data and qualitative data. Quantitative data is collected with a structured questionnaire which may have closed ended questions (i.e. with a list of options to choose from) and/or open ended questions, depending on the type of information you need.
Qualitative data is often essential to understanding the context and explaining the quantitative data. It is generally collected as free text, which may be translated into numbers by classifying the information or assigning codes. It is recommended that qualitative information be used sparingly, where the possible responses are not known in advance and will add value to your survey. This is because qualitative answers tend to take longer to check, clean and process.
Most questionnaires will gain value using a combination of both types of data. Your questionnaire will depend on the objectives of your project. If you need data to monitor the status of water points across a city, you are likely to ask the following questions:
- What type of waterpoint is it? Respondents can select from a list of options, e.g. hand pump, well or tap.
- Where is the waterpoint located? Respondents can provide the name of the city/village and a GPS reading.
- Is the waterpoint functioning? You’ll need to clearly define functionality to ensure a common understanding for all data collectors.
- Is the water safe to drink? For this, you may need to test the water for certain parameters, document perceptions on water safety from respondents, or collect healthcare information from existing records.
- Who owns or is responsible for this water point and is it maintained? You can ask whether it is publicly owned (government) or privately owned, what type of repair (major or minor) has been done in the last few years, how much it cost and who paid for it. Again, define what constitutes major and minor repairs.
In the above example of the waterpoint monitoring, questions (a) and (b) are examples of (structured) questions to collect quantitative data. Questions (c), (d) and (e) could be framed to collect both quantitative and qualitative information. Question (c), where you ask about perceptions of safe water, is an example of qualitative information, where you will record the responses verbatim and enter the data as free text.
Step four: Choose the question type to match your data needs
There are three basic ways in which questions are designed in surveys:
|
1. Multiple choice |
e.g. Have you watched this movie? |
☐ Yes |
☐ No |
|
2. Numeric open ended |
e.g. How many times have you watched the movie? |
_____________ |
|
|
3.Text open ended |
e.g. What did you like about this movie? |
_____________ |
|
Rating Scales and Agreement Scales are two common types of questions also used to qualify multiple choice questions.
|
1. How would you rate the movie? |
☐ Excellent |
☐ Good |
☐ Fair |
☐ Poor |
|||||
|
2. On a scale, where ‘10’ means you have enjoyed this aspect the most, how would you rate the movie? |
|||||||||
|
Acting__ |
Editing__ |
Casting___ |
Direction__ |
Production__ |
Cinematography___ |
||||
|
3. How much do you agree/disagree with the following statements? |
||||
|
Strongly Agree |
Agree |
Disagree |
Strongly Disagree |
|
|
a) The movie has a strong social message |
||||
|
b) Children should not watch the movie |
||||
|
c) There is unnecessary violence in the movie |
||||
|
d) The storyline of the movie is weak |
||||
While designing a closed ended questionnaire, you should try to include the maximum possible list of relevant alternatives as answer choices. This helps to systemise and categorise respondent’s answers and saves time on text entries. However, this also reduces the scope for capturing detail and you will need to decide how flexible you want to be and to what extent the additional detail will improve the findings of the survey. Choosing a question type is largely based on how you want the data to come out, which depends on how you want to use the data.
You can pretest the questionnaire before the survey if you want to generate a list of alternative question types. When you’re unsure about the possible answer choices, use an open ended format by adding “other (specify)” as one of the alternatives. Also allow a “don’t know” or “not applicable” response to all questions, except to those in which you are certain that all respondents will have a clear answer.
In your survey, some questions may be dependent on responses to other questions. For example, in our case of the moviegoer, if the respondent’s answer to question one is “no,” i.e. they have not watched the movie, the rest of the questions would be redundant. In this case, you would add an instruction “continue survey only if the response is ‘yes’ in question one.”
Step five: Build in steps for quality control
Pretesting your questionnaire is recommended as it will help you to ensure that your survey design is in sync with your data needs. During a pretest, you get to know whether the questions have been worded properly and are soliciting the expected responses. You may need to edit/remove/add questions and explanations after a pretest. It also gives you a chance to check the quality of the data collectors.
Digital technology allows survey designs in which you can factor in quality checks. Questions can be marked as mandatory or optional and submission of forms can be made dependent on completion. Also, data quality can be periodically checked for timely feedback and course correction.
Do you want more tips on ensuring reliable data collection? Download our eBook!
.jpg)