Jennifer Henderson, Viamo
Offering local languages first in the language selector resulted in a 26% increase in participants choosing a language, and therefore making it to the beginning of the survey, during a 2016 A/B test conducted by Viamo and CDD (Center for Democratic Development) in Ghana. In this same a/b test, the participation of rural women increased by 28% when offering the local language first for the language selector.
Using a longer introduction increased response rates by 26% in a 2014 project led by Viamo, the World Bank and the Brazilian Ministry of Health. Explaining to respondents who the survey is coming from, and how it will be used to improve the respondent’s life, has a big impact on their willingness to engage with the survey, and creates an intrinsic incentive to answer mindfully.
We often see a 30% drop-off rate within the first 10 questions of a survey. Most partners approach Viamo with a request for a specific amount of complete responses in mind, depending on the scope of their project. Given that there is a drop-off in respondents as the survey progresses, it can be more useful to break down the survey in different sections and assign different quotas for each section. Typically, we would want a higher number of responses for questions on which we want high accuracy, or on which we want to conduct analysis of sub-groups of respondents (by gender, age, etc.). Those should come at the beginning of the survey. Those questions where accuracy is less important, or where we don’t wish to conduct sub-group analysis can be placed later in the survey, and with a lower quota. This can help save on airtime costs and could reduce the cost of running the survey by as much as 40%.
Offering incentives of free airtime for completing the survey can increase completion rates by 11-13%, based on surveys run in in Ethiopia, Mozambique and Zimbabwe with the World Bank. The benefits of offering an airtime raffle are often almost identical to offering guaranteed airtime, but with significantly lower costs. That said, the impact of incentives cannot be generalized across countries/cultures. For example, we have seen a decrease in response rates when offering airtime raffles (which can be seen as gambling) in majority Muslim countries, such as Afghanistan.
By providing feedback, 90% of “lost” participants returned and began participating in surveys again, in a 2014 project between Viamo and the Government of Ghana, funded by Star Ghana. In other words, when we ran multiple monthly waves of the survey, we saw a decrease in participation between each wave. We recovered most of that loss when we started sharing the results of each wave with the panel, closing the feedback loop.
Along with these specific pointers, there are other general tips to keep in mind with regards to survey design. These include: making sure the survey has intent and focus, asking clear, unbiased questions, including comprehensive and exclusive answer options, avoiding biases in question phrasing, using clear and specific language, and designing a survey that flows well. For more details on these general tips for creating a great survey, see our full article.