Insecurity, Remoteness Pose Steep Challenges in Surveying Afghanistan
November 18, 2015
Afghanistan remains one of the most challenging research environments in the world. The biggest challenge, particularly for a nationally representative sample, is access – not only because of remoteness and lack of roads, but also because of insecurity from armed anti-government groups, banditry, and the risk of encountering improvised explosive devices along the roads.
Random selection is central to the representativeness of any national survey. The Asia Foundation’s annual Survey of the Afghan People uses a household list of villages from the Afghan government’s Central Statistics Organization to make random selections of the villages targeted for sampling. As described in the methodology chapter of the survey book, for a substantial number of villages nationwide, access is untenable due to insecurity. This year, nearly 20 percent of villages had to be replaced due to lack of access from insecurity (332 out of 1,684 randomly selected sampling points).
Since the survey began in 2004, none of the 5,000+ enumerators have died while collecting field data due to insurgent attacks. Several have been injured in car accidents, and the threat of roadside improvised explosive devices is an ongoing concern. In fact, in this year’s survey findings, more than two-thirds of Afghans report that they always, often, or sometimes fear for their personal safety – the highest rate since 2006.
When access to a designated village for survey research is not possible, the survey uses a combination of two research methods to compensate. The first method is to replace the village with another randomly identified village within the same strata, meaning the same province and the same geography as urban or rural. This year, 22.8 percent of replacements are due to remoteness, while 68.9 percent are due to insecurity. In cases where it cannot be replaced, the survey uses a second research method that helps identify the bias caused by sampling replacements. This method, called intercept interviews, involves interviewing Afghans traveling to or from an inaccessible village, such as when encountering them in a bazaar or a bus stop in a more secure district. For all trend lines, these intercept interviews as a separate sample, give a fairly reliable estimate of the bias caused by sampling replacements. Using these, The Asia Foundation can estimate, for example, how the exclusion of Afghans living in inaccessible areas from the random national sample may affect the national average. Are those in insecure areas more likely or less likely to have confidence in the Afghan government, for example?
Not surprisingly, intercept interviews in this year’s survey, as well as in previous years’ surveys, are much more likely to cite insecurity as a major concern than the main sample, and are more likely to cite armed opposition groups as primary providers of local security. However, on many other issues, differences in the responses between intercept responses and the main sample are not significantly different. Wherever significant differences were discovered, these were reported in this year’s survey.
A second major challenge is finding qualified interviewers. The survey always conducts gender-matched interviews, meaning that female enumerators interview females and male enumerators interview males. The more than 900 enumerators responsible for the 2015 survey were carefully trained, and more than 80 percent are repeat enumerators who have worked on previous Asia Foundation surveys. Place of origin and residence of each enumerator is carefully matched to where they conduct field work.
For example, a well-trained enumerator who lives in Kabul province may be ill-equipped to conduct fieldwork in remote parts of Zabul province because he or she may not be a known entity and may obtain more biased responses. Enumerators should be perceived, as much as possible, as locals in the provinces where they are conducting survey fieldwork. This ensures better access to remote villages through local knowledge and contacts, and in most but not all cases, and on most but not all issues, locals are considered more trustworthy for sharing of true opinions and attitudes.
A third challenge is monitoring fieldwork when access is limited, particularly for areas that are insecure. GPS devices are used to monitor some fieldwork, but enumerators cannot carry GPS devices in highly insecure areas. In such areas, a person carrying a GPS unit may be considered an intelligence officer, or as someone with links to the Afghan government or to foreign organizations, all of which may put the enumerator’s physical safety at risk.
Monitoring extends beyond the validation of GPS coordinates. Each year, the survey hires a third-party monitoring company to conduct direct observation, back-checks, and real-time validation of fieldwork. This monitoring is conducted strategically in the most insecure areas, allowing The Asia Foundation to detect and adapt to challenges as they emerge in the field. A final step in the monitoring process is the application of more than 45 logic tests to ensure that the data is valid. Logic tests help to measure the quality of the data, and an enumerator who has a high rate of failure is flagged for investigation, and in some cases, disciplinary action. Logic tests range from the simple to the complex. A simple test might include a respondent’s gender (e.g., “male”) crossed with occupation (e.g., “housewife”), and would thus be flagged as invalid.
Together, these steps are used to ensure quality data for decision-making, as well as information for objective analysis. Importantly, no survey in Afghanistan is without bias. Bias comes in many forms and in every type of survey. In a face-to-face survey, it comes from the replacement of insecure villages for more secure ones. In mobile phone surveys, it comes from the slight pre-selection of richer, more educated Afghans who are more likely to have mobile phones.
The Asia Foundation is committed to implementing best practices in survey research for Afghanistan, and to presenting the data in a neutral and transparent way. All data from all years (more than 75,000 interviews) is public and free. For more on the 2015 survey’s methodology, see Chapter 9 in the book. A full list of sampling replacements and reasons for replacement, is available here.
Zach Warren is The Asia Foundation’s director of Surveys and Research in Afghanistan. He can be reached at [email protected]. The views and opinions expressed here are those of the author and not those of The Asia Foundation.
About our blog, InAsia
InAsia is posted and distributed every other Wednesday evening, Pacific Time. If you have any questions, please send an email to [email protected].
ContactFor questions about InAsia, or for our cross-post and re-use policy, please send an email to [email protected].
The Asia Foundation
465 California St., 9th Floor
San Francisco, CA 94104
PO Box 193223
San Francisco, CA 94119-3223
HIGHLIGHTS ACROSS ASIA
2020 Annual Report
Addressing the global crisis