Survey of the Afghan People: Data Reliability in Challenging Landscapes
December 12, 2012
It is 11 in the morning on June 21, 2012. The phone has been ringing at ACSOR’s (Afghan Centre for Socio-Economic and Opinion Research) Central office in Kabul. The project management team has been busy assigning new sampling points to those calling in and seeking a replacement. The phone rings again and this time it is the field team supervisor from Badakhshan province. One of the villages assigned as a sampling point in the province is discovered to be under the control of the Taliban and so a replacement is sought. Then a team from Farah Province calls in for a replacement reporting no motorable road to the Gala Jowi village; this call is quickly followed by one from the team unable to reach Beatne Village, in Nangarhar, where the way has been blocked by a recent flood.
The above scene is part of the intense process during the data collection phase of The Asia Foundation’s annual Survey of the Afghan People. To ensure the safety of this year’s 742 interviewers (314 of them women), any location chosen by the central office, using a multistage, random sampling methodology, where, at the time of the fieldwork, active fighting is ongoing, or where anti-government elements are in control, is replaced by another where there is no active clash. This is the same process for villages experiencing natural disasters, like floods, or those that are so remote they are unreachable by road or foot, or villages that are untraceable (owing to settlement lists that are remnants from the 1979 census). A replacement is made by selecting another point within the same district, or at least within the same province in cases where the whole district is facing ongoing warfare or is under Taliban control. The ethnic composition and the geographical location of the replacement are kept as close to the original point as possible in an effort to derive similar response trends. To account for the needed replacements, an error factor is calculated and applied to the survey findings.
The total replacements accounted for 31 percent of the total sampling points this year, 16 percent of those for security reasons. In 2011, the replacements due to security were 11 percent; in 2010 also 16 percent; and in 2009, 12 percent. A detailed list of replacements made in 2012 can be found in the methodology section of the Survey, Appendix 2.
Several measures are taken to increase the reliability of the data in the face of replacement problems and other difficulties with conducting interviews in Afghanistan.
Additional error margin due to replacement of sampling points:
The sampling (stochastic) error for the survey which considers the sample size and the sample design has been consistent at +/-2.4 percent each year since 2006. In addition, a replacement-related error (systemic error) has been calculated every year since 2009, when the need for replacements began to increase. The survey from 2008 has been taken as the base for the error calculations. The locations that have been replaced due to inaccessibility in subsequent years, but were surveyed in 2008, are removed from the data set of 2008, and the two 2008 data sets, the original and the revised, are compared. The highest difference between responses for any particular question is taken as the error factor. So for this year, the 2012 sampling points that were not accessible were removed from the 2008 data set and the two data sets were compared and differences in responses noted. This year, the systemic component of margin of error has been calculated at +/-2.7 percent, so combining the stochastic and the systemic error there is a total margin of error of +/-5.1 percent.
Increased sampling points:
To expand and enhance the sampling reach, and at the recommendation of our partner, ACSOR, this year six interviews were conducted in each sampling point as opposed to eight in previous years. This increased our sampling points to 1,055 in 2012, compared to about 880 in previous years. Increasing the spread of the sample across the country increases its representativeness and minimizes homogeneity in responses that may arise among respondents living in a cluster. Once the points are determined, the field team supervisors and the enumerators are assigned their locations. A laudable effort is made by the teams to reach all the points on their lists; a replacement is only made when absolutely necessary.
Increased quality control:
All survey questionnaires undergo computer-aided checks. For example, if a set of questionnaires, especially those administered by a single interviewer, reveals similar responses to a large section of the questionnaire, then they are pulled out for further scrutiny. In addition, 25 percent of the interviews were supervised in person during the survey. Other quality control methods included supervisor accompanied interviews (3.5%), back-checks by ACSOR’s central office (5.6%), and back-checks by the Foundation’s staff (6%). The accompanied checks and back-checks make up over 40 percent of the interviews this year. This year 76 questionnaires were rejected after these quality checks.
Social desirability bias:
One concern that exists in all surveying is that respondents are not frank in their responses. As social scientists know, sometimes survey respondents give answers considered to be socially desirable rather than what they truly feel. During the questionnaire development for the Survey of the Afghan people, great care is taken to reduce questions that are potentially sensitive. Again, to address ethnic sensitivities during the process – a major consideration in a place like Afghanistan – we have local interviewers from the same ethnic background and same province who are deeply familiar with the region conduct the interview. At the start of the interview, the interviewers introduce themselves as members of ACSOR, an independent research firm that conducts several such surveys. The respondent is told that their participation in the survey is confidential and that their names will not be given out and their responses will be analyzed along with thousands of others from across the country. Of course, these steps don’t always guarantee frank and uninhibited responses. Sixty-four percent of the respondents said that it is generally not acceptable to talk negatively about the government in public. So that could always weigh on the responses. There are unanswered questions, and follow-up qualitative research to get deeper insight into some issues is a goal, including an independent analysis on aspects of social desirability bias in the survey, which is planned for early 2013.
The Survey of the Afghan People aims to represent the voices of the whole nation, not just the voices of the elite, the well-read, the development worker, the government servant, or the political analyst. The demographic profile of the 6,290 randomly selected Afghan citizens includes 58 percent who never went to school, 77 percent from villages outside of larger cities, and 38 percent who are housewives. As such, the survey offers a rare glimpse into the views expressed by regular Afghan citizens. Taken together, the 6,000 plus samples in each of these years allows for very robust analysis and insight into Afghan perception and the direction of the country.
Sunil Pillai is a technical advisor for The Asia Foundation in Afghanistan. He has been involved with all of the surveys conducted by the Foundation since 2004. He can be reached at [email protected]. The views and opinions expressed here are those of the individual author and not those of The Asia Foundation.
About our blog, InAsia
InAsia is posted and distributed every other Wednesday evening, Pacific Time. If you have any questions, please send an email to [email protected].
ContactFor questions about InAsia, or for our cross-post and re-use policy, please send an email to [email protected].
The Asia Foundation
465 California St., 9th Floor
San Francisco, CA 94104
PO Box 193223
San Francisco, CA 94119-3223
HIGHLIGHTS ACROSS ASIA
2020 Annual Report
Addressing the global crisis