Avoiding a blank stare

Using cognitive pretesting to improve the WEAI

As a researcher, how often have you wondered whether survey respondents understand your survey questions? Ensuring that survey questions generate the intended information is especially tricky when asking questions on topics that don’t have standardized modules. When a carefully crafted question is met by a blank stare, giggle, or unexpected response, it can be challenging to determine what went wrong and how to fix it. 

Enter cognitive pretesting, which involves conducting a survey while simultaneously collecting observational or verbal information about the responses (Willis, 2005).  The purpose is to systematically identify and analyze sources of response error in surveys and to use that information to improve the quality and accuracy of survey instruments (Johnson, 2013).  In practice, cognitive testing may include following a questionnaire module with a series of standardized questions about how well the respondent understood the module. Questions could cover terms that might have been ambiguous or complex, whether the respondent found the question difficult or thought others would, or whether the respondent understood the question as intended, by asking him or her to repeat it back in their own words. This process allows researchers to ensure the validity of their questions and is especially important with new or revised instruments and when surveys are implemented across cultures or languages (Johnson, 2013). IFPRI researchers recently put this tool to use on a revised and updated version of the Women’s Empowerment in Agriculture Index (WEAI 2.0). [1]

Following collection of the baseline WEAI surveys in 19 USAID Feed the Future countries, a 2013 learning event among partners and practitioners helped reveal the survey’s most challenging components.  The most problematic modules across all countries included time use, autonomy in production, and group membership.  In addition, they felt the survey was too long. Partners decided to address these issues by developing another version of the questionnaire that was shorter and that clarified the problematic sections for respondents while ensuring cross-national comparability.  IFPRI conducted cognitive pretesting of the WEAI 2.0 in Bangladesh and Uganda in 2014.

Following cognitive testing guidelines, we used a diverse sample of at least 10-15 households per language group. We interviewed 120 respondents in Uganda (from four major language groups) and 70 in Bangladesh (with one major language).

Time frame and recall issues

Through standardized probing, we investigated whether the timeframe specified in a question was the same as the timeframe referenced in the response.  In Uganda, the most glaring recall issue came from the question, “Did you yourself participate in [ACTIVITY] in the past 12 months (that is, during the last [one/two] cropping seasons)?”  When asked how much time respondents included in their response, answers ranged anywhere from 3-12 months; 35% of respondents either could not come up with the recall period used or referred to a timeframe other than 12 months.  Modification of the question to, “Did you yourself participate in [ACTIVITY] in the past 12 months (that is, during the last [one/two] cropping seasons), from [PRESENT MONTH] last year to [PRESENT MONTH] this year?”, resulted in much less recall error with only 6% of respondents stating periods of less than one year.

WEAI pic

Photo credit: Chiara Kovarik

Cognitive interviews can also be helpful in testing different recall periods.  Time use data were initially collected using a 24-hour recall module, which was cited as problematic because it took substantial time to administer, did not capture seasonality, and may not be indicative of a usual day.  We therefore tested an alternative one-week recall.  The findings show that in Uganda, roughly 59% of respondents in both rounds said their activities change from day to day and 16.1% of these respondents cited seasonality as the reason. In Bangladesh, half of respondents in the first pretest and two-thirds of respondents in the second pretest said their schedule changed from day to day.  These findings illustrate the need to develop time use modules that address daily variations.

Difficult questions

Another way to identify cognitive problems is to ask the respondent both whether they thought a question was difficult and whether they thought others would find it difficult.  Inconsistencies between these answers can reveal comprehension problems. This was apparent in testing questions regarding autonomy in production using vignettes, which in the WEAI 2.0 were short stories about farmers engaged in various agricultural production activities.  In Uganda, 7-14% of respondents said they found the questions difficult, but 29-60% thought others would have difficulty.  In Bangladesh, few respondents noted these questions as difficult to answer themselves but 29-39% said they thought others would have difficulty.The examples here highlight the valuable insights gained from cognitive pretesting.  While all surveys face resource constraints, conducting cognitive testing as part of a standard pretest can ultimately save time for respondents, enumerators, and analysts while reducing response error. It is especially useful for new instruments or ones that will be implemented across diverse settings.

Works Cited:

Willis, Gordon B. Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage, 2005.

Johnson, Kiersten. Cognitive Pretesting of Cross-nationally Comparable Survey Instruments in a Developing Country Context Seminar. International Food Policy Research Institute. Washington, DC. 9 May 2014.

 


[1] The WEAI was developed by the United States Agency for International Development (USAID), the International Food Policy Research Institute (IFPRI), and the Oxford Poverty & Human Development Initiative (OPHI) in 2012.  For more information, please see: http://www.ifpri.org/book-9075/ourwork/program/weai-resource-center.

 

About authors:

SONY DSCChiara Kovarik is a Senior Research Assistant in the Poverty, Health, and Nutrition Division at IFPRI. Since joining in 2012 she has worked on a variety of gender projects, including the Gender, Agriculture, and Assets Project (GAAP), the Women’s Empowerment in Agriculture Index (WEAI), and other projects on gender and natural resources. She also previously served a two year term as IFPRI’s Gender Task Force RA. She received her Master’s degree in International Policy from Stanford University in 2012.

 

Malapit_HazelDr. Hazel Malapit is a Research Coordinator at the Poverty, Health and Nutrition Division at the International Food Policy Research Institute (IFPRI). She coordinates research, training and technical assistance on the implementation of the Women’s Empowerment in Agriculture Index (WEAI), manages and coordinates the integration of gender into the research of the CGIAR Research Program on Agriculture for Nutrition and Health (A4NH), and conducts research on gender, women’s empowerment, agriculture, health and nutrition issues. Before joining IFPRI, she held the Herman Postdoctoral Fellowship in Gender and Economics at the University of Michigan, Ann Arbor in 2009-2010, and conducted research on gender, labor market and data issues at the World Bank’s Gender and Development unit (PRMGE). She received herMA in Economics from the University of the Philippines, and her PhD in Economics from American University.

 

Katie SprouleKathryn Sproule is a Senior Research Assistant in the Poverty, Health and Nutrition Division (PHND) at IFPRI.  She has worked on a variety of projects related to gender and agriculture since joining IFPRI in 2013, primarily including the Women’s Empowerment in Agriculture Index, research related to the status and determinants of women’s land ownership in Asia, and the Gender, Agriculture, and Assets Project (GAAP).  Prior to joining IFPRI she worked as a Research Associate in Malawi on a girls social cash transfer project related to schooling and health outcomes funded by the World Bank.  She received her Master’s in International Development from the University of California, San Diego’s School of International Relations and Pacific Affairs.

 

Featured photo credit: ILRI/Collins

This post is part of EnGendering Data, a blog on collecting and analyzing sex-disaggregated data to improve the knowledge base on the role of gender in agriculture and food security,  maintained by the CGIAR Research Program on Policies, Institutions, and Markets (PIM). 

 

Speak Your Mind

*

*