A recent policy focus on work opportunities for women and youth has driven a stream of studies on the livelihood strategies of these groups. Such analyses—crucial to designing effective poverty programs—often rely on lengthy, multi-topic household surveys. However, the quality of this underlying micro-data can be affected by response fatigue.
Labor modules in many surveys ask households’ primary respondents to report not only their own work activities, but those of each family member as well. Each reported work activity leads to a number of follow-up questions, gathering further information about these tasks or earnings. As the interview progresses and if a respondent gets tired (or bored) they might report fewer new activities in order to finish the interview faster.
In our recent discussion paper, Are We Done Yet? Response Fatigue and Rural Livelihoods, we examine this under-studied issue, finding that fatigue leads to underreporting of labor activities and contributions for those coming later in the question order. We also document that these effects are particularly strong for women and young people.
To identify these impacts, instead of following the order of the original household listing, we randomized the order of household members in the labor module (maintaining the respondent in the first position). Fewer unique labor activities are reported for household members listed later in the labor module by an average of 2.2% of total reported activities lost per roster position. Furthermore, we find that these effects differ by an individual’s gender and age such that labor activities of women and youth are disproportionately impacted.
We use these estimates along with initial household roster orders to estimate what aggregate losses from fatigue would have been if we had followed this non-randomized roster order (as is done conventionally). This is especially important because following the initial household ordering further biases the resulting data as youth and, to a lesser extent, women are systematically listed later, exposing them to greater amounts of response fatigue.
In our preferred model that allows for gender by age heterogeneity, we estimate average losses in reported labor activities of nearly 8%. However, these losses are not evenly distributed, as summarized in the figure below. Losses are almost twice as large for women (10%) as for men (5.3%) and are 4-5 times larger for younger age groups 14-24 (10%) and 25-34 (11.9%) than they are for the oldest age tercile, 35-59 (2.1%).
This analysis demonstrates that comparisons of labor contributions by gender or age, either within the household or across broad populations, could be substantially biased by response fatigue in analyses that employ similarly structured labor modules. Similar biases could also affect any topic and survey module where sequences of questions are repeated over a long list of similar items such as household consumption, expenditures, birth histories, or plot level inputs. Thus fatigue-induced losses could be similarly biasing our understanding of household nutrition, resource allocation, infant health, or farm production.
How can this problem be addressed? For labor modules, one solution could be to insist on every family member being interviewed personally. However, this can be costly and may not always be feasible. A middle approach, such as that followed in most Living Standards and Measurement Study surveys, would be to allow for or encourage direct consultation with the family member being asked about. While this could help for those who are available, it may also introduce other biases linked to the availability of household members. Finally, survey design could be tailored to the study’s objectives. If the aim is, in fact, to make comparisons across demographic groups, randomization of the household order might be the best approach; whereas if the study primarily wants to capture aggregate household resources, intentionally ordering the household from greatest to least contributions could be the best approach for minimizing fatigue-linked losses.
This post first appeared on the IFPRI Blog
Kate Ambler and Sylvan Herskowitz are Research Fellows with IFPRI's Markets, Trade, and Institutions Division (MTID); Mywish Maredia is a Professor in International Development at Michigan State University and co-leader of PIM's research on Science Policy and Innovation Systems for Sustainable Intensification. This post is based on research that is not yet peer-reviewed.
This project was funded by 3ie and the IFPRI-led CGIAR Research Program on Policies, Institutions, and Markets (PIM).
Photo credit: ILRI/Tu Mai
