Presenters: Nana Apenem Dagadu (Save the Children); Qundeel Khattak (Save the Children); Jessica Sadye Wolff (Stanford University); Beza Tesfaye (Mercy Corps); Katherine Armeier (Save the Children)
Populations on the move present unique challenges for program evaluators. Maintaining contact with these vulnerable populations is often extraordinarily difficult, especially when individuals are moving back and forth across international borders. This session brings together presenters from across three humanitarian and research institutions to discuss evaluations of programs for Venezuelan migrants in Peru and Colombia, and for nomadic populations in Kenya and Niger. Colleagues from Save the Children, Mercy Corps, and the Immigration Policy Lab at Stanford University will highlight their successful evaluation approaches, practical challenges, use of innovative technologies, and the resulting research findings.
Summary of recommendations for evaluations of populations on the move
Session organizer: Katherine Armeier, Save the Children
When conducting evaluations of populations on the move, it is critically important to plan for high levels of attrition and to incorporate innovative technologies to ensure communication across changing geography. These four evaluation examples show how innovative approaches such as GPS monitoring systems, mobile phone surveys, and WhatsApp can enable better, more reliable communication with nomadic and/or migrant populations. More importantly, these evaluations underscore the importance of using research to “shine the light” on the unique needs, vulnerabilities, and experiences of nomadic and migrant populations. As the Kenya evaluation shows, good quality data can help shape better policies and services for these populations.
Evaluation Without Borders: An Innovative Model for Nomadic Healthcare in Kenya
Presenter: Nana Apenem Dagadu, Save the Children
Introduction: Kenya’s Wajir and Mandera counties, where over 60 percent of the population are pastoralists, have the lowest modern contraceptive rate (mCPR; under three percent) and highest total fertility rate (over 5). The area also experiences frequent insecurity and climate-induced agricultural challenges. The Nomadic Health Project has used a mixed methods learning approach to assess underlying social norms, support health system readiness and responsiveness, and develop a cost-effective service delivery model tailored to the transitory, mobile nature of this population.
Methods: Formative research: 16 focus group discussions, 48 individual interviews, four egocentric social network analyses with women, 12 key informant interviews (community and religious leaders and service providers), a facility assessment, a developmental evaluation, and GPS-infused monitoring systems informed program design, implementation, and the project’s adaptive learning approach.
Results: As a result of the formative research studies, the project team was able to make adjustments to several components of the model. We increased the planned number of targeted health facilities from 25 to 53 to address the management, health infrastructure, and supervision gaps identified in the health facility assessment. The new Ministry of Health harmonized curriculum for family planning, which is used to train health workers, was revised to include all methods in one curriculum, arranged in a modular fashion. Revisions also included the introduction of new methods such as Sayana Press and a post-training follow-up of longer duration to sustain learning and improve capacity. As a result of gender biases identified in the assessment, Community Unit training schedules were adapted and costs increased to allow participation of female CHVs who were not able to travel outside of their homesteads to attend the training without their husbands accompanying them. To round out strategic pivots based on findings, the team invested in a policy review and increased exposure and immersion in social and behavior change activities that complemented service delivery improvements. During the COVID-19 pandemic, we leveraged the strengthened Community Unit structure to remain in contact with health workers via WhatsApp and phone surveys for supervision and monitoring data collection.
Conclusions: The current health service delivery system is not responsive to nomadic and semi-nomadic populations’ needs and lifestyle. While our solution is primarily geared towards increasing access to FP services, the model can be adapted for other health and nutrition services, basic education services, civil registration and reporting of births and deaths, and addressing harmful cultural practices.
Technology and Innovation for Evaluation: Use of Cell Phone Surveys with Venezuelan Migrants in Peru
Presenter: Qundeel Khattak, Save the Children
Introduction: SC Peru implemented a fully phone-based evaluation for a Multipurpose Cash ‘Plus’ program for Venezuelan migrants. While the baseline survey was conducted face-to-face, we quickly realized that a third of the target population of migrants were in transit to another location and those planning to settle or had already settled in Peru were dispersed, and it would be a challenge to locate them physically. SC took advantage of the high ownership rates of and access to cell phones to conduct phone-based surveys for the final evaluation.
Methods: Given the transient nature of the target population, SC, from the start, had increased the non-response rate from the usual 5% to 10% and effectively doubled the sample size from 600 to 1,200 to ensure a representative sample both at baseline and endline and to allow for a panel design. SC used Kobo Toolbox, a mobile data collection tool, via tablets and phones to administer standardized questionnaires, which allowed the team to collect data offline, if necessary. The evaluation co-leads, external consultants, also conducted key informant interviews via phone with beneficiaries, partners, and community leaders. Focus group discussions were omitted given movement restrictions as result of COVID-19 and SC believed they could not be conducted in a meaningful way virtually, particularly not knowing beneficiaries’ access to smartphones and/or internet.
Results: SC faced challenges with the panel design due to difficulties reaching participants at the end of the program. Out of the 1,234 panel participants sampled at baseline, only 636 could be reached at endline. Nearly half of the panel sample had left Peru while others had changed phone numbers. However, the phone-based methods were overall successful and cost-effective. SC saved both time and financial resources as enumerators and consultants did not have to travel to the field, and because the evaluation was conducted in the midst of the COVID-19 lockdown, the health of staff and beneficiaries was not endangered. Phone surveys also allowed beneficiaries to request a call back at a time of their convenience. Additionally, using Kobo Toolbox and tablets not only reduced the burden of manual data collection and entry for SC, but also reduced the risk of data manipulation and corruption.
Conclusions: Top three recommendations for conducting evaluations with populations on the move:
1.Sample size: due to high non-response rates, sample size should be increased to ensure representativeness and maintain comparability rigor.
2.Technology: leverage technology to reach transient populations remotely; when using phones, obtain multiple forms of contact such as personal phone numbers, alternative phone numbers, WhatsApp.
3.Survey tools: reduce the overall length of the questionnaire by focusing on ‘must have’ instead of ‘nice to have’, such as donor-required indicators and information that will be shared back with beneficiaries.
Strategies and Technologies for Conducting Panel Research with Mobile Populations: Examples from Colombia
Presenter: Jessica Sadye Wolff, Immigration Policy Lab at Stanford University
Introduction: In partnership with Mercy Corps, the Immigration Policy Lab at Stanford University designed a new survey method to conduct low-cost automated surveys entirely within WhatsApp. It was tested with a project to survey Venezuelan refugees in Colombia who were screened for a cash assistance program at 3, 6 and 9 months after the screening.
Methods: By leveraging the WhatsApp for Business API, Twilio and Google platforms, the research team designed a survey method that makes WhatsApp as a systematic survey tool. The method automates sending out a survey, and each respondent sends messages that follow pre-designed logic to complete the survey.
Results (Preliminary, after 3 months)
|
Participants with WhatsApp
|
2,172
|
|
Survey started
|
1,651 (76% of sample)
|
|
Response rate
|
1,625 (97% of started surveys)
|
|
Respondent’s identity matched to baseline interview
|
1,450 (89% of completed surveys)
|
The average cost per completed survey was $0.37 in messaging fees for a 22-question survey, plus $1.43 (5000 Colombian pesos) in phone credit upon completion.
Conclusions: This project demonstrates that WhatsApp is a viable survey tool to engage with Venezuelan refugees in Colombia. This method also offers significant data collection cost savings. Future research will assess this method in additional contexts.
Research on the Move: Findings from Recent Program Evaluations with Migrant Populations in Colombia, Niger, and Ethiopia
Presenter: Beza Tesfaye, Mercy Corps
Introduction: Evaluations with mobile populations often face significant design challenges. Because Mercy Corps routinely works with migrants and refugees, it is important to assess the impact of our support on their wellbeing. Overcoming methodological challenges in such evaluations require new tools and ideas.
Methods: In Colombia, Niger and Ethiopia we have ongoing research to help understand how development programs effect refugees and migrants’ mobility and wellbeing.
Results:
Some of the key challenges and new approaches we’ve identified in evaluations that require us to work differently with mobile populations include:
-Data challenge: High rates of attrition in traditional in-person survey
-Opportunity: Explore ways to use phone surveys to keep in touch with migrants/refugees as they move
-Data challenge: Limited existing data
-Opportunity: Going beyond survey measures, examining the potential for using administrative data (where available) or satellite images of settlements or population densities (where appropriate) to triangulate with surveys. If relying on survey measures, understanding the different ways of measuring migration outcomes (i.e. past experiences, intentions, aspirations)
Conclusions: New approaches to studying populations on the move are emerging and hold promise to overcome common challenges. These include exploring different sources of secondary and primary data, piloting different data collection methods (e.g. mobile surveys) and using different types of questions that help capture the effects of programs on mobility and wellbeing.