Methods for Evaluating the Impact of Vertical Programs on Health Systems: Protocol for a Study on the Impact of the Global Polio Eradication Initiative on Strengthening Routine Immunization and Primary Health Care

"Our experience, described throughout this paper, indicates that both qualitative and quantitative approaches, undertaken systematically, have clear and substantial explanatory power. Vertical programs are of course planned and carried out by people - including doctors, surveillance officers, community health workers, and transnational monitors and experts, among others - and qualitative work has the power to illuminate the processes that drive human behavior in context."

This paper describes a research protocol integrating use of comparative ethnographies to assess the historical impact of the Global Polio Eradication Initiative (GPEI) on routine immunisation (RI) and primary health care (PHC) in diverse country and programme implementation contexts. The context of this investigation is, as the authors assert, the fact that the impact of "vertical programs on health systems is a much-debated topic, and more evidence on this complex relationship is needed." (Vertical programmes focus on specific health conditions; for more information on the distinction between vertical and horizontal approaches, click here for an explanation from the World Health Organization, WHO). With the intention that this protocol could be adapted and built upon to evaluate the strategies or impact of other vertical programmes, the researchers describe the protocol and the challenges and opportunities it presented, making recommendations to other researchers embarking on similar studies in the future.

Coordinated global polio eradication activities began in the 1990s in most of Africa and Asia, with most countries initially performing 2-3 polio vaccination campaigns per year. In those contexts, the study evaluates the impact of the GPEI on RI and PHC by: (a) combining quantitative and qualitative work into one coherent study design; (b) using purposively selected qualitative case studies (8 in total) to systematically evaluate the impact of key contextual variables; and (c) making extensive use of the method of participant observation to create comparative ethnographies of the impact of a single vertical programme administered in varied contexts (each of the 8 case studies generated around 50 interview transcripts, participant observation fieldnotes, and a range of relevant technical and operational documents). The authors' quantitative analyses used variation in the intensity of polio eradication activities (the number of campaigns per year) to get around the problem of a lack of control countries that have never carried out polio eradication activities. They also used qualitative approaches to determine why polio eradication activities had positively and negatively impacted RI and PHC in specific instances, identifying mediating factors, some of which are context-specific and others shared across different times and places. In the qualitative work, the researchers paid special attention to the impact of concurrent health system reforms and other potential contextual confounders. They aimed to evaluate two hypotheses: (1) that the initial scale-up of polio eradication activities would lead to positive effects attributed to the creation of improved quality health systems; and (2) that increases in the intensity of the number of polio vaccination campaigns (to as many as 11 per year in parts of South Asia) would be tied to negative effects due to the quantity of work that the eradication of polio demanded, siphoning already-limited resources from the health system.

According to the researchers, the study design has 4 major benefits:

1. The careful selection of a range of qualitative case studies allowed for systematic comparison - "For example, in a number of case studies - though not all of them - qualitative data supports the hypothesis that polio contributed to the development of cold chain infrastructure in the mid-90s. This was a subtle effect we could not pick up in analysis of the quantitative data available....Other factors varied across the case studies, often in patterned ways....This study design, then, allowed us to address general trends, as well as identify best practices in specific contexts that might be adapted for use in other places. "
2. The use of participant observation yielded insights on how policy is put into practice - "By visiting local health posts both during campaign days and outside of campaigns, we were able to gather information on what constituted 'normal' health post functioning, and learn how that might or might not change during the campaign."
3. Results from the quantitative analysis could be explained by results from qualitative work - "As an example of just one of many small effects, our qualitative data showed numerous examples of small-scale impact where supervision and monitoring and evaluation were performed at higher levels during the campaigns, and provided models for the health system that could be used for other services."
4. This research protocol can inform the creation of actionable recommendations - "For example, the quantitative and qualitative data that was collected in this research project could be used as baseline information to design more complex system dynamics models to test how various strategies might play out in the future."

The researchers identify several challenges they encountered in the course of this project related to:

• Inaccessibility of data: In the quantitative arm of the study, they faced difficulties accessing reliable data for their analyses, particularly on where and when polio campaigns had occurred. In some countries such data did not exist, while in others government officials were reluctant to share them. They explain that the collection of qualitative data was challenged by accessibility as well. In-country approval procedures including Institutional Review Boards (IRBs) are continually evolving and becoming more complex, and the steps required are not always clear. Thus, they recommend assigning a local partner to the position of a full-time in-country point-person. He or she would not be engaged in actual data collection, but would be responsible for nurturing political will, creating partnerships, and leading the study’s interactions with local IRBs.
• Problems of security and political instability: "For example, when our researcher in Karachi, Pakistan received a text message warning her not to go to the study site as she might be targeted, she scaled back her participant observation activities..."
• Variability in the experience and skill level of qualitative field researchers: To support their work, the authors of this paper recommend: (1) providing samples of appropriate adaptations in the research guide; (2) pairing less experienced researchers with more experienced researchers when possible; and (3) holding a series of orientation sessions for all research staff prior to conducting the research to discuss questions of appropriate modifications.
• Too few interviewees with deep historical memory: "[R]ich qualitative data illuminating past trends is especially important. Interview protocols should also pay special attention to the need to learn about events in the past."

In conclusion, the authors reflect on this study's illustration of the utility of mixed-methods research designs. "Using qualitative and quantitative methods together, including participant observation, and making use of comparative ethnography are useful tools for crafting health systems research that is attentive to local context even as it has the ability to describe global trends."