Evaluating the Impact of Interventions to Improve Full Immunisation Rates in Haryana, India

"[T]his programme demonstrated that demand-side interventions to promote immunizations are feasible at scale, and can...complement...infrastructure [designed] to improve access [to] immunization, in particular investments in m-Health interventions."

In Haryana, India, rates decline rapidly from the initial vaccines in the immunisation schedule (e.g., Bacille Calmette-Guérin (BCG), Pentavalent-1) to the final vaccines (e.g., Pentavalent-3, measles). The rapid decline in rates suggests that the main barrier to completing full infant immunisation, which only 52.1% of children aged 12-24 months did in Haryana in 2012-13, is neither driven by a supply-side problem nor a deep-seated resistance to immunisation (such as hesitancy based on religious or personal beliefs). Hence, the researchers supported the government to implement 3 interventions in 140 government primary health centres (PHCs) across 7 districts, comprising 2,359 villages, to address potential causes of demand-side issues and possible policy responses to them. These interventions included: provision of small incentives to caregivers, a community intervention that leverages communication through social networks, and an information campaign through phone calls and text message reminders. Funded by the International Initiative For Impact Evaluation (3Ie) and the Bill & Melinda Gates Foundation, this study evaluates the project.

This programme built a common information infrastructure leveraging an mhealth software application (app) and used it to evaluate the impact of the interventions described below. This app, known in Hindi as Teekakaran Protsahan Karyakram (TPK), which roughly translates to Immunisation Encouragement Programme, was used by frontline health workers called Auxiliary Nurse Midwives (ANMs) to record details of every child who was immunised at an immunisation camp in the study sample. The ANMs were trained to use the app on tablets through a Master Trainer (or Training of Trainer) model, which was the only feasible method for a very small programme team to train almost 1,400 ANMs.

The 3 interventions, cross-randomised at different levels, were:

1. Small incentives (mobile recharges) provided to caregivers each time they brought their child to a vaccination camp. The incentives were delivered to the mobile phone number provided at the time of immunisation. Depending on which of 4 treatment arms the caregiver fell under, she would either receive a:
   • High incentive, flat payment: 90 rupees per immunisation (450 rupees total)
   • High incentive, increasing (sloped) payment: 50 rupees for the first 3 immunisations, 100 for the fourth, 200 for the fifth (450 rupees total)
   • Low incentive, flat payment: 50 rupees per immunisation (250 rupees total)
   • Low incentive, increasing (sloped) payment: 10 rupees for the first 3 immunisations, 60 for the fourth, 160 for the fifth (250 rupees total)

   ANMs and Accredited Social Health Activists (ASHAs) informed potential beneficiaries of the incentive structure and amount in the relevant villages, and a poster at every immunisation camp shared the same information.

2. Monthly text messages and voice calls sent to community-nominated key people or "seeds" in the community social network, or to a randomly selected volunteer, asking them to spread information about immunisation. A rapid survey was conducted among 17 randomly selected households to obtain the names of people who are generally good at transmitting information ("gossip"), people who are generally trusted ("trusted"), or people who are generally good at transmitting information and trusted ("trusted gossip").
3. Targeted text and voice call reminders sent to caregivers to remind them it was time for their child to receive a specific shot. This intervention also had 3 treatment arms: no reminders, reminders to 33% of caregivers, and reminders to 66% of caregivers in the treatment arm.

Each intervention had an underlying theory of change:

• The first intervention is based on the idea that small incentives can be effective in offsetting small costs, "nudging" a mother who may not have a strong view on whether to immunise her child to help overcome procrastination or compensate for the decreased salience of later vaccines.
• The second intervention tested the role of village social networks. Prior work by the authors found that community members who are more connected (had high centrality) are more effective at spreading information in their network. Further work showed that when communities are asked to nominate individuals whom they think can spread information effectively, they tend to nominate individuals with a high degree of centrality.
• The third intervention drew on the extensive body of evidence on reminders delivered to mobile phones, which shows they increase salience across many different contexts on numerous behaviours, such as saving money or taking medication. In a subset of the sample, the SMS (text messages) and voice call reminders were used to correct misconceptions about immunisation.

All interventions were implemented between December 2016 and April 2018. However, due to a technical glitch, the implementation of the targeted reminders intervention was discontinued in November 2017. All participants of the study - those who attended a session camp - received a congratulatory SMS on the registered mobile phone. For all text messages that were sent as part of the study, there was also a corresponding voice call that was sent so as to be inclusive of illiterate participants.

To estimate the impact of the incentives, communication, and reminders programmes (and cross-treatment effects), the researchers used a randomised controlled trial design, where all interventions were cross-randomised. Over the course of the project, they conducted multiple survey exercises, which are detailed in section 5 of the report. For instance, the revised endline survey was conducted from May-June 2018, and 18 focus groups with ANMs across 4 programme districts were conducted between November 2017 and February 2018. A second qualitative data collection activity was conducted with 20 primary caregivers, in the form of semi-structured, in-depth interviews (IDIs). Key methodological details:

• Incentives were randomised at the PHC level, where 70 PHCs were randomly assigned to receive incentives, and 70 PHCs were randomly assigned not to receive incentives. Within PHCs in the treatment group, the level and slope of the incentives were varied to allow for the identification of the most effective package of incentives.
• The communications experiment was cross-randomised with the incentive experiment at the village level. The team randomly selected 529 of the sample villages to receive 1 of 4 communications interventions (gossip, trust, trusted gossip, random seed) and randomised the sub-treatment arms to which they belong.
• The reminders experiment was similarly cross-randomised with the incentive treatment at the level of sub-centres (SC), the peripheral outpost of the Indian healthcare system that has a referral linkage to a PHC.

Overall, implementation of the programme was found to be successful, especially given the scale at which implementation took place. Some of the implementation highlights and challenges are outlined in the report. Overall, ANMs judged the programme as acceptable, and were generally enthusiastic about using the tablet to record data. Nevertheless, most ANMs listed technical and practical limitations in using the tablets. While it was not an original part of the programme's theory of change, the tablets seemed to have raised the self-esteem of many ANMs, who were excited to learn something new, be confident about their potential to harness technology, and impress both caregivers and their own families. For their part, most primary caregivers were positive about the programme/camp in principle, but were indifferent in terms of their experience of it. Perception of the programme seemed driven by a combination of implementer behaviour and primary caregiver expectations.

Main findings in terms of impact:

• The mobile recharge incentive led to an 11.8% increase in immunisation coverage rates in children in the intervention areas. This effect holds only when the recharge is based on a sloped schedule, where the amount is higher for the final 2 vaccines a child should receive in her first year.
• Where the "gossips" were chosen to relay information, 1,321 additional children were fully immunised every month compared to the pure control villages. The impact of "gossip seeds" appears to be larger than any other kind of seeds. In the gossip seed villages, 1.5 additional children each came for the Penta-3 vaccine, and 1.6 additional children got the Measles-1 vaccine/were fully immunised every month. ("[T]his suggests that there was no gain from explicitly trying to identify trustworthy people, even for a decision that probably requires some trust. It is very encouraging that the simplest seed identification is effective.") In brief, the mechanism by which this intervention worked was that information is "spread through social networks in a diffused manner, rather than a direct, targeted manner - it is about stirring up conversation....This was confirmed by the very brief interviews...with nominated gossip seeds: where they remembered receiving information about immunization, they could not name specific people they had passed information to and stated they would just mention it to people 'around the village'." It is important to note that the findings show that gossips only work when there are no incentives; in villages where there were both slope incentives and gossips, the effect of the gossip disappears.
• The targeted and voice call reminders component had no significant effect on immunisation outcomes. The researchers explain that these findings may be a product of the study design, and there were only 8 months of tablet data to analyse.

Just a few additional qualitative insights:

• Overall, ANMs feel the programme has been able to attract more primary caregivers to immunisation camps and has made them get vaccinations on time. Most people are reported to be aware of the importance of immunisation prior to programme, but ANMs believe that the promise of reward at the end of the process has helped. Some ANMs attribute part of the success of the programme to the tablets themselves, where primary caregivers now feel the information is legitimised.
• The topic of immunisation is uncomplicated for many primary caregivers: They don't think about it very consciously, and they don't have strong opinions on the subject. While it was difficult to establish a clear causal pathway of the incentives on immunisation behaviour from the caregiver interviews, there is evidence that incentives are liked in principle, and that they have had an impact on at least some primary caregivers. It seems incentives won't work where there are either strong negative views around immunisation (usually linked to fear of side effects) or some degree of underlying concern around side effects plus indifference to the importance of immunisation and/or an environmental or socio-demographic barrier. A key finding from the caregiver interviews is that what other people - e.g., the ASHA/ANM, other mothers, and people with positions or power or respect in the community, such as religious leaders or the village Sarpanch - say or do is an important source of primary caregivers' knowledge, attitudes, and behaviour around immunisation. "There is a sense that if everyone says it is important, or if everyone is going to the immunization camp, then it must be good."

The report outlines a number of challenges, both on the implementation side (e.g., failure of text message delivery) and the evaluation side (e.g., multiple, complex data collection activities and datasets). Based on these challenges, the researchers indicate some lessons learned and some implications for policy and practice:

• Identifying community-nominated disseminators or ambassadors to spread information about immunisation and other health services is a cost-effective option for decision-makers.
• Provided there are no major supply-side issues, and there is high coverage of households owning mobile phones, an mhealth-based system for tracking and providing even small incentives to caregivers on a sloped schedule could improve immunisation rates in a cost-effective way.
• Building on administrative data, training frontline workers, and providing continuous implementation support, supervision, and monitoring to them are critical for an m-health-based system to work.

Some implications for further research:

• Specific subgroups may need to be reached through targeted interventions: Incentives may not work for those who hold strong attitudinal resistance or experience geographic or economic or socio-demographic barriers such as migrant workers, daily wagers, and those belonging to certain religious communities.
• Buy-in from key stakeholders such as the ANMs, government officials at all levels, and the research team's field staff is critical. The researchers regularly explained the purpose and benefit of the programme to these stakeholders and solicited their feedback in terms of feasibility and acceptability of the programme and the activities they were participating in.

The research team has signed a memorandum of understanding with the Haryana government's National Health Mission directorate and the State Health Systems Resource Centre. Through the partnership, health-related information is disseminated through gossips, the approach found to be the most cost effective in the evaluation, in one of the most challenging "aspirational" districts by leveraging the government's own administrative platform.

Editor's note: A finalised version of this report was published in September 2020; click here to view or download it (121 pages).